我有一个带有模式的DataFrame
root
|-- label: string (nullable = true)
|-- features: struct (nullable = true)
| |-- feat1: string (nullable = true)
| |-- feat2: string (nullable = true)
| |-- feat3: string (nullable = true)
虽然我能够使用筛选器筛选数据框
val data = rawData
.filter( !(rawData("features.feat1") <=> "100") )
我无法使用
drop命令删除列。
val data = rawData
.drop("features.feat1")
我在这里做错了什么吗？我还尝试过（不成功地）执行 drop(rawData("features.feat1"))，尽管这样做没有太多意义。
提前致谢，
Nikhil
Make Structs Easy* 库使得在嵌套数据结构中执行添加、删除和重命名字段等操作变得容易。该库在 Scala 和 Python 中都可用。
假设您有以下数据：
import org.apache.spark.sql.functions._
case class Features(feat1: String, feat2: String, feat3: String)
case class Record(features: Features, arrayOfFeatures: Seq[Features])
val df = Seq(
Record(Features("hello", "world", "!"), Seq(Features("red", "orange", "yellow"), Features("green", "blue", "indigo")))
).toDF
df.printSchema
// root
// |-- features: struct (nullable = true)
// | |-- feat1: string (nullable = true)
// | |-- feat2: string (nullable = true)
// | |-- feat3: string (nullable = true)
// |-- arrayOfFeatures: array (nullable = true)
// | |-- element: struct (containsNull = true)
// | | |-- feat1: string (nullable = true)
// | | |-- feat2: string (nullable = true)
// | | |-- feat3: string (nullable = true)
df.show(false)
// +-----------------+----------------------------------------------+
// |features |arrayOfFeatures |
// +-----------------+----------------------------------------------+
// |[hello, world, !]|[[red, orange, yellow], [green, blue, indigo]]|
// +-----------------+----------------------------------------------+
然后从features中删除feat2就像这样简单：
import com.github.fqaiser94.mse.methods._
// drop feat2 from features
df.withColumn("features", $"features".dropFields("feat2")).show(false)
// +----------+----------------------------------------------+
// |features |arrayOfFeatures |
// +----------+----------------------------------------------+
// |[hello, !]|[[red, orange, yellow], [green, blue, indigo]]|
// +----------+----------------------------------------------+
我注意到其他解决方案的跟进评论中有很多人问是否有一种方法可以删除嵌套在数组内的结构体内部的列。通过将Make Structs Easy库提供的函数与spark-hofs库提供的函数组合，可以实现这一目标，具体做法如下：
import za.co.absa.spark.hofs._
// drop feat2 in each element of arrayOfFeatures
df.withColumn("arrayOfFeatures", transform($"arrayOfFeatures", features => features.dropFields("feat2"))).show(false)
// +-----------------+--------------------------------+
// |features |arrayOfFeatures |
// +-----------------+--------------------------------+
// |[hello, world, !]|[[red, yellow], [green, indigo]]|
// +-----------------+--------------------------------+
*完全透明：我是Make Structs Easy库的作者，该库在此答案中被引用。
.columns()方法获取所有列，从序列中删除不需要的列，然后执行select(myColumns:_*)。这样会更简洁一些。 - TheMP