=========== 更新 ========
我在我的 JSON(struct_c 和 array_d)中添加了更多细节,以便更清楚地了解我遇到异常的原因。
============================
您好,
我有一个具有嵌套结构数组的 Spark DataFrame。我想从该结构中选择一列,但出现了错误消息:“org.apache.spark.sql.AnalysisException: 由于数据类型不匹配,无法解析 'home
.array_a
.array_b
['a']',第二个参数需要整数类型,但 'a' 的类型是字符串类型”。
以下是我的数据:
{
"home": {
"a_number": 5,
"a_string": "six",
"array_a": [
{
"array_b": [{"a": "1", "b": 2}],
"struct_c": {"a": 1.1, "b": 1.3},
"array_d": ["a", "b", "c"]
},
{
"array_b": [{"a": "3", "b": 4}],
"struct_c": {"a": 1.5, "b": 1.6},
"array_d": ["x", "y", "z"]
}
]
}
}
以下是我的数据架构:
mydf1 = spark.read.option("multiline", "true").json("myJson.json")
mydf1.printSchema()
root
|-- home: struct (nullable = true)
| |-- a_number: long (nullable = true)
| |-- a_string: string (nullable = true)
| |-- array_a: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- array_b: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- a: string (nullable = true)
| | | | | |-- b: long (nullable = true)
| | | |-- array_d: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- struct_c: struct (nullable = true)
| | | | |-- a: double (nullable = true)
| | | | |-- b: double (nullable = true)
当我从数组array_a中选择struct_c或array_d(字符串数组)中的数据时,没有任何问题。
mydf1.select("home.array_a.array_d").show(10, False)
+----------------------+
|array_d |
+----------------------+
|[[a, b, c], [x, y, z]]|
+----------------------+
mydf1.select(col("home.array_a.struct_c.a").alias("struct_field_inside_arrayA")).show(10, False)
+--------------------------+
|struct_field_inside_arrayA|
+--------------------------+
|[1.1, 1.5] |
+--------------------------+
这就是失败的地方:
mydf1.select("home.array_a.array_b.a").printSchema()
mydf1.select("home.array_a.array_b.a").show()
我需要的是一个包含字符串的二维数组([["1", "3"]] 是我的样例JSON)
你能帮忙解释一下为什么它失败了吗?
谢谢你的帮助。
执行第4行失败: mydf1.select("home.array_a.array_b.a").printSchema() 回溯信息(最近的调用在最上面): An error occurred while calling o15300.select.: org.apache.spark.sql.AnalysisException: 由于数据类型不匹配,无法解析 'home
.array_a
.array_b
['a']':参数2需要整数类型,但 ''a'' 的类型为字符串。;; 'Project [home#18213.array_a.array_b[a] AS a#18217] +- Relation[home#18213] json
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:115)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$3.applyOrElse(CheckAnalysis.scala:107)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:278)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:277)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply(TreeNode.scala:275)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:326)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:324)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:275)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$transformExpressionsUp$1.apply(QueryPlan.scala:93)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$1.apply(QueryPlan.scala:105)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:104)
at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:116)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1$2.apply(QueryPlan.scala:121)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.plans.QueryPlan.org$apache$spark$sql$catalyst$plans$QueryPlan$$recursiveTransform$1(QueryPlan.scala:121)
at org.apache.spark.sql.catalyst.plans.QueryPlan$$anonfun$2.apply(QueryPlan.scala:126)
at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:187)
at org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:126)
at org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:93)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:107)
at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
at org.apache.spark.sql