我想将我的字典列表转换为DataFrame。这是列表:
mylist =
[
{"type_activity_id":1,"type_activity_name":"xxx"},
{"type_activity_id":2,"type_activity_name":"yyy"},
{"type_activity_id":3,"type_activity_name":"zzz"}
]
这是我的代码:
from pyspark.sql.types import StringType
df = spark.createDataFrame(mylist, StringType())
df.show(2,False)
+-----------------------------------------+
| value|
+-----------------------------------------+
|{type_activity_id=1,type_activity_id=xxx}|
|{type_activity_id=2,type_activity_id=yyy}|
|{type_activity_id=3,type_activity_id=zzz}|
+-----------------------------------------+
我认为我应该为每个列提供一些映射和类型,但我不知道该怎么做。
更新:
我也尝试了这个:
schema = ArrayType(
StructType([StructField("type_activity_id", IntegerType()),
StructField("type_activity_name", StringType())
]))
df = spark.createDataFrame(mylist, StringType())
df = df.withColumn("value", from_json(df.value, schema))
但是我得到了null
值:
+-----+
|value|
+-----+
| null|
| null|
+-----+
spark.createDataFrame(Row(**x) for x in mylist)
方法时,我遇到了下游问题,其中值被放置在错误的列中。可能与 https://issues.apache.org/jira/browse/SPARK-26200 相关。 - Daniel Himmelstein