我有大量的json文件,Spark 2.0可以在36秒内读取这些文件,但是Spark 3.0要花费近33分钟才能读取。经过更仔细的分析,看起来Spark 3.0选择的DAG与Spark 2.0不同。有没有人知道发生了什么?Spark 3.0是否存在任何配置问题。
Spark 2.4
scala> spark.time(spark.read.json("/data/20200528"))
Time taken: 19691 ms
res61: org.apache.spark.sql.DataFrame = [created: bigint, id: string ... 5 more fields]
scala> spark.time(res61.count())
Time taken: 7113 ms
res64: Long = 2605349
Spark 3.0
scala> spark.time(spark.read.json("/data/20200528"))
20/06/29 08:06:53 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
Time taken: 849652 ms
res0: org.apache.spark.sql.DataFrame = [created: bigint, id: string ... 5 more fields]
scala> spark.time(res0.count())
Time taken: 8201 ms
res2: Long = 2605349
以下是翻译的结果:
以下是详细信息: