SparkContext: 运行Google DataProc时初始化SparkContext出错
在将google dataproc版本从1.3.62-debian9升级到1.4-debian后,所有的spark data proc作业都开始因以下错误而失败:
22/01/09 00:36:50 INFO org.spark_project.jetty.server.Server: Started 3339ms
22/01/09 00:36:50 INFO org.spark_project.jetty.server.AbstractConnector: Started
22/01/09 00:36:50 WARN org.apache.spark.scheduler.FairSchedulableBuilder: Fair
Scheduler configuration file not found so jobs will be scheduled in FIFO order. To use
fair scheduling, configure pools in fairscheduler.xml or set
spark.scheduler.allocation.file to a file that contains the configuration.
ERROR org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.NumberFormatException: For input string: "30s"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1441)
我在spark配置文件或sparkConf对象中没有设置“30s”。
我在代码中初始化了SparkContext:
val conf = new SparkConf().setAppName(getMainName().toString)
val sc = new SparkContext(conf)
Spark 版本 - 2.3.0
我看到׳spark.scheduler.maxRegisteredResourcesWaitingTime׳的默认设置与其它地方的数值相同( https://spark.apache.org/docs/latest/configuration.html#spark-configuration )但我没有更改或更新它。
我不明白这个值是从哪里来的,以及为什么它与更新 dataproc 有关。