我一直在寻找以下问题的解决方案。我使用的是Scala 2.11.8
和Spark 2.1.0
。
Application application_1489191400413_3294 failed 1 times due to AM Container for appattempt_1489191400413_3294_000001 exited with exitCode: -104
For more detailed output, check application tracking page:http://ip-172-31-17-35.us-west-2.compute.internal:8088/cluster/app/application_1489191400413_3294Then, click on links to logs of each attempt.
Diagnostics: Container [pid=23372,containerID=container_1489191400413_3294_01_000001] is running beyond physical memory limits.
Current usage: 1.4 GB of 1.4 GB physical memory used; 3.5 GB of 6.9 GB virtual memory used. Killing container.
请注意,我分配的内存比这里报告的“1.4 GB”要多得多。由于我没有看到我的执行程序失败,所以我从这个错误中读取的信息是驱动程序需要更多内存。然而,我的设置似乎没有传播。
我将作业参数设置为yarn,如下所示:
val conf = new SparkConf()
.setAppName(jobName)
.set("spark.hadoop.mapred.output.committer.class", "com.company.path.DirectOutputCommitter")
additionalSparkConfSettings.foreach { case (key, value) => conf.set(key, value) }
// this is the implicit that we pass around
implicit val sparkSession = SparkSession
.builder()
.appName(jobName)
.config(conf)
.getOrCreate()
在additionalSparkConfSettings
中设置内存分配参数的代码如下:
HashMap[String, String](
"spark.driver.memory" -> "8g",
"spark.executor.memory" -> "8g",
"spark.executor.cores" -> "5",
"spark.driver.cores" -> "2",
"spark.yarn.maxAppAttempts" -> "1",
"spark.yarn.driver.memoryOverhead" -> "8192",
"spark.yarn.executor.memoryOverhead" -> "2048"
)
我的设置真的没有传播吗?还是我误解了日志?
谢谢!
spark.yarn.driver.memoryOverhead
更改为 10240,但作业仍然以我上面提到的完全相同的错误失败。但是,当我将spark.driver.memory
增加了几个 GB 时,它成功了。看起来memoryOverhead
配置根本不起作用。 - Navneet