停止Spark Streaming的默认输出

Question

有没有一种方法可以禁用 Spark streaming 中显示的默认输出。以下是在运行 Spark streaming 应用程序时打印的默认输出。我希望只看到我添加的日志消息（在驱动程序中）被打印。

spark-submit --class a11_TCP_IP.a11_Basic.a11_HelloTcp target/scala-2.10/sparkstreamingexamples_2.10-1.0.jar

16/04/27 13:20:15 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:15 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
16/04/27 13:20:17 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:17 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777618000 ms
-------------------------------------------

16/04/27 13:20:19 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:19 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777621000 ms
-------------------------------------------

16/04/27 13:20:21 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data

- Raj

Answer 1

您的Spark文件夹应该有一个名为conf/的目录，在其中有一个log4j.properties文件。如果没有，请将conf/log4j.properties.template复制到conf/log4j.properties。

现在在编辑器中打开conf/log4j.properties，并替换此行：

log4j.rootCategory=INFO, console

使用以下两种方法：

log4j.rootCategory=INFO, null
log4j.appender.null=org.apache.log4j.varia.NullAppender

- Daniel Zolnai

---

谢谢。这确实解决了错误和警告。但我仍然得到这个输出 ------------------------------------------- Time: 1462045608000 ms -------------------------------------------。 - Raj

好的。我终于找到了问题所在。DStream 的输出操作 Dstream.print() 默认会给出那个输出（即 ---- Time: ....）。我使用了另一个输出操作 DStream.foreachRDD(..)，这帮助我解决了这个问题。 - Raj

Answer 2

我正在使用IntelliJ进行Spark Streaming，但无法看到conf文件夹。因此，我使用代码来调整日志记录。

Logger.getLogger("org").setLevel(Level.ERROR) //需要导入库

- Joy George Kunjikkuru