Ariflow [Errno 104] Connection reset by peer

4

我试图通过命令“airflow scheduler”运行任务,但在尝试运行其中一项dag后出现了以下错误。

Traceback (most recent call last):
File "/usr/local/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 839, in scheduler
job.run()
File "/usr/local/lib/python3.5/dist-packages/airflow/jobs.py", line 200, in run
self._execute()
File "/usr/local/lib/python3.5/dist-packages/airflow/jobs.py", line 1309, in _execute
self._execute_helper(processor_manager)
File "/usr/local/lib/python3.5/dist-packages/airflow/jobs.py", line 1441, in _execute_helper
self.executor.heartbeat()
File "/usr/local/lib/python3.5/dist-packages/airflow/executors/base_executor.py", line 132, in heartbeat
self.sync()
File "/usr/local/lib/python3.5/dist-packages/airflow/executors/celery_executor.py", line 88, in sync
state = async.state
File "/home/userName/.local/lib/python3.5/site-packages/celery/result.py", line 436, in state
return self._get_task_meta()['status']
File "/home/userName/.local/lib/python3.5/site-packages/celery/result.py", line 375, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "/home/userName/.local/lib/python3.5/site-packages/celery/backends/amqp.py", line 156, in get_task_meta
binding.declare()
File "/home/userName/.local/lib/python3.5/site-packages/kombu/entity.py", line 605, in declare
self._create_queue(nowait=nowait, channel=channel)
File "/home/userName/.local/lib/python3.5/site-packages/kombu/entity.py", line 614, in _create_queue
self.queue_declare(nowait=nowait, passive=False, channel=channel)
File "/home/userName/.local/lib/python3.5/site-packages/kombu/entity.py", line 649, in queue_declare
nowait=nowait,
 File "/home/userName/.local/lib/python3.5/site-packages/amqp/channel.py", line 1147, in queue_declare
nowait, arguments),
File "/home/userName/.local/lib/python3.5/site-packages/amqp/abstract_channel.py", line 50, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/home/userName/.local/lib/python3.5/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/home/userName/.local/lib/python3.5/site-packages/amqp/transport.py", line 258, in write
self._write(s)
**ConnectionResetError: [Errno 104] Connection reset by peer**

我使用的是 Python 3.5、Airflow 1.8、Celery 4.1.0 和 RabbitMQ 3.5.7 作为工作顺序: 看起来,我的 RabbitMQ 出现了问题,但我找不出原因。


你最终解决了这个问题吗?我也遇到了同样的问题。 - trejas
@trejas 我们之前没有解决这个问题。下面提出的解决方案 https://issues.apache.org/jira/browse/AIRFLOW-342 可能会有所帮助。对于我们来说,我们将我们的代理商从Rabbitmq改为SQL。 - SMDC
2个回答

0

报告的错误似乎是已在Airflow 1.10.0中解决的已知错误


0

遇到了相同的问题。

你的dag包含了对服务器的多个API调用,而你的airflow调度器有一定的限制,虽然没有明确规定可以同时发送多少个请求,但你需要不断尝试找到适合你的Airflow环境的数字。通常情况下,当你的dag有n个任务需要同时运行时,就会出现这个问题。

这个问题无法通过回答中提到的任何更新来解决,即使使用最新版本时我仍然遇到了这个错误。


网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接