我已经在Kubernetes集群中安装了Airflow。为了运行任务,我正在使用KubernetesPodOperator
。
当我运行一个任务并查看kubectl get pods
时,我看到正确地创建了一个pod,并且它也完成了。然而,当我查看Airflow时,我发现状态没有更新,它仍然显示为运行状态。
[2019-01-27 12:43:56,580] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): xxx> on 2019-01-20T00:00:00+00:00
[2019-01-27 12:43:56,581] {base_task_runner.py:118} INFO - Running: ['bash', '-c', 'airflow run xxx xxx 2019-01-20T00:00:00+00:00 --job_id 15 --raw -sd DAGS_FOLDER/xxx.py --cfg_path /tmp/tmpxx39wldz']
[2019-01-27 12:45:21,603] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance Not Already Running' FAILED: Task is already running, it started on 2019-01-27 12:43:56.565328+00:00.
[2019-01-27 12:45:21,639] {models.py:1355} INFO - Dependencies not met for <TaskInstance: xxx.xxx 2019-01-20T00:00:00+00:00 [running]>, dependency 'Task Instance State' FAILED: Task is in the 'running' state which is not a valid state for execution. The task must be cleared in order to be run.
[2019-01-27 12:45:21,641] {logging_mixin.py:95} INFO - [2019-01-27 12:45:21,641] {jobs.py:2614} INFO - Task is not able to be run
我需要做什么来将Pod的状态返回给Airflow?KubernetesPodOperator
定义如下:
do_something = KubernetesPodOperator(
task_id='xxx',
image='gcr.io/project/image',
namespace='default',
name='xxx',
arguments=['dummy'],
xcom_push=True,
in_cluster=True,
image_pull_policy='Always',
trigger_rule='dummy',
dag=dag,
)
编辑: 看起来基础容器已经完成了,但是airflow-xcom-sidecar
仍在运行。我需要做些什么来停止它呢?
/airflow/xcom/return.json
已经被正确创建,并且基础容器已经完成。 - aaaaahaaaaareturn.json
文件已由基础容器创建,这意味着问题不是边车一直在寻找它或其他什么原因。并且我在Airflow上收到了相同的错误消息(“...依赖项'Task Instance State'失败...”)。 - aaaaahaaaaa