我正在尝试通过传递一个无法正常工作的Bash命令行(thisshouldnotrun
)来故意使Airflow任务失败并出错。Airflow会输出以下内容:
[2017-06-15 17:44:17,869] {bash_operator.py:94} INFO - /tmp/airflowtmpLFTMX7/run_bashm2MEsS: line 7: thisshouldnotrun: command not found
[2017-06-15 17:44:17,869] {bash_operator.py:97} INFO - Command exited with return code 127
[2017-06-15 17:44:17,869] {models.py:1417} ERROR - Bash command failed
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
result = task_copy.execute(context=context)
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 100, in execute
raise AirflowException("Bash command failed")
AirflowException: Bash command failed
[2017-06-15 17:44:17,871] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
[2017-06-15 17:44:17,878] {models.py:1462} ERROR - Bash command failed
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/airflow", line 28, in <module>
args.func(args)
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
result = func(*args, **kwargs)
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
result = task_copy.execute(context=context)
File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 100, in execute
raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed
Airflow是否会为这些错误发送电子邮件?如果不会,那么发送这些错误的最佳方式是什么?
我甚至不确定airflow.cfg
是否设置正确... 由于最终目标是测试电子邮件警报通知,我想确保airflow.cfg已正确设置。以下是设置:
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = emailsmtpserver.region.amazonaws.com
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
# smtp_user = airflow_data_user
# smtp_password = password
smtp_port = 587
smtp_mail_from = airflow_data_user@domain.com
什么是smtp_starttls
?我在文档或在线上找不到任何信息。如果我们需要两步验证来查看电子邮件,那么这对Airflow是否会有问题?
这是我的Bash命令:
task1_bash_command = """
export PATH=/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
export rundate=`TZ='America/Los_Angeles' date +%F -d "yesterday"`
export AWS_CONFIG_FILE="/home/ubuntu/.aws/config"
/home/ubuntu/bin/snowsql -f //home/ubuntu/sql/script.sql 1> /home/ubuntu/logs/"$rundate"_dev.log 2> /home/ubuntu/logs/"$rundate"_error_dev.log
if [ -e /home/ubuntu/logs/"$rundate"_error_dev.log ]
then
exit 64
fi
我的任务:
task1 = BashOperator(
task_id = 'run_bash',
bash_command = task1_bash_command,
dag = dag,
retries = 2,
email_on_failure = True,
email = 'username@domain.com')
thisshouldnotrun
不是一个有效的命令,但它继续到下一行,因此似乎没有抛出任何错误。顺带一问 - 运行airflow test
时会发送电子邮件通知吗? - simplycodingairflow.cfg
设置有误,或者airflow test
在失败时没有发送电子邮件。 - simplycodingsubprocess.run(...)
或该Python库的变体之一来运行Linux命令。然后,您可以自己评估返回值,并在值不符合预期时手动抛出AirflowException
。如果抛出AirflowException,则始终将任务标记为失败。因此,请随意发挥创造力,不要受限于操作员 :) - Kyle Bridenstine