如何设置Airflow的电子邮件配置以在出现错误时发送电子邮件？

Question

如何设置Airflow的电子邮件配置以在出现错误时发送电子邮件？

12

我正在尝试通过传递一个无法正常工作的Bash命令行（thisshouldnotrun）来故意使Airflow任务失败并出错。Airflow会输出以下内容：

[2017-06-15 17:44:17,869] {bash_operator.py:94} INFO - /tmp/airflowtmpLFTMX7/run_bashm2MEsS: line 7: thisshouldnotrun: command not found
[2017-06-15 17:44:17,869] {bash_operator.py:97} INFO - Command exited with return code 127
[2017-06-15 17:44:17,869] {models.py:1417} ERROR - Bash command failed
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
    result = task_copy.execute(context=context)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 100, in execute
    raise AirflowException("Bash command failed")
AirflowException: Bash command failed
[2017-06-15 17:44:17,871] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
[2017-06-15 17:44:17,878] {models.py:1462} ERROR - Bash command failed
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/airflow", line 28, in <module>
    args.func(args)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/bin/cli.py", line 585, in test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
    result = func(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
    result = task_copy.execute(context=context)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/airflow/operators/bash_operator.py", line 100, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed

Airflow是否会为这些错误发送电子邮件？如果不会，那么发送这些错误的最佳方式是什么？

我甚至不确定airflow.cfg是否设置正确... 由于最终目标是测试电子邮件警报通知，我想确保airflow.cfg已正确设置。以下是设置：

[email]
email_backend = airflow.utils.email.send_email_smtp


[smtp]
# If you want airflow to send emails on retries, failure, and you want to use
# the airflow.utils.email.send_email_smtp function, you have to configure an
# smtp server here
smtp_host = emailsmtpserver.region.amazonaws.com 
smtp_starttls = True
smtp_ssl = False
# Uncomment and set the user/pass settings if you want to use SMTP AUTH
# smtp_user = airflow_data_user
# smtp_password = password
smtp_port = 587 
smtp_mail_from = airflow_data_user@domain.com

什么是smtp_starttls？我在文档或在线上找不到任何信息。如果我们需要两步验证来查看电子邮件，那么这对Airflow是否会有问题？

这是我的Bash命令：

task1_bash_command = """
export PATH=/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
export rundate=`TZ='America/Los_Angeles' date +%F -d "yesterday"`
export AWS_CONFIG_FILE="/home/ubuntu/.aws/config"

/home/ubuntu/bin/snowsql -f //home/ubuntu/sql/script.sql 1> /home/ubuntu/logs/"$rundate"_dev.log 2> /home/ubuntu/logs/"$rundate"_error_dev.log

if [ -e /home/ubuntu/logs/"$rundate"_error_dev.log ]
then
    exit 64
fi

我的任务：

task1 = BashOperator(
    task_id = 'run_bash',
    bash_command = task1_bash_command,
    dag = dag,
    retries = 2,
    email_on_failure = True,
    email = 'username@domain.com')

- simplycoding

3个回答

1

如果查看电子邮件需要双因素身份验证，那么这对于Airflow来说会有问题吗？

您可以使用Google应用程序密码来绕过双重身份验证。

https://support.google.com/mail/answer/185833?hl=en-GB

源 - https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html

。

- Smit Thakkar

0

使用以下链接创建Airflow DAG。
如何在本地时间午夜而不是UTC时间午夜触发每日DAG运行

方法1：您可以在本地设置SMTP并使其在作业失败时发送电子邮件。

[email]
email_backend = airflow.utils.email.send_email_smtp

[smtp]
smtp_host = localhost
smtp_starttls = False
smtp_ssl = False
smtp_port = 25
smtp_mail_from = noreply@company.com

方法二：您可以使用 Gmail 发送电子邮件。我已经撰写了一篇文章来完成这个操作。 https://helptechcommunity.wordpress.com/2020/04/04/airflow-email-configuration/

- Ganesh

网页内容由stack overflow 提供, 点击上面的

可以查看英文原文，
原文链接

- jhnclvr · Accepted Answer

smtp_starttls基本上意味着使用TLS。

如果您想使用SSL，则将其设置为False并将smtp_ssl设置为True。您可能需要smtp_user和smtp_password其中之一。

Airflow不会处理两步验证。但是，如果您使用的是AWS，则可能不需要它，因为您的SMTP（SES）凭据与您的AWS凭据不同。

请参见此处。

编辑：要使Airflow在失败时发送电子邮件，需要在任务上设置email_on_failure和email等几个属性。

例如，请参见此处：

def throw_error(**context):
    raise ValueError('Intentionally throwing an error to send an email.')



t1 = PythonOperator(task_id='throw_error_and_email',
                    python_callable=throw_error,
                    provide_context=True,
                    email_on_failure=True,
                    email='your.email@whatever.com',
                    dag=dag)