logo标签列表

Airflow: 值错误：无法配置处理程序 'processor' - wasb 日志记录器

airflowairflow-scheduler
12
我正在尝试配置使用Azure blob进行远程日志记录。
Airflow version: 1.10.2
Python: 3.6.5
Ubuntu: 18.04

以下是我的步骤:
  1. 在$AIRFLOW_HOME/config/log_config.py中，我将REMOTE_BASE_LOG_FOLDER设置为“wasb-airflow-logs”（这是容器内的一个文件夹（容器名称：airflow-logs））
  2. $AIRFLOW_HOME/config/下有一个空init.py文件
  3. 将$AIRFLOW_HOME/config/添加到$PYTHONPATH中
  4. 在$AIRFLOW_HOME/config/log_config.py中，将DEFAULT_LOGGING_CONFIG重命名为LOGGING CONFIG
  5. Airflow blob连接中定义的用户具有对REMOTE_BASE_LOG_FOLDER的读写访问权限
  6. $AIRFLOW_HOME/airflow.cfg中remote_logging = True logging_config_class = log_config.LOGGING_CONFIG remote_log_conn_id =
以下是错误信息:
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 382, in resolve
    found = getattr(found, frag)
AttributeError: module 'airflow.utils.log' has no attribute 'wasb_task_handler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
    self.importer(used)
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
    from airflow.contrib.hooks.wasb_hook import WasbHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
    from airflow.hooks.base_hook import BaseHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
    from airflow.models import Connection
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
    from airflow.utils.dag_processing import list_py_file_paths
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
    from airflow.settings import logging_class_path
ImportError: cannot import name 'logging_class_path'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 558, in configure
    handler = self.configure_handler(handlers[name])
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 708, in configure_handler
    klass = self.resolve(cname)
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 391, in resolve
    raise v
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 384, in resolve
    self.importer(used)
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/log/wasb_task_handler.py", line 23, in <module>
    from airflow.contrib.hooks.wasb_hook import WasbHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/contrib/hooks/wasb_hook.py", line 22, in <module>
    from airflow.hooks.base_hook import BaseHook
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 28, in <module>
    from airflow.models import Connection
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/models.py", line 86, in <module>
    from airflow.utils.dag_processing import list_py_file_paths
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/utils/dag_processing.py", line 49, in <module>
    from airflow.settings import logging_class_path
ValueError: Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/gsingh/venv/bin/airflow", line 21, in <module>
    from airflow import configuration
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/__init__.py", line 36, in <module>
    from airflow import settings, configuration as conf
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/settings.py", line 262, in <module>
    logging_class_path = configure_logging()
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 73, in configure_logging
    raise e
  File "/home/gsingh/venv/lib/python3.6/site-packages/airflow/logging_config.py", line 68, in configure_logging
    dictConfig(logging_config)
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 795, in dictConfig
    dictConfigClass(config).configure()
  File "/home/gsingh/anaconda3/lib/python3.6/logging/config.py", line 566, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'processor': Cannot resolve 'airflow.utils.log.wasb_task_handler.WasbTaskHandler': cannot import name 'logging_class_path'

我不确定我缺少哪个配置。有人面对过同样的问题吗？

3个回答
6

我遇到了同样的错误，但是如果我向上滚动得更高一些，我就可以看到在 ValueError 之前抛出了另一个异常。这是一个 PermissionError。

PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler'

我遇到错误的原因是在运行Airflow Docker容器之前没有创建初始的三个文件夹（dags，logs，plugins）。所以Docker似乎已经自动创建了这些文件夹，但权限不正确。
解决步骤：
  1. 停止当前容器
docker-compose down --volumes --remove-orphans
  1. 删除文件夹 dags、logs、plugins
  2. 以防万一，销毁已创建的镜像和卷（在 Docker Desktop 中）
  3. 从命令行重新创建文件夹
mkdir logs dags plugins
  1. 重新运行Airflow Docker
docker-compose up airflow-init
docker-compose up
4
在我的情况下，运行sudo chown 50000:0 dags logs plugins。我尝试以官方docker-compose.yml中的所有容器运行这些容器（这些容器依赖于这3个卷转发）或仅将airflow standalone包装到单个容器中以进行调试目的。结果发现卷具有根拥有权而不是airflows。
3
你需要安装Azure包。
pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]

根据 updating.md，现在应安装此项：
pip install apache-airflow[azure]

但这对我没用。
在Airflow 1.10.3中，pip install apache-airflow[azure]已被弃用； - Liangjun
1
好知道，文档不太全。 - Daniel Lee
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接