logo标签列表

Docker Airflow - 错误: 无法执行'--user'安装。用户 site-packages 在此 virtualenv 中不可见。

pythondockerpipdockerfileairflow
3

目标

  • 自 2022 年 9 月 19 日发布 Apache Airflow 2.4.0 以来
  • Airflow 支持 ExternalPythonOperator
  • 我也向主要贡献者询问过，应该能够将 2 个 python 虚拟环境添加到 Airflow Docker 2.4.1 的基础映像中，并能够在 DAG 内运行单个任务。
  • 我的目标是使用从本地 requirements.txt 构建的多个主机python虚拟环境。
  • 使用 ExternalPythonOperator 运行它们（我的每个 DAG 只执行定时的 Python 函数）

代码

Dockerfile

FROM apache/airflow:2.4.1-python3.8
RUN python3 -m venv /opt/airflow/venv1
COPY requirements.txt .
RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt

终端输入

docker build -t my-image-apache/airflow:2.4.1 .

终端输出

[+] Building 4.3s (9/9) FINISHED                                                                                                                
 => [internal] load build definition from Dockerfile                                                                                       0.0s
 => => transferring dockerfile: 1.55kB                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                          0.0s
 => => transferring context: 2B                                                                                                            0.0s
 => [internal] load metadata for docker.io/apache/airflow:2.4.1-python3.8                                                                  1.2s
 => [auth] apache/airflow:pull token for registry-1.docker.io                                                                              0.0s
 => CACHED [1/4] FROM docker.io/apache/airflow:2.4.1-python3.8@sha256:5f9f4eff86993e11893f371f591aed73cf2310a96d84ae8fddec11857c6345da     0.0s
 => [internal] load build context                                                                                                          0.0s
 => => transferring context: 37B                                                                                                           0.0s
 => [2/4] RUN python3 -m venv /opt/airflow/venv1                                                                                           2.2s
 => [3/4] COPY requirements.txt .                                                                                                          0.0s
 => ERROR [4/4] RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt                                                   0.8s
------                                                                                                                                          
 > [4/4] RUN . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt:
#9 0.621 ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
#9 0.763 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#9 0.763 You should consider upgrading via the '/opt/airflow/venv1/bin/python3 -m pip install --upgrade pip' command.
------
executor failed running [/bin/bash -o pipefail -o errexit -o nounset -o nolog -c . /opt/airflow/venv1/bin/activate && pip install -r requirements.txt]: exit code: 1

尝试的解决方案

FROM apache/airflow:2.4.1-python3.8

ENV VIRTUAL_ENV=/opt/airflow/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install dependencies:
COPY requirements.txt .
RUN pip install -r requirements.txt

与上述相同的错误

FROM apache/airflow:2.4.1-python3.8
ADD . /opt/airflow/
WORKDIR /opt/airflow/

RUN python -m venv venv
RUN venv/bin/pip install --upgrade pip
RUN venv/bin/pip install -r requirements.txt

与上述相同的错误

1个回答
5

Dockerfile [正确]

FROM apache/airflow:2.4.1-python3.8

# Compulsory to switch parameter
ENV PIP_USER=false

#python venv setup
RUN python3 -m venv /opt/airflow/venv1

# Install dependencies:
COPY requirements.txt .

# --user   <--- WRONG, this is what ENV PIP_USER=false turns off
#RUN /opt/airflow/venv1/bin/pip install --user -r requirements.txt  <---this is all wrong
RUN /opt/airflow/venv1/bin/pip install -r requirements.txt
ENV PIP_USER=true
1
非常感谢你分享解决方案。在我苦思冥想了两天后，终于找到了可行的解决方案，多亏了你！ - undefined
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接