Apache Oozie可以运行Docker容器吗？

Question

目前正在比较像Airflow和Luigi这样的DAG工作流工具，以安排通用的docker容器以及Spark作业。

Apache Oozie能否通过其shell操作运行通用的Docker容器？还是Oozie仅适用于诸如Pig和Hive等Hadoop工具？

Oozie与Hadoop堆栈的其余部分集成在一起，支持开箱即用的几种类型的Hadoop作业（例如Java map-reduce，Streaming map-reduce，Pig，Hive，Sqoop和Distcp），以及系统特定的作业（例如Java程序和shell脚本）。

- Kermit

---

Oozie 从 HDFS 中获取作业可执行文件和资源，并启动 YARN 作业（即使是普通的 Bash shell）。句号。故事结束。 - Samson Scharfrichter

Answer 1

我已经尝试通过Shell操作来运行Docker容器，并且它是可以工作的。由于Shell操作可以在集群的任何节点上执行，因此必须在任何节点上安装Docker。

从Hue创建的workflow.xml文件。

<workflow-app name="Test docker" xmlns="uri:oozie:workflow:0.5">
    <start to="shell-5c29"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="shell-5c29">
        <shell xmlns="uri:oozie:shell-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>test_docker.sh</exec>
            <file>/test_docker.sh#test_docker.sh</file>
        </shell>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

test_docker.sh

docker run hello-world > output.txt
hdfs dfs -put -f output.txt /output.txt
echo 'done'

生成的output.txt文件内容

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub (amd64)
 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

- AGL