Pytorch在Docker容器上使用GPU时出现错误 - 没有检测到支持CUDA的设备

Question

我正在尝试在我的Docker容器中使用Pytorch与GPU。

1. 在主机上 - 我已经安装了nvidia-docker、CUDA驱动程序等。

以下是主机上的nvidia-smi输出：

    Fri Mar 20 04:29:49 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P8    28W / 149W |     16MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1860      G   /usr/lib/xorg/Xorg                            15MiB |
+-----------------------------------------------------------------------------+

2. 在Docker容器中（应用程序的Dockerfile和下面的Docker Compose文件）-

FROM ubuntu:latest
FROM dsksd/pytorch:0.4
#FROM nvidia/cuda:10.1-base-ubuntu18.04 
#FROM nablascom/cuda-pytorch
#FROM nvidia/cuda:10.0-base

RUN apt-get update -y --fix-missing
RUN apt-get install -y python3-pip python3-dev build-essential
RUN apt-get install -y sudo curl
#RUN sudo apt-get install -y nvidia-container-toolkit
#RUN apt-get install -y curl python3.7 python3-pip python3.7-dev python3.7-distutils build-essential
#RUN apt-get install -y curl
#RUN apt-get install -y sudo
#RUN curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
#RUN sudo apt-get install cuda -y
#----------
# Add the package repositories
#RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
#RUN sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
#RUN sudo systemctl restart docker
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/usr/local/cuda-10.1/compat/
ENV PYTHONPATH $PATH
#----------
ENV LC_ALL=mylocale.utf8
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python3"]
EXPOSE 5000
CMD ["hook.py"]

当我尝试在GPU上运行我的代码时，遇到以下问题:

>>> torch.cuda.current_device()
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 193, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50

我使用以下命令启动容器：docker-compose up --build

这是我的docker-compose.yaml文件：

version: '3.6'
services:
  rdb:
    image: mysql:5.7
    #restart: always
    environment:
      MYSQL_DATABASE: 'c_rdb'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password'
      MYSQL_ROOT_PASSWORD: '123123'
    #ports:
    #  - '3306:3306'
    #expose:
    #  - '3306'
    volumes:
      - rdb-data:/var/lib/mysql
      - ./init-db/init.sql:/docker-entrypoint-initdb.d/init.sql
  mongo:
    image: mongo
    #restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 12312323
      MONGO_INITDB_DATABASE: chronicler_ndb
    volumes:
      - ndb-data:/data/db
      - ./init-db/init.js:/docker-entrypoint-initdb.d/init.js
    ports:
      - '27017-27019:27017-27019'
  mongo-express:
    image: mongo-express
    #restart: always
    depends_on:
        - mongo
        - backend
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: rooer
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123123
  redis:
    image: redis:latest
    command: ["redis-server", "--appendonly", "yes"]
    hostname: redis
    #ports:
    #  - "6379:6379"
    volumes:
      - cache-data:/data
  backend:
    build: ./app
    ports:
     - "5000:5000"
    volumes:
     - backend-data:/code
    links: 
     - rdb
     - redis

volumes:
  rdb-data:
    name: c-relational-data
  ndb-data:
    name: c-nosql-data
  cache-data:
    name: redis-data
  backend-data:
    name: backend-engine

- user10837800

---

你如何运行容器？ - Smankusors

我有一个 docker-compose.yaml 文件，它基于这个“app” Dockerfile 构建，命令：docker-compose up --build。 - user10837800

那你的docker-compose.yml文件在哪里？你也需要发布它 :/ - Smankusors

在原帖中添加。谢谢！ - user10837800

是的，没错。 - user10837800

Answer 1

这需要一些运行时选项，但是在compose文件格式3中并没有可用的运行时选项。因此，有一些选择：

1. 将您的compose文件版本降级到2，就像这样：

version: 2
  backend:
    build: ./app
    ports:
     - "5000:5000"
    volumes:
     - backend-data:/code
    links: 
     - rdb
     - redis
    runtime: nvidia

2. 或者，使用docker run命令手动运行容器，并加上--runtime=nvidia参数。

我也建议使用nvidia构建的镜像，而不是ubuntu:latest

---

如需更多信息，请参阅此处的问题

- Smankusors

---

你知道如何将docker-compose文件转换的示例/建议吗？ - user10837800

你的意思是将docker-compose转换为docker run吗？如果是这样，你可以在这里阅读文档：https://docs.docker.com/engine/reference/run - Smankusors

Answer 2

我获取了由cudaError_t定义的cudaErrorNoDevice，意思是：

这表示安装的CUDA驱动程序未检测到任何支持CUDA的设备。

当我关闭笔记本电脑屏幕（而没有关机，只是合上盖子）并且docker没有退出时，出现了这种情况。重新启动docker可以解决这个问题。

- Crawl.W