运行时错误：cudaGetDevice()失败。状态：无效的参数。

Question

我正在尝试在配备Quadro GPU的Windows 10笔记本电脑上运行tensorflow-gpu。

-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.21       Driver Version: 465.21       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro T2000       WDDM  | 00000000:01:00.0  On |                  N/A |
| N/A   59C    P0    14W /  N/A |   2708MiB /  4096MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

在尝试验证一切是否正常工作时，我发现 device_lib.list_local_devices() 出现了错误：

**RuntimeError: cudaGetDevice() failed. Status: invalid argument**


2021-01-13 11:30:14.735823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: Quadro T2000 computeCapability: 7.5
coreClock: 1.5GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.34GiB/s
2021-01-13 11:30:14.736173: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cudart64_110.dll
2021-01-13 11:30:14.736376: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cublas64_11.dll
2021-01-13 11:30:14.736590: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cublasLt64_11.dll
2021-01-13 11:30:14.736801: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cufft64_10.dll
2021-01-13 11:30:14.737016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
curand64_10.dll
2021-01-13 11:30:14.737221: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cusolver64_10.dll
2021-01-13 11:30:14.737418: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cusparse64_11.dll
2021-01-13 11:30:14.737590: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library
cudnn64_8.dll
2021-01-13 11:30:14.737787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\D041705\AppData\Local\Programs\Python\Python38\lib\site-packages\tensorflow\python\client\device_lib.py", line
43, in list_local_devices
    _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
RuntimeError: cudaGetDevice() failed. Status: invalid argument

有什么提示可以解决这个问题吗？ 我有CUDA 11.2、Python 3.8.7，还安装了最新的tf和tf-gpu包。

- kohlerm

Answer 1

我已经解决了这个问题。我认为通用的解决方法是降级你的CUDA和GPU驱动程序版本。

---

首先，根据最新的问题，TensorFlow 2.4不兼容Cuda 11.2或11.1，应使用Cuda 11.0。

其次，如果你使用的是最新的GPU驱动程序，在运行nvidia-smi时会发现CUDA版本是11.3。在我的情况下，将GPU驱动程序降级到旧版本461.09可以解决问题。

---

1. Win10 20H2
2. GTX1070m
3. tensorflow '2.4.1'

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 461.09       Driver Version: 461.09       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+

and

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:35_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.relgpu_drvr445TC445_37.28845127_0

为了测试Cuda是否正常工作，您应该使用Tensorflow进行计算，列出设备可能不会有所帮助。

import tensorflow as tf
tf.debugging.set_log_device_placement(True)

a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
c = tf.matmul(a, b)

输出：

Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0

- Jay