Keras和GPU的使用

Question

我正在使用带有Theano后端的keras 1.1.0。为了使用GPU，我设置了os.envorn如下：

os.environ['THEANO_FLAGS'] = "device=gpu0"

在导入Keras之前，我收到了以下信息：

Using gpu device 0: TITAN X (Pascal) (CNMeM is disabled, cuDNN 5105)
/home/library_python/lib/python/Theano-0.8.2-py2.7.egg/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.
  warnings.warn(warn)

我原以为我已经正确设置了设备。然而，当我使用keras训练模型时，nvidia-smi显示以下信息：

Fri May  5 13:05:33 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.51                 Driver Version: 375.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 730      On   | 0000:01:00.0     N/A |                  N/A |
| 30%   38C    P8    N/A /  N/A |     56MiB /   980MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    On   | 0000:02:00.0     Off |                  N/A |
| 36%   63C    P2    87W / 250W |    373MiB / 12189MiB |     35%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    On   | 0000:03:00.0     Off |                  N/A |
| 23%   42C    P8    18W / 250W |      3MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
|    1      1520    C   /usr/bin/python                                155MiB |
|    1     14996    C   python                                         215MiB |
+-----------------------------------------------------------------------------+

显示我只使用了215MB的GPU内存。我预计会有10GB的使用量。请问我错过了什么？我的GPU设置正确吗？

- FairyOnIce

Answer 1

更改Theano_flag后，问题已得到解决：

os.environ['THEANO_FLAGS'] = "device=gpu0,floatX=float32,lib.cnmem=1"

现在我正在使用11.7G的内存，正如预期的那样。

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.51                 Driver Version: 375.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 730      On   | 0000:01:00.0     N/A |                  N/A |
| 30%   38C    P8    N/A /  N/A |     56MiB /   980MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    On   | 0000:02:00.0     Off |                  N/A |
| 28%   47C    P2    78W / 250W |  11923MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    On   | 0000:03:00.0     Off |                  N/A |
| 23%   42C    P8    18W / 250W |      3MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
|    1      1520    C   /usr/bin/python                                155MiB |
|    1     15543    C   python                                       11765MiB |
+-----------------------------------------------------------------------------+

- FairyOnIce

---

建议将 cnmem 裁剪到 0.95。 - Autonomous

其实，我可以问一下cnmem=1具体是做什么的吗？ - FairyOnIce

它基本上是预分配GPU内存。据观察，设置“cnmem”可以提高速度。但是，您不希望阻止GPU的所有内存，因此惯例是将其剪切到“0.95”。 - Autonomous

Answer 2

如果你决定使用tensorflow，你可以这样做：

#Take advantage of GPU for parallel processing, but grow as needed
cfg = tf.ConfigProto()
cfg.gpu_options.allow_growth=True
config = tf.estimator.RunConfig(session_config=cfg)

- Arnav