TensorFlow: Blas GEMM启动失败

Question

当我尝试使用gpu在Keras中使用TensorFlow时，出现以下错误消息:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:2: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, epochs=2, validation_data=<keras.pre..., validation_steps=4200)`
  from ipykernel import kernelapp as app

Epoch 1/2

InternalError                             Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1038     try:
-> 1039       return fn(*args)
   1040     except errors.OpError as e:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1020                                  feed_dict, fetch_list, target_list,
-> 1021                                  status, run_metadata)
   1022 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    465           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466           pywrap_tensorflow.TF_GetCode(status))
    467   finally:

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-13-2a52d1079a66> in <module>()
      1 history=model.fit_generator(batches, batches.n, nb_epoch=2, 
----> 2                     validation_data=val_batches, nb_val_samples=val_batches.n)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1108                                         workers=workers,
   1109                                         pickle_safe=pickle_safe,
-> 1110                                         initial_epoch=initial_epoch)
   1111 
   1112     @interfaces.legacy_generator_methods_support

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1888                     outs = self.train_on_batch(x, y,
   1889                                                sample_weight=sample_weight,
-> 1890                                                class_weight=class_weight)
   1891 
   1892                     if not isinstance(outs, list):

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
   1631             ins = x + y + sample_weights
   1632         self._make_train_function()
-> 1633         outputs = self.train_function(ins)
   1634         if len(outputs) == 1:
   1635             return outputs[0]

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
   2227         session = get_session()
   2228         updated = session.run(self.outputs + [self.updates_op],
-> 2229                               feed_dict=feed_dict)
   2230         return updated[:len(self.outputs)]
   2231 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    776     try:
    777       result = self._run(None, fetches, feed_dict, options_ptr,
--> 778                          run_metadata_ptr)
    779       if run_metadata:
    780         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    980     if final_fetches or final_targets:
    981       results = self._do_run(handle, final_targets, final_fetches,
--> 982                              feed_dict_string, options, run_metadata)
    983     else:
    984       results = []

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1030     if handle is None:
   1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032                            target_list, options, run_metadata)
   1033     else:
   1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1050         except KeyError:
   1051           pass
-> 1052       raise type(e)(node_def, op, message)
   1053 
   1054   def _extend_graph(self):

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

Caused by op 'dense_1/MatMul', defined at:
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-10-1e7a3b259f23>", line 4, in <module>
    model.add(Dense(10, activation='softmax'))
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
    output_tensor = layer(self.outputs[0])
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
    output = K.dot(inputs, self.kernel)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
    out = tf.matmul(x, y)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]

当我尝试使用CPU来使用TensorFlow和Keras时，我遇到了以下错误信息：

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py:5: UserWarning: Update your `fit_generator` call to the Keras 2 API: `fit_generator(<keras.pre..., 37800, validation_steps=4200, validation_data=<keras.pre..., epochs=2)`
Epoch 1/2
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1038     try:
-> 1039       return fn(*args)
   1040     except errors.OpError as e:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
   1020                                  feed_dict, fetch_list, target_list,
-> 1021                                  status, run_metadata)
   1022 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\contextlib.py in __exit__(self, type, value, traceback)
     65             try:
---> 66                 next(self.gen)
     67             except StopIteration:

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py in raise_exception_on_not_ok_status()
    465           compat.as_text(pywrap_tensorflow.TF_Message(status)),
--> 466           pywrap_tensorflow.TF_GetCode(status))
    467   finally:

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
     [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

During handling of the above exception, another exception occurred:

InternalError                             Traceback (most recent call last)
<ipython-input-14-f66b4d3d5b88> in <module>()
      3 with tf.device('/cpu:0'):
      4     history=model.fit_generator(batches, batches.n, nb_epoch=2, 
----> 5                     validation_data=val_batches, nb_val_samples=val_batches.n)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1108                                         workers=workers,
   1109                                         pickle_safe=pickle_safe,
-> 1110                                         initial_epoch=initial_epoch)
   1111 
   1112     @interfaces.legacy_generator_methods_support

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_q_size, workers, pickle_safe, initial_epoch)
   1888                     outs = self.train_on_batch(x, y,
   1889                                                sample_weight=sample_weight,
-> 1890                                                class_weight=class_weight)
   1891 
   1892                     if not isinstance(outs, list):

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py in train_on_batch(self, x, y, sample_weight, class_weight)
   1631             ins = x + y + sample_weights
   1632         self._make_train_function()
-> 1633         outputs = self.train_function(ins)
   1634         if len(outputs) == 1:
   1635             return outputs[0]

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py in __call__(self, inputs)
   2227         session = get_session()
   2228         updated = session.run(self.outputs + [self.updates_op],
-> 2229                               feed_dict=feed_dict)
   2230         return updated[:len(self.outputs)]
   2231 

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata)
    776     try:
    777       result = self._run(None, fetches, feed_dict, options_ptr,
--> 778                          run_metadata_ptr)
    779       if run_metadata:
    780         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    980     if final_fetches or final_targets:
    981       results = self._do_run(handle, final_targets, final_fetches,
--> 982                              feed_dict_string, options, run_metadata)
    983     else:
    984       results = []

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1030     if handle is None:
   1031       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1032                            target_list, options, run_metadata)
   1033     else:
   1034       return self._do_call(_prun_fn, self._session, handle, feed_dict,

C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args)
   1050         except KeyError:
   1051           pass
-> 1052       raise type(e)(node_def, op, message)
   1053 
   1054   def _extend_graph(self):

InternalError: Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
     [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'dense_1/MatMul', defined at:
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\traitlets\config\application.py", line 658, in launch_instance
    app.start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelapp.py", line 477, in start
    ioloop.IOLoop.instance().start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\ioloop.py", line 177, in start
    super(ZMQIOLoop, self).start()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\zmq\eventloop\zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tornado\stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 235, in dispatch_shell
    handler(stream, idents, msg)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\ipkernel.py", line 196, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\ipykernel\zmqshell.py", line 533, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2683, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2787, in run_ast_nodes
    if self.run_code(code, result):
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\IPython\core\interactiveshell.py", line 2847, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-12-1e7a3b259f23>", line 4, in <module>
    model.add(Dense(10, activation='softmax'))
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\models.py", line 466, in add
    output_tensor = layer(self.outputs[0])
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 585, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\layers\core.py", line 840, in call
    output = K.dot(inputs, self.kernel)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 936, in dot
    out = tf.matmul(x, y)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
    transpose_b=transpose_b, name=name)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\nicol\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(64, 784), b.shape=(784, 10), m=64, n=10, k=784
     [[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](flatten_1/Reshape, dense_1/kernel/read)]]
     [[Node: Assign_3/_84 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_374_Assign_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

在这两种情况下，错误都是由于"InternalError (see above for traceback): Blas GEMM launch failed"。请问如何使Blas GEMM启动？我在3.5版本的python anaconda环境中安装了tensorflow和keras，并安装了所有必需的模块（numpy、pandas、scipy、scikit-learn）。我的Windows 10系统配备了一张可以使用CUDA的NVIDIA显卡。我下载了CUDA和cuDNN。我在Chrome上使用Jupyter笔记本。

有时当我运行代码时，代码会开始运行，但随后崩溃。崩溃后，我无法在jupyter笔记本上做任何事情，过一段时间后就会弹出一个窗口询问我是否要终止页面。以下是崩溃后我得到的图像。!(http://www.hostingpics.net/viewer.php?id=647186tensorflowError.png)

附言：我知道我的问题与这个问题类似： Tensorflow基本示例错误：CUBLAS_STATUS_NOT_INITIALIZED 但在那里它没有被解决，我不确定这个问题是否足够清楚或者是否与我的问题完全相同，所以我用自己的错误消息发布它。 这个问题与以下问题不同： TensorFlow：InternalError：Blas SGEMM启动失败 因为我有一个关于GEMM的问题而不是SGEMM，我的问题既涉及GPU又涉及CPU，并且它没有被这个问题的答案解决。

- Nicolas

---

GPU 3090 上出现了相同的问题。 - GoingMyWay

检查您的GPU使用情况。当我的GPU已满时，我遇到了这个错误。 - Yogesh

Answer 1

这对我在TensorFlow 2.1.0上起作用了（参见：https://www.tensorflow.org/api_docs/python/tf/config/experimental/set_memory_growth）。

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU') 
for device in physical_devices:
    tf.config.experimental.set_memory_growth(device, True)

- socasanta

---

谢谢，这个方法在Tensorflow 2.1.0上对我也有效。 - charelf

请问一下，关于TensorFlow 2.0呢？ - MasterControlProgram

我不得不对代码进行一些修改以适应多GPU机器，但最终发现它可以正常工作。 - Tae-Sung Shin

谢谢，你救了我的一天。我在使用Keras的R语言中出现了“failed to create cublas handle”的错误，但是通过这个方法我解决了它。 - Matteo De Felice

在添加上述命令之前，请确保重新启动笔记本。 - Yacine Rouizi

Answer 2

这只是一个简单的修复，但要找到所有内容真是一场噩梦。

在Windows上，我在Anaconda3\Lib\site-packages\keras中找到了Keras安装文件夹。

参考来源:

https://www.tensorflow.org/guide/using_gpu

https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py

在keras/tensorflow_backend.py文件中查找以下内容，在两个位置都添加config.gpu_options.allow_growth= True。

if _SESSION is None:
            if not os.environ.get('OMP_NUM_THREADS'):
                config = tf.ConfigProto(allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            else:
                num_thread = int(os.environ.get('OMP_NUM_THREADS'))
                config = tf.ConfigProto(intra_op_parallelism_threads=num_thread,
                                        allow_soft_placement=True)
                config.gpu_options.allow_growth=True
            _SESSION = tf.Session(config=config)
        session = _SESSION

- Linda MacPhee-Cobb

---

很好读懂为什么要这样做 :) 对我来说，解决方案是后台没有使用GPU。 - MasterControlProgram

以上行代码在TensorFlow 2.0中不存在。 - Hrushi

Answer 3

确保没有其他正在使用GPU的进程运行。运行nvidia-smi来检查。

来源：@reedwm提出的一个问题.

- liwy

---

例如，当您同时使用PyCharm和Jupyter时！ - Phil

谢谢。之前的答案对我也有用，但我想知道为什么问题突然出现了。看到这个后，我终止了正在运行的进程，问题自动解决了。 - Pritam

另一个例子：在JupyterLab中运行多个内核。仅仅重新启动带有此错误的内核是不够的；我必须先关闭所有其他内核。 - Denziloe

这对我来说是个问题。我不得不杀掉之前正在运行的使用tensorflow-gpu的Jupyter Notebook，才能让我的第二个Notebook代码进行训练。 - Suprateem Banerjee

这是正确的，默认情况下，tf会使用cuda:0，我们可以通过os.environ["CUDA_VISIBLE_DEVICES"] = "7"来指定特定的GPU。 - K. Symbol

这是正确的，默认情况下，tf会使用cuda:0，我们可以通过os.environ["CUDA_VISIBLE_DEVICES"] = "7"来指定特定的GPU - undefined

Answer 4

在导入后添加以下行解决了问题：

configuration = tf.compat.v1.ConfigProto()
configuration.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=configuration)

- burhan rashid

---

@sɐunıɔןɐqɐp，这个解决方案很好用。您想详细解释一下它是做什么的以及为什么有效吗？谢谢。 - user288609

@user288609：其实你应该问这个问题给Burhan Rashid，而不是问我。 - sɐunıɔןɐqɐp

@user288609：可能这是他的来源：https://kobkrit.com/using-allow-growth-memory-option-in-tensorflow-and-keras-dc8c8081bc96 - sɐunıɔןɐqɐp

如果在Tensorflow或Keras环境中没有使用“allow_growth”选项，它会导致图形卡的内存完全分配给该进程。实际上，它可能只需要一小部分内存来运行。这将防止在同一台机器上运行消耗GPU内存的任何新GPU进程。只需在Tensorflow或Keras中启用“allow_growth”设置即可。以下代码可以在Tensorflow中设置allow_growth内存选项。这将增加显卡的利用率，不限制主机机器可容纳的进程数。 - sɐunıɔןɐqɐp

Answer 5

这个回答与Tensorflow密切相关：

有时，Windows上创建Tensorflow会出现失败的情况。

在大多数情况下，使用gpu重新启动笔记本可以解决这个问题。

如果这不起作用，请尝试在代码中添加以下选项后重新启动笔记本。

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.9)

tf.Session(config=tf.ConfigProto(gpu_options=gpu_options,allow_soft_placement=True)

在使用Keras时，我从未遇到过这样的错误。但是尝试重新启动你的笔记本电脑。

- Michael Yadidya

---

代码中缺少一个右括号。 - bit_scientist

很遗憾，下一个版本2中没有GPUOptions()。能否更新一下呢？谢谢！ - MasterControlProgram

Answer 6

我也曾遇到过同样的错误。可能与TensorFlow分配了所有GPU内存的问题有关（参见此处）。但是，那里推荐的解决方法对我不起作用，而且目前还无法通过keras.json或命令行来限制TensorFlow使用GPU内存。将Keras的后端切换到Theano可以解决我的问题（如何操作请参见此处）。

- fotis j

Answer 7

对我而言，使用Python关闭并重新启动进程是可行的。

我尝试了一些方法，但它们都没有起作用。例如，

    os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

我的代码出现了错误，我认为是因为我正在使用较新版本的Keras和Tensorflow。我在网上阅读的很多内容，包括官方Keras教程，都无法正常工作，因为版本冲突。

但是我看到有几篇文章提到了多个Python进程运行的问题。所以我关闭了Jupyter、Anaconda和PyCharm，并重新启动了所有程序。然后这个错误消失了。这可能是适用于你的解决方法，不妨一试。

- Paul J Abernathy

Answer 8

我遇到了完全相同的错误信息。我意识到我的CUDA安装存在问题，具体来说是cuBLAS库存在问题。

您可以通过运行示例程序simpleCUBLAS（它随CUDA安装一起提供，您可能会在CUDA主文件夹中找到它：$CUDA_HOME\samples\7_CUDALibraries\simpleCUBLAS）来检查是否存在相同的问题。

• 如果您没有下载示例或其他方式获得它们，请访问GitHub / 查看NVIDIA网站上的文档。

尝试运行此程序。如果测试失败，则说明您的CUDA安装存在问题。您应该尝试重新安装它。这就是我在这里解决同样问题的方法。

将cublas64_10.dll重命名为cublas64_100.dll可能是一种解决方法。

- Tairone

---

如果有人偶然看到这个：将 cublas64_10.dll 重命名为 cublas64_100.dll 对我起了作用。 - asymmetryFan

Answer 9

当我尝试运行多个服务器来使用模型进行预测时，我遇到了这个问题。由于我没有在训练模型而只是在使用它，因此使用GPU或CPU的差异很小。对于这种特定情况，可以通过“隐藏”GPU来强制Tensorflow使用CPU来避免出现问题。

最初的回答翻译如上。

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"  # Force TF to use only the CPU

- Edgar H

---

非常感谢！经过多次尝试和错误，终于解决了我的问题。 - undefined

Answer 10

我遇到了相同的错误，幸运的是我已经解决了它。 我的错误是：上一次我打开 tensorflow sess = tf.Session()，但我忘记关闭会话。

所以我打开终端，输入命令：

ps -aux | grep program_name

找到进程的PID，然后输入kill命令杀死该进程：

kill -9 PID

好的，GPU已经发布。

- xiaxia wang