Keras: Blas GEMM launch failed

[sjsy]

Hi,
My installation of keras worked fine, but now it display the following error:
Using TensorFlow backend.

---

Layer (type) Output Shape Param #

dense_1 (Dense) (None, 512) 62464

---

dense_2 (Dense) (None, 512) 262656

---

dense_3 (Dense) (None, 512) 262656

---

dense_4 (Dense) (None, 512) 262656

---

dense_5 (Dense) (None, 121) 62073

Total params: 912,505
Trainable params: 912,505
Non-trainable params: 0

---

C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\backend\tensorflow_backend.py:2252: UserWarning: Expected no kwargs, you passed 1
kwargs passed to function are ignored with Tensorflow backend
warnings.warn('\n'.join(msg))
Train on 1860480 samples, validate on 328320 samples
Epoch 1/200
2017-05-30 12:02:16.753478: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.754291: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.754551: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.754804: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.755067: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.755326: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.755579: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:16.755831: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-05-30 12:02:17.079085: I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX 1080
major: 6 minor: 1 memoryClockRate (GHz) 1.835
pciBusID 0000:01:00.0
Total memory: 8.00GiB
Free memory: 6.66GiB
2017-05-30 12:02:17.079335: I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:908] DMA: 0
2017-05-30 12:02:17.079452: I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:918] 0: Y
2017-05-30 12:02:17.079824: I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\common_runtime\gpu\gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0)
2017-05-30 12:02:20.773830: E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_blas.cc:365] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2017-05-30 12:02:20.774021: W c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\stream.cc:1550] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
return fn(*args)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
status, run_metadata)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\contextlib.py", line 66, in __exit__
next(self.gen)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(128, 121), b.shape=(121, 512), m=128, n=512, k=121
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_dense_1_input_0/_9, dense_1/kernel/read)]]
[[Node: mul_1/_39 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_797_mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:/Users/user/c o d e/my_tez/PycharmProjects/urban100/source2.py", line 140, in
validation_data=(x_test, y_test))
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\models.py", line 868, in fit
initial_epoch=initial_epoch)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\engine\training.py", line 1501, in fit
initial_epoch=initial_epoch)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\engine\training.py", line 1155, in _fit_loop
outs = f(ins_batch)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\backend\tensorflow_backend.py", line 2231, in __call__
feed_dict=feed_dict)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
run_metadata_ptr)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(128, 121), b.shape=(121, 512), m=128, n=512, k=121
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_dense_1_input_0/_9, dense_1/kernel/read)]]
[[Node: mul_1/_39 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_797_mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'dense_1/MatMul', defined at:
File "D:/Users/user/c o d e/my_tez/PycharmProjects/urban100/source2.py", line 111, in
model.add(Dense(512, activation='relu', input_dim=121))
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\models.py", line 442, in add
layer(x)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\engine\topology.py", line 585, in __call__
output = self.call(inputs, **kwargs)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\layers\core.py", line 840, in call
output = K.dot(inputs, self.kernel)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\keras-2.0.4-py3.5.egg\keras\backend\tensorflow_backend.py", line 936, in dot
out = tf.matmul(x, y)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1801, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 1263, in _mat_mul
transpose_b=transpose_b, name=name)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
op_def=op_def)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "C:\Users\user\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
self._traceback = _extract_stack()

InternalError (see above for traceback): Blas GEMM launch failed : a.shape=(128, 121), b.shape=(121, 512), m=128, n=512, k=121
[[Node: dense_1/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/gpu:0"](_recv_dense_1_input_0/_9, dense_1/kernel/read)]]
[[Node: mul_1/_39 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_797_mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Process finished with exit code 1

[sjsy]

I update anaconda using the following command
conda update --all
then the system restarted due to power failure (!). And now the problem resolved.
Thanks.

[Kajiyu]

I had the same problem and solved by updating all conda-packages, thanks.

[davidmmiller7]

I had a very similar problem, the only solution that has worked for me so far is:
https://stackoverflow.com/a/52132383/4584857

[JuanMost]

Thanks a lot sjsy !.
conda update --all , and problem soved in 2020