Keras-yolo3: Blas SGEMM launch failed

[MinghuiJ]

when I run "python yolo_video.py --image", it loads model successfully. But then I input the name of test picture, error occurs as follows:

---

Input image filename:1.jpg
(416, 416, 3)
2019-03-07 14:23:30.749652: E tensorflow/stream_executor/cuda/cuda_blas.cc:647] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu, conv2d_3/kernel/read)]]
[[Node: boolean_mask_29/GatherV2/_3363 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6054_boolean_mask_29/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

---

My GPU is RTX2080Ti 8G*2, tensorflow-gpu 1.9.0, keras 2.2.4.
I feel so annoyed by this problem, who can tell me what happend and how to solve it?

[zhoujungis]

The same problem as you, can you tell me what happend and how to solve it?

[Ifeng96]

I have met it many times...how to solve it?

[Ifeng96]

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

[MinghuiJ]

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

Thank you so much. A few days ago, I run the some code with two 1080Ti GPUs, it works. So I guess there might be some problems in CUDA support in 2080Ti GPU as you said :)

[MinghuiJ]

The same problem as you, can you tell me what happend and how to solve it?

The comments above may help you~

[swy755565055]

you are right! Thanks

[CforU]

Thank u so much!

[ShuttleDDD]

With those patches installed, my RTX2070 on win10 is still having the same issue.

[hmhtb]

With those patches installed, my RTX2070 on win10 is still having the same issue.

I have the same GPU RTX2070 and having the same problem in Ubunto 16.04!
Cuda 9 + 4 extra patches

Tensorflow 1.8

[Ifeng96]

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found

直接sudo sh cuda_9.0.176_384.81_linux.run会报错？run不需要的

[Angel0003]

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

Thank you so much. A few days ago, I run the some code with two 1080Ti GPUs, it works. So I guess there might be some problems in CUDA support in 2080Ti GPU as you said :)
hi，when i execute the command：Run sudo sh cuda_9.0.176_384.81_linux.run,it shows an error

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found
直接sudo sh cuda_9.0.176_384.81_linux.run会报错？run不需要的

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

[Ifeng96]

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

The base installer is available for download below.
There are 4 patches available. These patches require the base installer to be installed first.
CUDA下载界面的提示，这是额外的，可能是为了兼容最新出的显卡开发的

[Angel0003]

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

The base installer is available for download below.
There are 4 patches available. These patches require the base installer to be installed first.
CUDA下载界面的提示，这是额外的，可能是为了兼容最新出的显卡开发的

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

[Ifeng96]

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛

[Angel0003]

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛
Input image filename:dog.jpg
(416, 416, 3)
2019-07-12 11:28:11.879346: E tensorflow/stream_executor/cuda/cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-12 11:28:11.879368: W tensorflow/stream_executor/stream.cc:2001] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 16, in detect_img
r_image = yolo.detect_image(image)
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 124, in detect_image
K.learning_phase(): 0
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'conv2d_3/convolution', defined at:
File "yolo_video.py", line 75, in
detect_img(YOLO(*vars(FLAGS)))
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 45, in __init__
self.boxes, self.scores, self.classes = self.generate()
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 70, in generate
self.yolo_model = load_model(model_path, compile=False)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
printable_module_name='layer')
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
process_node(layer, node_data)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
layer(unpack_singleton(input_tensors), *kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
output = self.call(inputs, **kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 171, in call
dilation_rate=self.dilation_rate)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3650, in conv2d
data_format=tf_data_format)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution
return op(input, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 868, in __call__
return self.conv_op(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 520, in __call__
return self.call(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 204, in __call__
name=self.name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
安装完补丁之后还是这个错，知道是啥错吗？装了一遍补丁貌似没用，搞一天啦，这啥问题呢

[tanf]

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛
Input image filename:dog.jpg
(416, 416, 3)
2019-07-12 11:28:11.879346: E tensorflow/stream_executor/cuda/cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-12 11:28:11.879368: W tensorflow/stream_executor/stream.cc:2001] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 16, in detect_img
r_image = yolo.detect_image(image)
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 124, in detect_image
K.learning_phase(): 0
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'conv2d_3/convolution', defined at:
File "yolo_video.py", line 75, in
detect_img(YOLO(vars(FLAGS)))
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 45, in *init
self.boxes, self.scores, self.classes = self.generate()
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 70, in generate
self.yolo_model = load_model(model_path, compile=False)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
process_node(layer, node_data)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
layer(unpack_singleton(input_tensors), kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in *call
output = self.call(inputs, kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 171, in call
dilation_rate=self.dilation_rate)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3650, in conv2d
data_format=tf_data_format)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution
return op(input, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 868, in **call
return self.conv_op(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 520, in call
return self.call(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 204, in call
name=self.name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InternalError (see above for traceback): Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
安装完补丁之后还是这个错，知道是啥错吗？装了一遍补丁貌似没用，搞一天啦，这啥问题呢

有解决吗 ？跟你一样的问题 docker环境TensorFlow1.8.0 cuda9.0 keras2.2.4

[Xiaokeai18]

I used cude9.0 with tf1.8 at first, and then I fixed this problem by changing it to cuda9.2 .
If you use anaconda, just type in conda install cudatoolkit=9.2
Hope this would help you.

[JoeyforJoy]

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

It does work for me, thanks a lot

[Mrbatman-YJ]

Input image filename:images/img00030.jpg
(416, 416, 3)
Traceback (most recent call last):
File "yolo_video.py", line 73, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 15, in detect_img
r_image = yolo.detect_image(image)
File "E:\YOLOV3\K-yolo3\yolo.py", line 125, in detect_image
self.input_image_shape: [image.size[1], image.size[0]],
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 705, in __hash__
raise TypeError("Tensor is unhashable if Tensor equality is enabled. "
TypeError: Tensor is unhashable if Tensor equality is enabled. Instead, use tensor.experimental_ref() as the key.

help me

[BryanBo-Cao]

Try reinstalling TensorFlow to 1.6.0 version mentioned in README.md?

pip install tensorflow==1.6.0

[Resul-Tech]

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

thank you so much

[gmt710]

I've tried tensorflow==1.12.0 (cpu version). It's ok.