Theano: test GPU configuration fails

[colinpriest]

When I run check_blas.py with GPU, I get the following error:

nvcc fatal : Value 'sm_52' is not defined for option 'gpu-architecture'

['nvcc', '-shared', '-O3', '-arch=sm_52', '-m64', '-Xcompiler', '-fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=11b90075e2397c684f9dc0f7276eab8f,-D NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC', '-Xlinker', '-rpath,/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray', '-I/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray', '-I/usr/include', '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include', '-I/usr/include/python2.7', '-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda', '-o', '/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpi_77Ve/c8d7bd33dfef61705c2854dd1f0cb7ce.so', 'mod.cu', '-L/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray', '-L/usr/lib', '-lpython2.7', '-lcudart', '-lcublas', '-lcuda_ndarray']
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/misc/check_blas.py", line 23, in
import theano
File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 98, in
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/tests/test_driver.py", line 32, in test_nvidia_driver1
profile=False)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 266, in function
profile=profile)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 511, in pfunc
on_unused_input=on_unused_input)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1466, in orig_function
defaults)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1324, in create
input_storage=input_storage_lists)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 519, in make_thunk
output_storage=output_storage)[:3]
File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 897, in make_all
no_recycling))
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/init.py", line 259, in make_thunk
compute_map, no_recycling)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 739, in make_thunk
output_storage=node_output_storage)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1073, in make_thunk
keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1015, in compile
keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1442, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1076, in module_from_key
module = lnk.compile_cmodule(location)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1354, in compile_cmodule
preargs=preargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/nvcc_compiler.py", line 423, in compile_str
'for cmd', ' '.join(cmd))
Exception: ('The following error happened while compiling the node', GpuCAReduce{add}{1}(

[nouiz]

The problem is that your GPU is too recent for your version of cuda version
and cuda driver.

You can update your driver and cuda version. A work around it to force the
use of older GPU specific optimization. This can be done with one of those
Theano flag:

nvcc.flags=-arch=sm_50
nvcc.flags=-arch=sm_30

On Tue, Jan 5, 2016 at 12:24 PM, Colin Priest [email protected]
wrote:

When I run check_blas.py with GPU, I get the following error:

nvcc fatal : Value 'sm_52' is not defined for option 'gpu-architecture'

['nvcc', '-shared', '-O3', '-arch=sm_52', '-m64', '-Xcompiler',
'-fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=11b90075e2397c684f9dc0f7276eab8f,-D
NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC', '-Xlinker',
'-rpath,/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray',
'-I/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray',
'-I/usr/include',
'-I/usr/local/lib/python2.7/dist-packages/numpy/core/include',
'-I/usr/include/python2.7',
'-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda', '-o',
'/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpi_77Ve/c8d7bd33dfef61705c2854dd1f0cb7ce.so',
'mod.cu',
'-L/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray',
'-L/usr/lib', '-lpython2.7', '-lcudart', '-lcublas', ' -lcuda_ndarray']
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/theano/misc/check_blas.py",
line 23, in
import theano
File "/usr/local/lib/python2.7/dist-packages/theano/_init_.py", line 98,
in
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File
"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/tests/test_driver.py",
line 32, in test_nvidia_driver1
profile=False)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py",
line 266, in function
profile=profile)
File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py",
line 511, in pfunc
on_unused_input=on_unused_input)
File
"/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py",
line 1466, in orig_function
defaults)
File
"/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py",
line 1324, in create
input_storage=input_storage_lists)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
519, in make_thunk
output_storage=output_storage)[:3]
File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 897,
in make_all
no_recycling))
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/_init_.py",
line 259, in make_thunk
compute_map, no_recycling)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 739,
in make_thunk
output_storage=node_output_storage)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1073,
in make_thunk
keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1015,
in _compile_
keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1442,
in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line
1076, in module_from_key
module = lnk.compile_cmodule(location)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1354,
in compile_cmodule
preargs=preargs)
File
"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/nvcc_compiler.py",
line 423, in compile_str
'for cmd', ' '.join(cmd))
Exception: ('The following error happened while compiling the node',
GpuCAReduce{add}{1}(), '\n', 'nvcc return status', 1, 'for cmd', 'nvcc
-shared -O3 -arch=sm_52 -m64 -Xcompiler
-fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-Wno-write-strings,-DCUDA_NDARRAY_CUH=11b90075e2397c684f9dc0f7276eab8f,-D
NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC -Xlinker
-rpath,/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray
-I/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/cuda_ndarray
-I/usr/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include
-I/usr/include/python2.7
-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -o
/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-2.7.10-64/tmpi_77Ve/c8d7bd33dfef61705c2854dd1f0cb7ce.so
mod.cu
-L/home/colin/.theano/compiledir_Linux-4.2--generic-x86_64-with-Ubuntu-15.10-wily-x86_64-
2.7.10-64/cuda_ndarray -L/usr/lib -lpython2.7 -lcudart -lcublas
-lcuda_ndarray', '[GpuCAReduce{add}{1}()]')

—
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/3835.

[colinpriest]

Sorry but I'm a newby. Where do I set the nvcc.flags ?

[colinpriest]

I solved the problem. There was more than one version of CUDA on my computer.

sudo rm -R -f /usr/lib/nvidia-cuda-toolkit
sudo rm /usr/bin/nvcc

[basirshariat]

@colinpriest you should add the nvcc.flags in your .theanorc file like this:
nvcc.flags=-arch=sm_50
or in your .bashrc/bash_profile file like this:
export THEANO_FLAGS='nvcc.flags=-arch=sm_50'