Theano: CUDA is installed, but device gpu0 is not available

Created on 17 Apr 2016 · 18Comments · Source: Theano/Theano

Hey friends, I'd like to ask for some help,

I recently bought a Nvidia G 970 card, and would like to try theano on this new GPU.

I think I've followed everything stated on theano page (CUDA, enviroment, config, etc.) and also some other blogs.

But, it is still not working.

I am running on:
os = ubuntu 16.04; theano = 0.8.1; language = python3.5; gpu = Nvidia GTX 970;

And I have tested CUDA using its own samples, no problem, the tests all PASS.

But when use theano:
I always get the error msg (no matter on theano.test(), or my own pgm):

['/usr/local/cuda-7.5/bin/nvcc', '-shared', '-O3', '-use_fast_math', '-m64', '-Xcompiler', '-DCUDA_NDARRAY_CUH=mc72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden', '-Xlinker', '-rpath,/home/aaron/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/cuda_ndarray', '-Xlinker', '-rpath,/usr/local/cuda-7.5/lib', '-Xlinker', '-rpath,/usr/local/cuda-7.5/lib64', '-I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/theano/sandbox/cuda', '-I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/numpy/core/include', '-I/usr/include/python3.5m', '-I/home/aaron/Git/.virtualenvs/ts/include/python3.5m', '-I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/theano/gof', '-o', '/home/aaron/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/cuda_ndarray/cuda_ndarray.so', 'mod.cu', '-L/usr/lib', '-lcublas', '-lpython3.5m', '-lcudart']
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 1, 'for cmd', '/usr/local/cuda-7.5/bin/nvcc -shared -O3 -use_fast_math -m64 -Xcompiler -DCUDA_NDARRAY_CUH=mc72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden -Xlinker -rpath,/home/aaron/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/cuda_ndarray -Xlinker -rpath,/usr/local/cuda-7.5/lib -Xlinker -rpath,/usr/local/cuda-7.5/lib64 -I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/theano/sandbox/cuda -I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/numpy/core/include -I/usr/include/python3.5m -I/home/aaron/Git/.virtualenvs/ts/include/python3.5m -I/home/aaron/Git/.virtualenvs/ts/lib/python3.5/site-packages/theano/gof -o /home/aaron/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-3.5.1+-64/cuda_ndarray/cuda_ndarray.so mod.cu -L/usr/lib -lcublas -lpython3.5m -lcudart')
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not available  (error: cuda unavailable)

What should I do?

Thx

Source

sjhddh

Most helpful comment

Dear All,

I have same problem and I solved it with the following task:

Add environment variable:

sudo nano /etc/profile

Add following to the file:

PATH=/usr/local/cuda/bin:$PATH
export PATH

Then execute :

source /etc/profile

Then you can import keras or theano library in python without any problem.
Enjoy!

MinaRe on 15 Nov 2016

🎉7 ❤4 👍4

All 18 comments

Hi,
It seems that CUDAToolkit was not installed properly.
run nvidia-smi and check the output for gpu0
Additionally, run theano-cache clear and theano-cache purge.
Also, check for the version of g++ and nvcc and check from the CUDA docs if nvcc supports the version of g++ available.

More info at:

https://groups.google.com/forum/#!topic/theano-users/-N-W6GKL4eU
https://groups.google.com/forum/#!topic/theano-users/xW9jmHzOwp0
https://github.com/Theano/libgpuarray/issues/19

Hope that helps!

gokul-uf on 17 Apr 2016

👍1

@gokul-uf Thank you very much

I checked nvidia-smi, I think there is no problem"

| NVIDIA-SMI 361.42     Driver Version: 361.42         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     Off  | 0000:01:00.0      On |                  N/A |
| 22%   30C    P8    18W / 151W |    257MiB /  4092MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0       878    G   /usr/lib/xorg/Xorg                             135MiB |
|    0      1275    G   compiz                                          44MiB |
|    0      1774    G   ...s-passed-by-fd --v8-snapshot-passed-by-fd    57MiB |
+-----------------------------------------------------------------------------+

And, I also ran theano-cache and theano-cache purge before.

The versions for other things:
g++ (Ubuntu 5.3.1-14ubuntu2) 5.3.1 20160413
nvcc: Cuda compilation tools, release 7.5, V7.5.17

According to one of the google discussion, it might be a unsupported g++ version?
But that post is about a year ago, does it still fit?

Looking for wards to heard more,
Cheers!

sjhddh on 17 Apr 2016

Yes, it's probably because of the compiler. nvcc 7.5 requires g++ 4.8 or 4.9

abergeron on 17 Apr 2016

Ahh, I've re-linked g++ and gcc to 4.9 version. But still....

sjhddh on 17 Apr 2016

You can change your gcc version to something that nvcc support. Google for that, there is frequently doc about this. Something it is simple as asking your package manager to install an older gcc.

nouiz on 18 Apr 2016

Thank you @nouiz

But, as I said in the previous reply, I tried that. i.e. 4.8 / 4.9. But the situation is still not changed..

sjhddh on 18 Apr 2016

Try NVIDIA sample to make sure that nvidia is working well in your setup.
Ubuntu 16.04 is very new and probably they don't support it. If that is the
case, we can't help. If those sample work, then they support it.

On Mon, Apr 18, 2016 at 9:30 AM, Aaron J. Sun [email protected]
wrote:

well, as I said in the previous reply, I tried that. i.e. 4.8 / 4.9. But
the situation is still not changed..

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/Theano/Theano/issues/4384#issuecomment-211379143

nouiz on 19 Apr 2016

Please ask for support on theano-dev or StackOverflow. There, please provide the _whole_ error message.
We try to keep the tickets only for bug reports and feature requests.

lamblin on 19 Apr 2016

try this
sudo ldconfig /usr/local/cuda-7.5/lib64

xiyuanHou on 5 Aug 2016

👍10

Hello, I am having the same problem,
my output is like this, any suggestion on what is going on?
screenshot from 2016-09-15 16-58-18

DanlanChen on 15 Sep 2016

This is a very old GPUs. It isn't supported by newer driver and CUDA
version. I'm not sure the current gpu back-end in Theano support it in all
cases. Also, it is a low end GPUs, so it won't give you any speed up.

if you still want to make it work, make sure CUDA examples work. Without
this, Theano can't work.

But personnaly, I htink it is wasted time to try to make Theano run on that
GPU.

On Thu, Sep 15, 2016 at 3:57 AM, Danlan Chen [email protected]
wrote:

Hello, I am having the same problem,
my output is like this, any suggestion on what is going on?
Thu Sep 15 16:55:24 2016

+------------------------------------------------------+

| NVIDIA-SMI 4.304... Driver Version: 304.131 |

|-------------------------------+----------------------+----
------------------+
| GPU Name | Bus-Id Disp. | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+====
==================|
| 0 NVS 300 | 0000:05:00.0 N/A | N/A |
| N/A 68C N/A N/A / N/A | 53% 269MB / 511MB | N/A Default |
+-------------------------------+----------------------+----
------------------+

+-----------------------------------------------------------
------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|===========================================================
==================|
| 0 Not Supported |
+-----------------------------------------------------------
------------------+

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/4384#issuecomment-247262050, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-wbcxOGlOZisFPsP_-0322PGsXXQks5qqPp6gaJpZM4IJDUD
.

nouiz on 15 Sep 2016

I had the same issue and was able to solve my issue by doing two things:

Install gcc 4.9 and linking/usr/bin/gccto /usr/bin/gcc-4.9 as well as /usr/bin/g++ to/usr/bin/g++-4.9
Adding this flag to THEANO_FLAGS='-D_FORCE_INLINES' since apparently a bug in glibc 2.23 causes this issue

also see:
https://github.com/Theano/Theano/issues/4430
and
https://github.com/Theano/Theano/pull/4369

richard-vogl on 18 Sep 2016

❤3 👍2

Dear All,

I have same problem and I solved it with the following task:

Add environment variable:

sudo nano /etc/profile

Add following to the file:

PATH=/usr/local/cuda/bin:$PATH
export PATH

Then execute :

source /etc/profile

Then you can import keras or theano library in python without any problem.
Enjoy!

MinaRe on 15 Nov 2016

🎉7 ❤4 👍4

@MinaRe doesn't work for me. same error msg as before

ShuaiW on 1 Dec 2016

👍1

Email the information on theano-user to get user help.

On Thu, Dec 1, 2016 at 12:45 PM, Shuai Wang notifications@github.com
wrote:

@MinaRe https://github.com/MinaRe doesn't work for me

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/4384#issuecomment-264242066, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC--eDfEKvIoiOLoqOBvX_YBTAki-Iks5rDwe7gaJpZM4IJDUD
.

nouiz on 1 Dec 2016

Same problem. Any solution to this issue?

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+