Theano: libgpuarray / cudnn issues

[RyanHope]

I once had a working cuda/theano/libgpuarry setup working on windows 10 but since a reinstall I have not been able to get a python setup working with cuda and libgpuarray. Using both Anaconda's python and WinPython, when using the old backend, the common theano gpu test works.

python D:\test.py
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release.  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 1060 3GB (CNMeM is enabled with initial size: 10.0% of memory, cuDNN 5110)
[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.402000 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

But when I switch to using the libgpuarray backend cudnn can no longer be found and gpu is not used.

python D:\test.py
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
c:\users\ryanhope\appdata\local\temp\try_flags_zq9bx6.c:4:19: fatal error: cudnn.h: No such file or directory
compilation terminated.

Mapped name None to device cuda: GeForce GTX 1060 3GB (0000:01:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.433000 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the cpu

FWIW, I can import pygpu no problem but pygpu.test() eventually crashes.

[abergeron]

If you build the packages yourself, you must make sure that you use the same compiler for the library as the one that was used for the python binary otherwise you can encounter weird crashes.

As for why the header is not found you can try setting the Theano flags dnn.include_path and dnn.library_path to the path where you installed the files. The difference is probably that the old backend includes the cuda folders by default and the new one doesn't.

For the anaconda python you can try the release candidate packages with conda install -c rdonnelly pygpu.

[RyanHope]

I cleaned up my system of any trace of libgpuarray and pygpu and installed them with conda. That at least allowed all of the pygpu tests to pass. However, things are still not quite right. With no other changes to my .theanorc I still had the issue of the cudnn headers not being found. I used your suggestion to add the dnn.include/library options to my config but on loading theano a new header could not be found. That header was back in the cuda include directory so I realised this issue must be related to the spaces in the path name. I copied the whole cuda directory to a new location without spaces and updated my .theanorc and theano was able to load cleanly.

Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jun 29 2016, 11:07:13) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import theano
Using cuDNN version 5110 on context None
Preallocating 307/3072 Mb (0.100000) on cuda
Mapped name None to device cuda: GeForce GTX 1060 3GB (0000:01:00.0)
>>>

While better this does not look perfect. I have "device = cuda" in my .theanorc so why is libgpuarray say None? At this point I decided to run the cpu/gpu test script and got this error:

Exception: ('The following error happened while compiling the node', GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), '\n', 'Compilation failed (return status=1): C:\\Users\\RyanHope\\AppData\\Local\\Theano\\compiledir_Windows-10-10.0.14393-Intel64_Family_6_Model_42_Stepping_7_GenuineIntel-2.7.12-64\\tmpnbs0d4\\mod.cpp:4:28: fatal error: gpuarray/array.h: No such file or directory\r. compilation terminated.\r. ', '[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>)]')

I fixed this by copying the gpuarray include folder ino the pygpy directory in site-packages. This then led to the gpuarray lib not being found:

Exception: ('The following error happened while compiling the node', GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), '\n', 'Compilation failed (return status=1): D:/mingw-w64/x86_64-5.4.0-posix-seh-rt_v5-rev0/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/5.4.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lgpuarray\r. collect2.exe: error: ld returned 1 exit status\r. ', '[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>)]')

Since none of its installed locations were on any of the linker paths I copied the gpuarray.lib to the base Anaconda2 folder. That allowed the test script to run without any errors but it did not run on the gpu.

[ozancaglayan]

There is also another issue regarding libgpuarray backend + cuDNN.

I had CuDNN installed under /opt/CUDNN/cudnn-v5.1 and CUDA under /opt/cuda-7.5. The CUDA installations are clean CUDA SDK, without CUDNN libraries or headers copied inside.

The old backend works like a charm with the following configuration:

[cuda]
root = /opt/cuda-7.5

[dnn]
library_path = /opt/CUDNN/cudnn-v5.1/lib64
include_path = /opt/CUDNN/cudnn-v5.1/include

but the libgpuarray fails probably because it expects CUDNN libs and headers inside CUDA installation.

THEANO_FLAGS = device=cuda0
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
In file included from /tmp/try_flags_IeoZNM.c:4:0:
/opt/CUDNN/cudnn-v5.1/include/cudnn.h:63:26: erreur fatale: driver_types.h : Aucun fichier ou dossier de ce type

I think changing backend should not necessitate moving/copying files around and the behavior should be identical.

Update: I think this may be because of the fact that old backend compiles the test code with nvcc while the new uses gcc. nvcc probably has builtin include_path tied to the SDK folder.

[ghost]

I run into the same problem,
"Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
c:\users\ryanhope\appdata\local\temp\try_flags_zq9bx6.c:4:19: fatal error: cudnn.h: No such file or directory"
but is ok with gpu backend

[valtron]

I got that same error, worked around it by copying cudnn into the cuda directory. See here for a guide.

Now I'm running into a ton of errors during compilation:

%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:698:16: error: 'input_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type

                ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:671:29: note: in definition of macro 'APPLY_SPECIFIC'
 #undef ITEMSIZE_INPUT_5
                             ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:699:16: error: 'output_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type
 double __DUMMY_11;
                ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:671:29: note: in definition of macro 'APPLY_SPECIFIC'
 #undef ITEMSIZE_INPUT_5
                             ^
%HOME%\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_60_Stepping_3_GenuineIntel-3.6.0-64\tmpkzqaxhi8\mod.cpp:700:16: error: 'kerns_node_m150592bf75eb7b96b1e07943b9cb84a9_0' does not name a type
 __label_13:
                ^

... etc ...

The line numbers of the errors seem to be wrong, because it points to parts of the code that have nothing to do with the error. Does anyone know how to fix this?

[abergeron]

PR #5544 fixes the issues with empty path and/or spaces in the path.

For the other issues, the best solution would be to set LIBRARY_PATH and CPLUS_INCLUDE_PATH in the environment to the required directories if there are more than one to allow the compile to succeed.

[ghost]

I worked around the error caused by spaces in the path. But python.exe crashed while import theano

[nouiz]

What output do you have?

[ghost]

just a typical window pop up saying that python.exe has stopped working. no further information and no other output

[lamblin]

@zero2hero2020 This one usually indicates a mismatch between the compiler of Python and one of its loaded modules (maybe one compiled by Theano), or at least between the runtime they are linked with.

Which version of GCC do you have? Is it the one provided by Anaconda as a package?

[RyanHope]

This problem is not unique to Windows. I just setup a fresh Ubuntu 16.04 box. Fetched and built latest libgpuarray which passes tests but within Theano the new backend does not work. The thing that worries me is this:

Using cuDNN version 5110 on context None
Mapped name None to device cuda0: GeForce GTX 1060 3GB (0000:01:00.0)

I have device = cuda in my .theanorc, why does this pick up my device as None?

[ghost]

@lamblin I am using mingw 4.7 from Anaconda package

[RyanHope]

Why is this marked fixed? My problem has not yet been addressed!

[nouiz]

It got closed as a PR that was just merged had a comment that told it fixed this issue. The PR was merged 16m ago.

The device "none" in the output just mean that it is an unnamed device. The Theano flag "contexts" (or "context", I forgot) allow to give name to context to help multi-GPU in the same process. This isn't a ready to use. But the none isn't a problem, it mean the device was initialized.

In the description of this issue, there is output

[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.433000 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the cpu

The first line write GpuElemwise. This mean it is really executed on the GPU. What is your file test.py? I think the problem is that the file don't detect correctly if it run on CPU or GPU and print the bad think in the last line.

[ozancaglayan]

@abergeron yes CPLUS_INCLUDE_PATH fixed the issue but still I find a little bit weird that with the same .theanorc old backend works but new one requires this env variable.

[balcilar]

I had the same problem and I could solve this problem with merging cudnn and CUDA directory. After I defined dnn.include_path and dnn.library_path with CUDA include and CUDA lib/x64 directories.
I use CUDA8 and Cudnn5.1, i tried my solution on miniconda2 with Windows and Ubuntu 16.04.

Here is my steps

• For windows, I copied CUDA8 directory in my home directory. Now my CUDA directory is C:Users\LFC\CUDA\v8.0 for ubuntu it is not need

• I downloaded cudnn 5.1 and I extracted zip file into the CUDA directory. For windows I extracted into the C:Users\LFC\CUDA\v8.0, for ubuntu /usr/local/cuda-8.0/

• I defined dnn include and lib directory in theano config file. Here is my configs for windows. You can define appropriate path for ubuntu easily.

[global]
device = cuda
floatX = float32
[dnn]
enabled = True
include_path=C:/Users/LFC/CUDA/v8.0/include
library_path=C:/Users/LFC/CUDA/v8.0/lib/x64

For testing you should type following command in console on both win and ubuntu machine.
python -c "import theano"

After this command, you should take some message about cudnn version and your gpu device.

The second test can be performed by pygpu.test() . You should type following commands on windows machine.
set DEVICE=cuda
python -c "import pygpu;pygpu.test()"

on ubuntu machine you should type following command.
DEVICE=cuda python -c 'import pygpu;pygpu.test()'

[deltaz0]

@zero2hero2020 I know this is months late but I had the same issue as you and did not see it resolved here yet.

The issue being: Windows 10, theano 0.9.0, new backend, after enabling cudnn; python -c "import theano" crashes with generic "python.exe has stopped working" error.

It turned out it was caused by an entry in my path. I was able to resolve it by removing the entry C:\Anaconda2\Scripts and then deleting the directory C:\Users\username\AppData\Local\Theano.

Heh, actually that was overkill, don't remove that from the path. A better solution is to change the file C:\Anaconda2\Scripts\g++.bat to say:

@echo off
"%~f0\..\..\pkgs\m2w64-gcc-5.3.0.6\Library\mingw-w64\bin\g++.exe" %*

That resolved it for me. There's probably a better way than changing the default g++ for Anaconda but it works.

[thomasklau]

For future users that find this thread, what specifically fixed it for me was:
import os
os.environ['THEANO_FLAGS'] = 'device=cuda,floatX=float32'
os.environ['CPLUS_INCLUDE_PATH'] = '/usr/local/cuda/include'

import keras

/usr/local/cuda is the default install location on Ubuntu 16.04

[github-kk]

In my case, CPLUS_INCLUDE_PATH solves the problem.
Ubuntu 16.04, running theano 0.9.0 under keras 2.0.5. export CPATH including cudnn/include/, cuda/include, include of gpuarray.

And export CPLUS_INCLUDE_PATH = $CPATH.
I can run python lstm_benchmark.py in keras/examples under the backend theano.

[nouiz]

Note, there is the Theano flags: dnn.include_path and dnn.library_path that
can be used to help on that.

We tried to document it more.

On Thu, Jun 29, 2017 at 8:23 PM hash map [e-mail] wrote:

In my case, CPLUS_INCLUDE_PATH solves the problem.
Ubuntu 16.04, running theano 0.9.0 under keras 2.0.5. export CPATH
including cudnn/include/, cuda/include, include of gpuarray. And export
CPLUS_INCLUDE_PATH = $CPATH. I can run python lstm_benchmark.py in
keras/examples under the backend theano.

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/5463#issuecomment-312142268, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-2oMSmlm0ZTUUw9j9ebAuYJcb5sbks5sJD_7gaJpZM4LwQLp
.

[loretoparisi]

@nouiz Was this issue has been solved?

I'm running the gpu test from theano docs and get

root@1ddfb2570bdb:~# python gpu_test.py 
Using cuDNN version 5110 on context None
Mapped name None to device cuda0: GRID K520 (0000:00:03.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float64, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.837682 seconds
Result is [ 1.23178032  1.61879341  1.52278065 ...,  2.20771815  2.29967753
  1.62323285]
Used the gpu

and

root@1ddfb2570bdb:~# python -c "import theano; print( theano.__version__)"
Using cuDNN version 5110 on context None
Mapped name None to device cuda0: GRID K520 (0000:00:03.0)
0.9.0

The libgpu array it works (installed from sources):

root@1ddfb2570bdb:~# export DEVICE="cuda0"
root@1ddfb2570bdb:~# python -c "import pygpu;pygpu.test()"
pygpu is installed in /usr/local/lib/python2.7/dist-packages/pygpu-0.6.9-py2.7-linux-x86_64.egg/pygpu
NumPy version 1.13.1
NumPy relaxed strides checking option: True
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
nose version 1.3.7
*** Testing for GRID K520
...............................................................................................................................................................
Ran 7290 tests in 294.530s

OK

[ankushbbbr]

I am getting a similar issue with libgpuarray. Tried removing the spaces from the path, still getting the same error.
I'm using the following THEANO_FLAGS
THEANO_FLAGS=floatX=float32,device=cuda0,optimizer_including=cudnn,gpuarray.preallocate=0.8,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic,dnn.include_path=F:\Softwares\fast_ai\CUDA\v8.0\include,dnn.library_path=F:\Softwares\fast_ai\CUDA\v8.0\lib\x64

Error Message -
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'C:\Users\ANKUSH1\AppData\Local\Temp\try_flags_5d3sdl5t.c:4:19: fatal error: cudnn.h: No such file or directory\r\ncompilation terminated.\r\n'
Preallocating 3276/4096 Mb (0.800000) on cuda0
Mapped name None to device cuda0: GeForce GTX 850M (0000:0A:00.0)
Traceback (most recent call last):
File "cpu_gpu_test.py", line 10, in
f = function([], tensor.exp(x))
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\compile\function.py", line 326, in function
output_keys=output_keys)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\compile\pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\compile\function_module.py", line 1794, in orig_function
output_keys=output_keys).create(
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\compile\function_module.py", line 1474, in __init__
optimizer_profile = optimizer(fgraph)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 98, in __call__
return self.optimize(fgraph)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 87, in optimize
ret = self.apply(fgraph, args, *kwargs)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 87, in optimize
ret = self.apply(fgraph, args, *kwargs)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gof\opt.py", line 87, in optimize
ret = self.apply(fgraph, args, *kwargs)
File "C:Users\ankushbabbar\Miniconda2\envs\dlwin36\lib\site-packages\theano\gpuarray\dnn.py", line 3081, in apply
dnn_available.msg)
AssertionError: cuDNN optimization was enabled, but Theano was not able to use it for context None. We got this error:
cannot compile with cuDNN. We got this error:
b'C:\Users\ANKUSH1\AppData\Local\Temp\try_flags_5d3sdl5t.c:4:19: fatal error: cudnn.h: No such file or directory\r\ncompilation terminated.\r\n'

[coranholmes]

ubuntu 14.04, Theano 1.0.1, gpuarray 0.7.5 have the same issue here. configuring CPLUS_INCLUDE_PATH saves my life!

[wqp89324]

@coranholmes Is the CPLUS_INCLUDE_PATH solution specific to ubuntu and not Windows?