Theano 1.0.3 issue with pygpu: ERROR (theano.gpuarray): Could not initialize pygpu, support disabled

Created on 21 Jan 2019 · 13Comments · Source: Theano/Theano

Hey guys i am pretty new to Deep Learning and wanted to use Theano to run some neural nets using my gpu. However while trying to run theano with the .theanorc document i run into an issue which i can't really solve
I have a RTX 2080 Ti use Anaconda2 with Python2.7
The theano version i am using is 1.0.3 with CUDA 9 and cuDNN v7.4.2 (Dec 14, 2018), for CUDA 9.0

My .theanorc file looks like this
[global]
floatX = float32
device = cuda

[cuda]
root = C:Program FilesNVIDIA GPU Computing ToolkitCUDAv9.0

[nvcc]
flags = -LC:Userslewu7369Anaconda2libs
compiler_bindir = C:Program Files (x86)Microsoft Visual Studio 12.0VCbin

fastmath = True

However i get the following error:

import theano
C:Userslewu7369Anaconda2libsite-packagestheanogpuarraydnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7402 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 159, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
GpuArrayException: ('nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

Unsing CUDA 8.0 and cuDNN v6.0 (April 27, 2017), for CUDA 8.0
gives the same Error:

import theano
Can not use cuDNN on context None: Use cuDNN 7.0.2 or higher for Volta.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 159, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
GpuArrayException: ('nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

Source

Wish12

Most helpful comment

@crazii I was also trying this with cuda 10.1, and it doesn't work because that version comes with a newer nvrtc64_101_0.dll, not the expected nvrtc64_70.dll. I tried installing some other random cuda versions with no luck: 10.0 (nvrtc64_100_0.dll) and 9.0 (nvrtc64_90.dll).

So I built libgpuarray myself:

conda install cmake
git clone https://github.com/Theano/libgpuarray.git
cd libgpuarray
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 15 Win64"

I built the release version using visual studio and replaced my existing gpuarray.dll with the new version:

cp ~/Anaconda3/Library/bin/gpuarray.dll ~/Anaconda3/Library/bin/gpuarray.dll.orig
cp lib/Release/gpuarray.dll ~/Anaconda3/Library/bin/gpuarray.dll

Getting closer:

pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_101.dll": The specified module could not be found.\r\n'

I copied the existing dll into my anaconda libs and renamed it to the expected filename:

cp "NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/nvrtc64_101_0.dll" ~/Anaconda3/Library/bin/nvrtc64_101.dll

Getting closer:

pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)

And this is another case of mismatched filenames. I copied my cublas dll with the expected filename:

cp "NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/cublas64_10.dll" ~/Anaconda3/Library/bin/cublas64_101.dll"

And now finally I can use the gpu:

C:\Users\kvance\Anaconda3\lib\site-packages\theano\gpuarray\dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7500 on context None
Mapped name None to device cuda: GeForce GTX 1080 (0000:01:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, vector)>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.449541 seconds
Result is [1.2317803 1.6187935 1.5227807 ... 2.2077181 2.2996776 1.623233 ]
Used the gpu

cknave on 13 Apr 2019

❤8 👍7

All 13 comments

You need cuda10 for the 20?? GPU from NVIDIA.

On Mon, Jan 21, 2019 at 7:22 AM Wish12 notifications@github.com wrote:

Hey guys i am pretty new to Deep Learning and wanted to use Theano to run
some neural nets using my gpu. However while trying to run theano with the
.theanorc document i run into an issue which i can't really solve
I have a RTX 2080 Ti use Anaconda2 with Python2.7
The theano version i am using is 1.0.3 with CUDA 9 and cuDNN v7.4.2 (Dec
14, 2018), for CUDA 9.0

My .theanorc file looks like this
[global]
floatX = float32
device = cuda

[cuda]
root = C:Program FilesNVIDIA GPU Computing ToolkitCUDAv9.0

[nvcc]
flags = -LC:Userslewu7369Anaconda2libs
compiler_bindir = C:Program Files (x86)Microsoft Visual Studio
12.0VCbin

fastmath = True

However i get the following error:

import theano
C:Userslewu7369Anaconda2libsite-packagestheanogpuarraydnn.py:184:
UserWarning: Your cuDNN version is more recent than Theano. If you
encounter problems, try updating Theano or downgrading cuDNN to a version

= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7402 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 159, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
GpuArrayException: ('nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

Unsing CUDA 8.0 and cuDNN v6.0 (April 27, 2017), for CUDA 8.0
gives the same Error:

import theano
Can not use cuDNN on context None: Use cuDNN 7.0.2 or higher for Volta.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray_init_.py",
line 159, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
GpuArrayException: ('nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/6681, or mute the thread
https://github.com/notifications/unsubscribe-auth/AALC--NjLwtGpmIq5OvKzoPAyBt3iN4Hks5vFbEWgaJpZM4aKt6W
.

nouiz on 21 Jan 2019

But does Theano 1.0.3 support CUDA 10?

Wish12 on 21 Jan 2019

Yes.

On Mon, Jan 21, 2019 at 10:03 AM Wish12 notifications@github.com wrote:

But does Theano 1.0.3 support CUDA 10?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/Theano/Theano/issues/6681#issuecomment-456103471, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-yb18KbPb-AuEbsNGX_uYVqV7fAzks5vFda3gaJpZM4aKt6W
.

nouiz on 21 Jan 2019

I downloaded and installed CUDA 10 and the cuDNN for CUDA 10. I Changes my cuda root in my theanorc file to:
[cuda]
root = C:Program FilesNVIDIA GPU Computing ToolkitCUDAv10.0

and restarted my system. However I still get an error from pygpu saying nvrtc64_70.dll wasn't found :

import theano
C:Userslewu7369Anaconda2libsite-packagestheanogpuarraydnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 99, in init_dev
**args)
File "pygpugpuarray.pyx", line 658, in pygpu.gpuarray.init
File "pygpugpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
GpuArrayException: Could not load "nvrtc64_70.dll": Das angegebene Modul wurde nicht gefunden.

But in my Cudav10.0bin Folder there is nvrtc64_100_0.dll. When changing the name i get the following error:

import theano
C:Userslewu7369Anaconda2libsite-packagestheanogpuarraydnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7402 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 227, in
use(config.device)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 214, in use
init_dev(device, preallocate=preallocate)
File "C:Userslewu7369Anaconda2libsite-packagestheanogpuarray__init__.py", line 159, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)

Using conda list, I checked for a blas lib. I have blas 1.0 installed.

Wish12 on 22 Jan 2019

I get the same problem! I installed cuda v10.1 with GeForce 1060,but I get the same error,I guess it's the pugpu not supported cuda10?

NozomiWyane on 16 Mar 2019

Similar problem here.

I have TITAN RTX x2 and use Cuda 10.1, cuDNN 7.5.

Windows10 x64
Python 3.7.1 (anaconda)
Theano "Version: 1.0.3+2.g3e47d39ac.dirty"

.theanorc

[global]
device = cuda
floatX = float32

envs

C:\Users\Administrator>set cuda
CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1
CUDA_PATH_V10_1=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1
CUDA_PATH_V9_0=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0

C:\Users\Administrator>path
PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;D:\Program\Perforce;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.1\;C:\Program Files\TortoiseGit\bin;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NGX;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps;

Error log:

C:\ProgramData\Anaconda3\lib\site-packages\theano\gpuarray\dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gpuarray\__init__.py", line 227, in 
    use(config.device)
  File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gpuarray\__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "C:\ProgramData\Anaconda3\lib\site-packages\theano\gpuarray\__init__.py", line 99, in init_dev
    **args)
  File "pygpu\gpuarray.pyx", line 658, in pygpu.gpuarray.init
  File "pygpu\gpuarray.pyx", line 587, in pygpu.gpuarray.pygpu_init
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_70.dll": \xd5\xd2\xb2\xbb\xb5\xbd\xd6\xb8\xb6\xa8\xb5\xc4\xc4\xa3\xbf\xe9\xa1\xa3\r\n'

Was I missing something?

crazii on 8 Apr 2019

So I built libgpuarray myself:

conda install cmake
git clone https://github.com/Theano/libgpuarray.git
cd libgpuarray
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 15 Win64"

I built the release version using visual studio and replaced my existing gpuarray.dll with the new version:

cp ~/Anaconda3/Library/bin/gpuarray.dll ~/Anaconda3/Library/bin/gpuarray.dll.orig
cp lib/Release/gpuarray.dll ~/Anaconda3/Library/bin/gpuarray.dll

Getting closer:

pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_101.dll": The specified module could not be found.\r\n'

I copied the existing dll into my anaconda libs and renamed it to the expected filename:

cp "NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/nvrtc64_101_0.dll" ~/Anaconda3/Library/bin/nvrtc64_101.dll

Getting closer:

pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)

And this is another case of mismatched filenames. I copied my cublas dll with the expected filename:

cp "NVIDIA GPU Computing Toolkit/CUDA/v10.1/bin/cublas64_10.dll" ~/Anaconda3/Library/bin/cublas64_101.dll"

And now finally I can use the gpu:

C:\Users\kvance\Anaconda3\lib\site-packages\theano\gpuarray\dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7.
  warnings.warn("Your cuDNN version is more recent than "
Using cuDNN version 7500 on context None
Mapped name None to device cuda: GeForce GTX 1080 (0000:01:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, vector)>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.449541 seconds
Result is [1.2317803 1.6187935 1.5227807 ... 2.2077181 2.2996776 1.623233 ]
Used the gpu

cknave on 13 Apr 2019

❤8 👍7

Using cuDNN version 5110 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 164, in
use(config.device)
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 151, in use
init_dev(device)
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 100, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
pygpu.gpuarray.GpuArrayException: (b'nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

Is there any body resolve the question?please share the resolve idea to me!!Thank you!!

cehkongfu on 15 Apr 2019

@cknave I was able to get it to work just by duplicating nvrtc64_100_0.dll and cublas64_100.dll and renaming them to XXX_70.dll

stevenwong on 17 Apr 2019

👍7 😄6 🎉1

Using cuDNN version 5110 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 164, in
use(config.device)
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 151, in use
init_dev(device)
File "C:Anaconda3envstheano35libsite-packagestheanogpuarray__init__.py", line 100, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpublas.pyx", line 149, in pygpu.blas.gemm
File "pygpublas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
pygpu.gpuarray.GpuArrayException: (b'nvrtcCompileProgram: NVRTC_ERROR_INVALID_OPTION', 3)

Is there any body resolve the question?please share the resolve idea to me!!Thank you!!

Have you solved the problem? I have the same problem as you

ABAPPLO on 13 May 2019

have you solved the problem? I use Geforce RTX 2080, CUDA 8.0, cudnn 6.0 for CUDA 8.0.
I try to configure theano in gpu but when I import theano, there is no output informaiton, neither warning or gpu information. What is wrong with my configureration.

AeolianperishG on 15 Aug 2019

Unfortunatly not. I have read that the Theano support was discontinued. I could not firgure out on my own which config works. My last try was using the latest CUDA and cudnn. However I was not sucessful.
I switched to using Pytorch and tensorflow keras. Bothe of them are working pretty good.

Wish12 on 19 Aug 2019

👍1

@cknave could you post the binary gpuarray.dll? when I run this (using visual studio 2015 community) , the gpuarray.dll will cannot be loaded by pygpu - the gpuarray.dll is in the debug (not release) directory and is 418816 bytes long.

(srs2) D:\libgpuarray\build>python -c "import pygpu" Traceback (most recent call last): File "<string>", line 1, in <module> File "d:\Anaconda3\envs\srs2\lib\site-packages\pygpu\__init__.py", line 7, in <module> from . import gpuarray, elemwise, reduction ImportError: DLL load failed: %1 is not a valid Win32 application.