Theano 0.9 (theano.gpuarray): Could not initialize pygpu, support disabled

[jingweimo]

I just installed the latest theano. It works well without configuration to use gpu. I use anacond2 operated in windows 10 with navidia geforce 940 m. This system works for the old backend for gpu computation.

To use gpu, I create the .theanorc file as follows:

 [global]
 device = cuda
 floatX = float32

 [nvcc]
 fastmath = True

 [cuda]
 root = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.5

Then, I got the error as follows:

import theano

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
 Traceback (most recent call last):
 File "C:\Users\YL\Anaconda2\lib\site-packages\theano\gpuarray\__init__.py", line 175, in   <module>
  use(config.device)
 File "C:\Users\YL\Anaconda2\lib\site-packages\theano\gpuarray\__init__.py", line 162, in use
 init_dev(device, preallocate=preallocate)
 File "C:\Users\YL\Anaconda2\lib\site-packages\theano\gpuarray\__init__.py", line 65, in init_dev
sched=config.gpuarray.sched)
 File "pygpu\gpuarray.pyx", line 614, in pygpu.gpuarray.init (pygpu/gpuarray.c:9415)
 File "pygpu\gpuarray.pyx", line 566, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9106)
 File "pygpu\gpuarray.pyx", line 1021, in pygpu.gpuarray.GpuContext.__cinit__ (pygpu/gpuarray.c:13468)
 GpuArrayException: Error loading library: 0

It seem there is something wrong with my pygpu. Then I import and test it:

  import pygpu
  pygpu.test()

It results in error messages as follows

ERROR: Failure: RuntimeError (No test device specified.  Specify one using the DEVICE or

GPUARRAY_TEST_DEVICE environment variables.)

Traceback (most recent call last):
  File "C:\Users\YL\Anaconda2\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
 File "C:\Users\YL\Anaconda2\lib\site-packages\nose\importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
 File "C:\Users\YL\Anaconda2\lib\site-packages\nose\importer.py", line 94, in importFromDir
 mod = load_module(part_fqname, fh, filename, desc)
 File "C:\Users\YL\Anaconda2\lib\site-packages\pygpu\tests\test_basic.py", line 5, in <module>
from .support import (gen_gpuarray, context)
 File "C:\Users\YL\Anaconda2\lib\site-packages\pygpu\tests\support.py", line 32, in <module>
context = gpuarray.init(get_env_dev())
 File "C:\Users\YL\Anaconda2\lib\site-packages\pygpu\tests\support.py", line 29, in get_env_dev
raise 

 RuntimeError("No test device specified.  Specify one using the DEVICE or    GPUARRAY_TEST_DEVICE environment variables.")
RuntimeError: No test device specified.  Specify one using the DEVICE or
GPUARRAY_TEST_DEVICE environment variables.

[botev]

For pygpu you need something like this (assuming Linux shell):

[~]$ DEVICE="cuda0" python 
>>> import pygpu
>>> pygpu.test()

e.g. you need to set up the environmental variable DEVICE before you run it or as the error says GPUARRAY_TEST_DEVICE.
Please post the output of that here, however I personally have no experience with Windows so someone else might need to jump in to help.

[jingweimo]

@botev: I have updated cuda to 8.0, reinstalled anaconda and theano, but that doesn't help.
setting up DEVICE is to put the cuda 8.0 to the path of the environment variables?

[botev]

No that tells which device to use for the pygpu.test(). Otherwise, there are a few environmental variables which you need to setup for Theano to work correctly with CUDA and those are described on the installation page.

[Imorton-zd]

Hi, I have also encountered the same problem. Have you already solved this problem?

[lamblin]

Regarding GpuArrayException: Error loading library: 0, if you can update and compile the latest development version of libgpuarray, you will likely get a better error message.
There may be something wrong in your set-up of cuda.

[lamblin]

FYI, packages for version 0.6.8 are out, which contain the updated error messages I mentioned above.
Could you try with that version?

[GitHubKay]

same Error here

Traceback (most recent call last): File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module> use(config.device) File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use init_dev(device) File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 60, in init_dev sched=config.gpuarray.sched) File "pygpu\gpuarray.pyx", line 614, in pygpu.gpuarray.init (pygpu/gpuarray.c:9415) File "pygpu\gpuarray.pyx", line 566, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9106) File "pygpu\gpuarray.pyx", line 1021, in pygpu.gpuarray.GpuContext.__cinit__ (pygpu/gpuarray.c:13468) pygpu.gpuarray.GpuArrayException: Error loading library: 0

is there any solution so far?

[botev]

@GitHubKay Could you write a more detailed description of your setup - e.g. OS (Unix/Mac/Windows), CUDA version, the contents of .theanorc, python version and what environmental variables you have related to Theano. Also, have you run any basic CUDA tests without Theano?

[lamblin]

Also, the version of theano and libgpuarray you are using would be useful.
If you are not using the conda package for libgpuarray 0.6.8, could you upgrade to it?

[GitHubKay]

@botev @lamblin , thx for your quick response! Of cause! I totally forgot about these information:
first of all, i followed the instructions here to setup everything. And i took my time cause i've never setup an Environment with Theano. So i'm certain that i've done exactly as i was told. Right now i'm hanging at this step.

System:

• Windows 10 Education
• cuda_8.0.61_win10 - i haven't run the update
• cuDNN 8.0 v5.1

I haven't defined any config file .theanorc.
i've installed the following versions:

• Theano 0.9.0 with pip install keras==2.0.5
• libgpuarray 0.6.2 with conda install pygpu==0.6.2 nose
• python 3.6.1

Environmental variables:

• CUDA_PATH = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0
• PYTHON_HOME = C:\Users\RazZzoR\Anaconda3
• THEANO_FLAGS_CPU = float32,device=cpu
• THEANO_FLAGS_GPU = floatX=float32,device=cuda0,dnn.enabled=False,gpuarray.preallocate=0.8
• THEANO_FLAGS_GPU_DNN = floatX=float32,device=cuda0,optimizer_including=cudnn,gpuarray.preallocate=0.8,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic,dnn.include_path=e:/toolkits.win/cuda-8.0.61/include,dnn.library_path=e:/toolkits.win/cuda-8.0.61/lib/x64

PATH:

• %CUDA_PATH%\bin
• %CUDA_PATH%\libnvvp
• %PYTHON_HOME%
• %PYTHON_HOME%\Scripts
• %PYTHON_HOME%\Library\bin

Aditional GPU tests:

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))

The console log:

2017-07-10 09:50:41.042160: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950, pci bus id: 0000:04:00.0)
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 950, pci bus id: 0000:04:00.0
2017-07-10 09:50:41.043269: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\direct_session.cc:265] Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 950, pci bus id: 0000:04:00.0

MatMul_1: (MatMul): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.048296: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] MatMul_1: (MatMul)/job:localhost/replica:0/task:0/gpu:0
MatMul: (MatMul): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.049002: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] MatMul: (MatMul)/job:localhost/replica:0/task:0/gpu:0
b_1: (Const): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.050094: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] b_1: (Const)/job:localhost/replica:0/task:0/gpu:0
a_1: (Const): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.050975: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] a_1: (Const)/job:localhost/replica:0/task:0/gpu:0
b: (Const): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.054118: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] b: (Const)/job:localhost/replica:0/task:0/gpu:0
a: (Const): /job:localhost/replica:0/task:0/gpu:0
2017-07-10 09:50:41.057168: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\simple_placer.cc:847] a: (Const)/job:localhost/replica:0/task:0/gpu:0

[lamblin]

Thanks for the detailed answer.
Could you try conda install pygpu=0.6.8, assiming it does not break anything, and try to launch the Theano script again? Hopefully it will make the error message more clear.

[GitHubKay]

@lamblin i did as you proposed to do:

(dlwin36) C:\Users\RazZzoR\fast-ai>SET THEANO_FLAGS=%THEANO_FLAGS_GPU%

(dlwin36) C:\Users\RazZzoR\fast-ai>python cpu_gpu_test.py
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module>
    use(config.device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use
    init_dev(device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 60, in init_dev
    sched=config.gpuarray.sched)
  File "pygpu\gpuarray.pyx", line 634, in pygpu.gpuarray.init (pygpu/gpuarray.c:9424)
  File "pygpu\gpuarray.pyx", line 584, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9115)
  File "pygpu\gpuarray.pyx", line 1057, in pygpu.gpuarray.GpuContext.__cinit__ (pygpu/gpuarray.c:13417)
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_90.dll": Das angegebene Modul wurde nicht gefunden.\r\n'

[botev]

So it seems, at least from the glance of it, that it can't find this nvrtc64_90.dll. Could you search for that file in the location of the cuda install and see if you can find the file here it is. If that is not lib/x64 you will need to add its directory to the library path.

[GitHubKay]

i just found the ncrtc64_80.dll in the folder ./v8.0/bin

[botev]

hmm.. could you make a shortcut to it in the lib/x64 directory with the same name

[GitHubKay]

same error

[GitHubKay]

I found this git but have no idea where to set the GPUARRAY_CUDA_VERSION to 80

I just did set GPUARRAY_CUDA_VERSION = 80 but still get the same error.

[GitHubKay]

i did a few things:
First i run this script to validate the gpucudnn install with keras.

(dlwin36) C:\Users\RazZzoR\fast-ai>set GPUARRAY_CUDA_VERSION = "80"

(dlwin36) C:\Users\RazZzoR\fast-ai>set THEANO_FLAGS= %THEANO_FLAGS_GPU%

(dlwin36) C:\Users\RazZzoR\fast-ai>python mnist_cnn.py
Using TensorFlow backend.
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/12
2017-07-16 12:42:07.334857: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.334926: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.335786: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.336198: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.336226: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.336511: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.336799: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.337093: W c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-07-16 12:42:07.687884: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 950
major: 5 minor: 2 memoryClockRate (GHz) 1.2785
pciBusID 0000:04:00.0
Total memory: 2.00GiB
Free memory: 1.64GiB
2017-07-16 12:42:07.687977: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:961] DMA: 0
2017-07-16 12:42:07.689627: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0:   Y
2017-07-16 12:42:07.689997: I c:\tf_jenkins\home\workspace\release-win\m\windows-gpu\py\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950, pci bus id: 0000:04:00.0)
60000/60000 [==============================] - 16s - loss: 0.3462 - acc: 0.8934 - val_loss: 0.0785 - val_acc: 0.9752
Epoch 2/12
60000/60000 [==============================] - 13s - loss: 0.1175 - acc: 0.9650 - val_loss: 0.0543 - val_acc: 0.9833
Epoch 3/12
60000/60000 [==============================] - 13s - loss: 0.0874 - acc: 0.9740 - val_loss: 0.0446 - val_acc: 0.9846
Epoch 4/12
60000/60000 [==============================] - 148s - loss: 0.0749 - acc: 0.9783 - val_loss: 0.0384 - val_acc: 0.9867
Epoch 5/12
60000/60000 [==============================] - 13s - loss: 0.0634 - acc: 0.9809 - val_loss: 0.0345 - val_acc: 0.9878
Epoch 6/12
60000/60000 [==============================] - 13s - loss: 0.0576 - acc: 0.9830 - val_loss: 0.0337 - val_acc: 0.9884
Epoch 7/12
60000/60000 [==============================] - 13s - loss: 0.0512 - acc: 0.9853 - val_loss: 0.0358 - val_acc: 0.9878
Epoch 8/12
60000/60000 [==============================] - 13s - loss: 0.0476 - acc: 0.9858 - val_loss: 0.0294 - val_acc: 0.9896
Epoch 9/12
60000/60000 [==============================] - 13s - loss: 0.0450 - acc: 0.9867 - val_loss: 0.0301 - val_acc: 0.9888
Epoch 10/12
60000/60000 [==============================] - 13s - loss: 0.0413 - acc: 0.9873 - val_loss: 0.0277 - val_acc: 0.9904
Epoch 11/12
60000/60000 [==============================] - 13s - loss: 0.0377 - acc: 0.9884 - val_loss: 0.0254 - val_acc: 0.9909
Epoch 12/12
60000/60000 [==============================] - 13s - loss: 0.0372 - acc: 0.9889 - val_loss: 0.0276 - val_acc: 0.9907
Test loss: 0.027580858579
Test accuracy: 0.9907

with a few new error messages, but at least it used the GPU!

then i run the first script to validate the Theano installation. Still couldn't find nvrtc64_90.dll as mentioned before.

(dlwin36) C:\Users\RazZzoR\fast-ai>set GPUARRAY_CUDA_VERSION = "80"

(dlwin36) C:\Users\RazZzoR\fast-ai>set THEANO_FLAGS= %THEANO_FLAGS_GPU%

(dlwin36) C:\Users\RazZzoR\fast-ai>python cpu_gpu_test.py
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module>
    use(config.device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use
    init_dev(device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 60, in init_dev
    sched=config.gpuarray.sched)
  File "pygpu\gpuarray.pyx", line 634, in pygpu.gpuarray.init (pygpu/gpuarray.c:9424)
  File "pygpu\gpuarray.pyx", line 584, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9115)
  File "pygpu\gpuarray.pyx", line 1057, in pygpu.gpuarray.GpuContext.__cinit__ (pygpu/gpuarray.c:13417)
pygpu.gpuarray.GpuArrayException: b'Could not load "nvrtc64_90.dll": Das angegebene Modul wurde nicht gefunden.\r\n'
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 11.843616 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

I thought this problem would be fixed by set GPUARRAY_CUDA_VERSION = "80"!?
Soon there will be a fix for the version problem. Is there any way to get to a stable status, i need this environment to run some tutorials and learn!

[notoraptor]

Hello @GitHubKay , when you set GPUARRAY_CUDA_VERSION = 80, there should be not spaces around the equality sign:

set GPUARRAY_CUDA_VERSION=80
echo %GPUARRAY_CUDA_VERSION%
REM must print 80.

[GitHubKay]

Thx @notoraptor i tried this at some point, but as you can see:

(dlwin36) C:\Users\RazZzoR\fast-ai>echo %GPUARRAY_CUDA_VERSION%
80

(dlwin36) C:\Users\RazZzoR\fast-ai>set THEANO_FLAGS=%THEANO_FLAGS_GPU%

(dlwin36) C:\Users\RazZzoR\fast-ai>echo %THEANO_FLAGS%
floatX=float32,device=cuda0,dnn.enabled=False,gpuarray.preallocate=0.8

(dlwin36) C:\Users\RazZzoR\fast-ai>python cpu_gpu_test.py
Can not use cuDNN on context None: Disabled by dnn.enabled flag
WARNING: Preallocating too much memory can prevent cudnn and cublas from working properly
Preallocating 1638/2048 Mb (0.800000) on cuda0
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module>
    use(config.device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use
    init_dev(device)
  File "C:\Users\RazZzoR\Anaconda3\envs\dlwin36\lib\site-packages\theano\gpuarray\__init__.py", line 100, in init_dev
    pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
  File "pygpu\blas.pyx", line 149, in pygpu.blas.gemm (pygpu/blas.c:3535)
  File "pygpu\blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2025)
pygpu.gpuarray.GpuArrayException: (b'cublasCreate: (cublas) GPU ressource allocation failed.', 11)
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 12.100283 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

The first script now shows other errors. The second script breaks down without any error message.

[GitHubKay]

The second script now shows:

• could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
• error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
• could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
• Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)

Then it breaks down.

[notoraptor]

Weird.

Just a point: you have dnn.enabled=False into THEANO_FLAGS_GPU : could you remove it from these flags (as it forces Theano to disable cuDNN) ?

[GitHubKay]

@notoraptor you're my hero today 👍
Both scripts are running fine now! I can finally start my journey 💃
I hope this helps others as well... thx to all the others ;)

[afung-git]

@notoraptor where in the Environment Variables do you set GPUARRAY_CUDA_VERSION=80
what is the environment name and value?

I am new to theano and am currently experiencing the same Could not load "nvrtc64_90.dll" error. Any help in this would be appreciated.

[notoraptor]

@afung01 The environment name is GPUARRAY_CUDA_VERSION and value is 80.

In command line you can just type set GPUARRAY_CUDA_VERSION=80.

Or you can set it in Windows Control Panel.

[notoraptor]

@afung01 See also my comment here: https://github.com/Theano/Theano/issues/6145#issuecomment-319958909 .

[afung-git]

@notoraptor Thanks! I guess in the past after I created the variable I never bothered to closed my control panel before retesting, works now!

[JeffreyYeTHU]

I know this issue has been closed, but I may have a similar problem.

Just as @GitHubKay , I am also follow this instruction, and encounter several issues, which in order is:

TheanoConfigWarning: Config key 'THEANO_FLAGS_GPU' has no value, ignoring it #17

theano issue #6006, i.e., this one

theano issue #5768

Following these instructions, all the issues seem to be solved. I run test code in this section, everything works fine, without error. But when I test this code, with theano backend, something went wrong, but fine with CNTK backend:

If I use CNTK as backend, the info suggest everything is fine.

(C:\ProgramData\Anaconda3) c:\Users\yejian\dlwin36>set KERAS_BACKEND=cntk
(C:\ProgramData\Anaconda3) c:\Users\yejian\dlwin36>python mnist_test.py
Using CNTK backend
Selected GPU[0] GeForce GTX 1070 as the process wide default device.
x_train shape: (50000, 28, 28, 1)
50000 train samples
10000 test samples
Train on 50000 samples, validate on 10000 samples
Epoch 1/12
C:\ProgramData\Anaconda3\lib\site-packages\cntk\core.py:351: UserWarning: your data is of type "float64", but your input variable (uid "Input113") expects "". Please convert your data beforehand to speed up training.
(sample.dtype, var.uid, str(var.dtype)))
50000/50000 [==============================] - 7s - loss: 2.3008 - acc: 0.1123 - val_loss: 2.2940 - val_acc: 0.1135
Epoch 2/12
50000/50000 [==============================] - 6s - loss: 1.0123 - acc: 0.6675 - val_loss: 0.3049 - val_acc: 0.9126
Epoch 3/12
50000/50000 [==============================] - 6s - loss: 0.3815 - acc: 0.8863 - val_loss: 0.2288 - val_acc: 0.9312
Epoch 4/12
50000/50000 [==============================] - 6s - loss: 0.3249 - acc: 0.9009 - val_loss: 0.1955 - val_acc: 0.9399
Epoch 5/12
50000/50000 [==============================] - 6s - loss: 0.2946 - acc: 0.9126 - val_loss: 0.1780 - val_acc: 0.9470
Epoch 6/12
50000/50000 [==============================] - 6s - loss: 0.2706 - acc: 0.9198 - val_loss: 0.1591 - val_acc: 0.9538
Epoch 7/12
50000/50000 [==============================] - 6s - loss: 0.2571 - acc: 0.9235 - val_loss: 0.1502 - val_acc: 0.9559
Epoch 8/12
50000/50000 [==============================] - 6s - loss: 0.2455 - acc: 0.9270 - val_loss: 0.1424 - val_acc: 0.9574
Epoch 9/12
50000/50000 [==============================] - 6s - loss: 0.2354 - acc: 0.9302 - val_loss: 0.1364 - val_acc: 0.9593
Epoch 10/12
50000/50000 [==============================] - 6s - loss: 0.2256 - acc: 0.9333 - val_loss: 0.1317 - val_acc: 0.9597
Epoch 11/12
50000/50000 [==============================] - 6s - loss: 0.2159 - acc: 0.9352 - val_loss: 0.1242 - val_acc: 0.9625
Epoch 12/12
50000/50000 [==============================] - 6s - loss: 0.2094 - acc: 0.9381 - val_loss: 0.1204 - val_acc: 0.9626
Test loss: 0.120412370583
Test accuracy: 0.9626

While changing backend to theano, it break down without any error message, like this:

(C:\ProgramData\Anaconda3) c:\Users\yejian\dlwin36>set KERAS_BACKEND=theano
(C:\ProgramData\Anaconda3) c:\Users\yejian\dlwin36>set THEANO_FLAGS=%THEANO_FLAGS_GPU_DNN%
(C:\ProgramData\Anaconda3) c:\Users\yejian\dlwin36>python mnist_test.py
Using Theano backend.
Using cuDNN version 5110 on context None
Preallocating 6553/8192 Mb (0.800000) on cuda0
Mapped name None to device cuda0: GeForce GTX 1070 (0000:01:00.0)
x_train shape: (50000, 28, 28, 1)
50000 train samples
10000 test samples

After this, it pop out a dialog saying that python has stop work, like this picture. I apologize for this link, I try to upload the picture, failed several times, I am new to git.

I already remove dnn.enabled=False in THEANO_FLAGS_GPU, and now I don't know what to do.

Do you guys have any suggestion?

Following is my detailed system setup.

System info

• win 10 pro
• cuda_8.0.61_win10
• cudnn-8.0-windows10-x64-v5.1
• Theano 0.9.0
• tensorflow-gpu 1.2.0
• python 3.6.1
• Keras 2.0.5

I only got the default root environment setting in anaconda, and I have't defined config file .theanorc

Environment variables

• THEANO_FLAGS = floatX=float32,device=cuda0,optimizer_including=cudnn,gpuarray.preallocate=0.8,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic,dnn.include_path=e:/toolkits.win/cuda-8.0.61/include,dnn.library_path=e:/toolkits.win/cuda-8.0.61/lib/x64
• THEANO_FLAGS_GPU = floatX=float32,device=cuda0,gpuarray.preallocate=0.8
• THEANO_FLAGS_GPU_DNN = floatX=float32,device=cuda0,optimizer_including=cudnn,gpuarray.preallocate=0.8,dnn.conv.algo_bwd_filter=deterministic,dnn.conv.algo_bwd_data=deterministic,dnn.include_path=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v8.0/include,dnn.library_path=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v8.0/lib/x64

[notoraptor]

@JeffreyYe1988 Could you try to:

1. use the latest pygpu version ? (conda install pygpu=0.6.9)
2. purge theano cache ? (theano-cache purge)
3. run your test code again ?

[JeffreyYeTHU]

@notoraptor you are my life saver! Now the test code run smoothly, thx!

[nouiz]

As this is fixed in the master, I'll close this issue. We are working on Theano 0.10 and should be out "shortly". If all goes well we will do an RC this week.

[GitHubKay]

Since Keras was running smoothly on tensorflow i now changed to theano and again the hustle with the gpu support starts. same configurations as before:

keras.json

"epsilon": 1e-07,
"backend": "theano",
"floatx": "float32",
"image_dim_ordering": "th",
"image_data_format": "channels_first"

.theanorc

[global]
floatX = float32
device = cuda0


[lib]
cnmem=0.5


[nvcc]
fastmath = True
arch=sm_50 
optimizer=fast_compile


[cuda]
root=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0

while loading Theano in Keras:

Using Theano backend.
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'In file included from C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v8.0\\include/driver_types.h:53:0,\r\n                 from C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v8.0\\include/cudnn.h:63,\r\n                 from C:\\Users\\RazZzoR\\AppData\\Local\\Temp\\try_flags_70r3bkok.c:4:\r\nC:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v8.0\\include/host_defines.h:84:0: warning: "__cdecl" redefined\r\n #define __cdecl\r\n ^\r\n<built-in>: note: this is the location of the previous definition\r\nC:/Users/RazZzoR/Anaconda3/envs/dlwin36_gpuTh/Library/mingw-w64/bin/../lib/gcc/x86_64-w64-mingw32/5.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lcudnn\r\ncollect2.exe: error: ld returned 1 exit status\r\n'
Mapped name None to device cuda0: GeForce GTX 1070 (0000:05:00.0)

thx guys!

[GitHubKay]

Okay not it works!

i found #5776 which solved my problem!
just add the dnn path to your .theanorc dose the trick!

[dnn]
library_path = C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0\lib\x64

have fun doing awesome shit!

[brendanruff]

Hi. I have the same problem on moving to the new gpu back-end but setting up the new environment variables with the paths for CUDA 8 and cudnn(it is the same path as included in the toolkit now) but I get GpuArrayException: could not load nvrtc64_70.dll the specified module could not be found. So theano is looking for the wrong cuda version and so trying to load a wrongly named file, ie it is not the path but the file name. I set the theano environment vars copied from the previous post. This is Windows 7 and CUDA 8 with cuDNN files in the same path (supplied in the toolkit download.

[nouiz]

Note, there was fixes related to the last comment to that merged.