Cntk: ValueError: Specified GPU device id (0) is invalid

Created on 1 May 2017 · 27Comments · Source: microsoft/CNTK

Hi,
i have built CNTK GPU successfully, and i can run the FeedForwardNet.py example.
However when i try to run the same example using GPU (try_set_default_device(cntk.device.gpu(0)))
I get the following error :

Traceback (most recent call last):
File "FeedForwardNet.py", line 82, in
try_set_default_device(cntk.device.gpu(0))
File "/CNTK/bindings/python/cntk/internal/swig_helper.py", line 69, in wrapper
result = f(args, *kwds)
File "/CNTK/bindings/python/cntk/device.py", line 94, in gpu
return cntk_py.DeviceDescriptor.gpu_device(device_id)
ValueError: Specified GPU device id (0) is invalid.

Build info:

    Built time: May  1 2017 22:44:50
    Last modified date: Mon May  1 12:04:32 2017
    Build type: release
    Build target: GPU
    With 1bit-SGD: no
    With ASGD: no
    Math lib: mkl
    CUDA_PATH: /usr/local/cuda-8.0/
    CUB_PATH: /usr/local/cub-1.4.1
    CUDNN_PATH: /usr/local/cudnn-5.1
    Build Branch: master
    Build SHA1: 4b9f8739c72068d70279f91b4b59923b2ae1fc3a (modified)

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+

Anyone has faced this issue ?
Thanks.

Source

youssefhb

👍2

Most helpful comment

I had exactly the same error on my freshly installed https://cntk.ai/PythonWheel/GPU/cntk-2.0rc2-cp35-cp35m-win_amd64.whl , Windows 7 x64, Anaconda 3 4.1.1 (64 bit). Check updates for your video driver, it works for me.

Sadler2 on 2 May 2017

👍2

All 27 comments

Sadler2 on 2 May 2017

👍2

I have the same problem.
It's random for me.
Usually, it works, but when I have MS Edge open, it usually does not work.

Ark-kun on 19 May 2017

It looks that you have installed the CPU version instead of the GPU version. I would recommend to reinstall the last GPU version and check that CNTK is using GPUs:

import cntk
cntk.logging.set_trace_level(2)
cntk.all_devices()

miguelgfierro on 30 May 2017

From build info, you can see that build target is GPU.
GPU is working fine with other libraries. The issue in CNTK side.

youssefhb on 30 May 2017

If you print cntk.all_devices() do you get CPU or GPU?

miguelgfierro on 30 May 2017

it's just empty.

youssefhb on 30 May 2017

That's odd. At least you should get CPU. I would try to install the last release RC3. I just installed this version today in a windows GPU VM and it worked.

miguelgfierro on 30 May 2017

I'll try the latest release. Hope it will resolve my issue.
Thank you.

youssefhb on 1 Jun 2017

@youssefhb is this issue resolved?

KeDengMS on 13 Jul 2017

I'm closing this issue due to lack of response. Assume this is resolved for the latest 2.0 release. Feel free to reopen this issue if needed. Thanks!

cha-zhang on 31 Jul 2017

I have install CNTK-2-0-Windows-64bit-GPU on windows 10, when I run cntk.all_devices() I get CPU, how to activate GPU. any answers

bpatil16 on 25 Oct 2017

Usually this is a CUDA issue. Please install cuda samples and run device query sample to make sure CUDA works.

KeDengMS on 25 Oct 2017

looks like even if I install the cuda I dont have GPU that CUDA-capable. I followed this

2.1. Verify You Have a CUDA-Capable GPU
To verify that your GPU is CUDA-capable, open the Control Panel ( Start > Control
Panel ) and double click on System. In the System Properties window that opens, click
the Hardware tab, then Device Manager. Expand the Display adapters entry. There you
will find the vendor name and model of your graphics card. If it is an NVIDIA card that
is listed in http://developer.nvidia.com/cuda-gpus, your GPU is CUDA-capable.
The Release Notes for the CUDA Toolkit also contain a list of supported products

bpatil16 on 25 Oct 2017

I also dont have a NIVIDIA graphic card on my machine.

bpatil16 on 25 Oct 2017

I am still having this issue.
My GPU card is GeForce GT 540M (capability 2.1)
Can you reopen the issue and give support to solve it please?
I've already try many things without success.
CNTK installed on Windows 7 64 bit via Python (3.6) wheel files - the GPU version of course.

More details

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe
Sun Jun 03 15:16:14 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 391.35                 Driver Version: 391.35                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 540M    WDDM  | 00000000:01:00.0 N/A |                  N/A |
| N/A   52C    P0    N/A /  N/A |     57MiB /  2048MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

and

python -c "import cntk; cntk.logging.set_trace_level(2); cntk.all_devices()"
-------------------------------------------------------------------
GPU info:

                Device[0]: cores = 96; computeCapability = 2.1; type = "GeForce
GT 540M"; total memory = 2048 MB; free memory = 1990 MB
-------------------------------------------------------------------

resulting in the error

python -c "import cntk as C; C.device.try_set_default_device(C.device.gpu(0)) "
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\dev\python\Jupiter\Conda\lib\site-packages\cntk\internal\swig_helper.
py", line 69, in wrapper
    result = f(*args, **kwds)
  File "C:\dev\python\Jupiter\Conda\lib\site-packages\cntk\device.py", line 94,
in gpu
    return cntk_py.DeviceDescriptor.gpu_device(device_id)
ValueError: Specified GPU device id (0) is invalid.

[CALL STACK]
    > CNTK::DeviceDescriptor::  GPUDevice
    - PyInit__cntk_py
    - PyCFunction_FastCallDict
    - PyObject_CallFunctionObjArgs
    - PyEval_EvalFrameDefault
    - Py_CheckFunctionResult
    - PyList_Size
    - PyEval_EvalFrameDefault
    - Py_CheckFunctionResult
    - PyObject_CallFunctionObjArgs
    - PyEval_EvalFrameDefault
    - Py_CheckFunctionResult
    - PyEval_EvalCodeEx
    - PyEval_EvalCode
    - PyArena_Free
    - PyRun_StringFlags

Depending on the CNTK, with the last one I get a crash of the anaconda python itself

giuliohome on 3 Jun 2018

CUDA 9 has dropped support for capability < 3.0. Please use CNTK 2.3 and install CUDA 8.0 instead.

KeDengMS on 3 Jun 2018

I've dowloaded and tried CUDA 8.0. The Visual Studio Integration (I have vs2015) fails and I have to uncheck that part.
Aside from that I have tried many versions of CNTK starting from 2.0 and I couldn't make it work... I will test with CNTK 2.3 again and let you know in case I have any postive result

giuliohome on 3 Jun 2018

@KeDengMS
This is the CNTK installed

(base) C:\Users\MyHome>pip install https://cntk.ai/PythonWheel/GPU/cntk-2.3-cp36
-cp36m-win_amd64.whl
Collecting cntk==2.3 from https://cntk.ai/PythonWheel/GPU/cntk-2.3-cp36-cp36m-wi
n_amd64.whl
  Downloading https://cntk.ai/PythonWheel/GPU/cntk-2.3-cp36-cp36m-win_amd64.whl
(311.0MB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 311.0MB 18kB/s
Requirement already satisfied: scipy>=0.17 in c:\dev\python\jupiter\conda\lib\si
te-packages (from cntk==2.3) (0.19.1)
Requirement already satisfied: numpy>=1.11 in c:\dev\python\jupiter\conda\lib\si
te-packages (from cntk==2.3) (1.13.3)
Installing collected packages: cntk
Successfully installed cntk-2.3

printing the version as expected

(base) C:\Users\MyHome> python -c "import cntk; print(cntk.__version__)"
2.3

I can't still have the GPU recognized as a valid device

(base) C:\Users\MyHome>python -c "import cntk; cntk.logging.set_trace_level(2);
cntk.all_devices()"
-------------------------------------------------------------------
GPU info:

                Device[0]: cores = 96; computeCapability = 2.1; type = "GeForce
GT 540M"; total memory = 2048 MB; free memory = 1990 MB
-------------------------------------------------------------------

giuliohome on 3 Jun 2018

Same issue from a trivial C# console app with Visual Studio 2015 after

PM> Install-Package CNTK.GPU -Version 2.3.0


Attempting to gather dependency information for package 'CNTK.GPU.2.3.0' with respect to project 'ConsoleCNTK', targeting '.NETFramework,Version=v4.6.2'
Gathering dependency information took 1,23 sec
Attempting to resolve dependencies for package 'CNTK.GPU.2.3.0' with DependencyBehavior 'Lowest'
Resolving dependency information took 0 ms
Resolving actions to install package 'CNTK.GPU.2.3.0'
Resolved actions to install package 'CNTK.GPU.2.3.0'
Retrieving package 'CNTK.GPU 2.3.0' from 'nuget.org'.
  GET https://www.nuget.org/api/v2/package/CNTK.GPU/2.3.0
  OK https://www.nuget.org/api/v2/package/CNTK.GPU/2.3.0 1204ms
Installing CNTK.GPU 2.3.0.
Adding package 'CNTK.GPU.2.3.0' to folder 'C:\CNTK\mySample\ConsoleCNTK\packages'
Added package 'CNTK.GPU.2.3.0' to folder 'C:\CNTK\mySample\ConsoleCNTK\packages'
Added package 'CNTK.GPU.2.3.0' to 'packages.config'
Successfully installed 'CNTK.GPU 2.3.0' to ConsoleCNTK
Executing nuget actions took 5,58 min
Time Elapsed: 00:05:36.3834711

VS exception in C#

Again, DeviceDescriptor.AllDevices() returns only one device [0] of .Type equals to CPU

giuliohome on 3 Jun 2018

I'm afraid that CNTK 2.3 does not support GPU with _compute capability_ 2.1 for CUDA (and not even CNTK 2.0)
see this line from the source of the branch release/2.3
CNTK: The GPU (%s) has compute capability %d.%d. CNTK is only supported on GPUs with compute capability 3.0 or greater"
Exactly this line is the crucial point in Common/BestGpu.cpp:
static const int MininumCCMajorForGpu = 3; // cntk supports GPUs with Compute Capability > 3.0

giuliohome on 3 Jun 2018

Please try a newer GPU then, capability 2.x is behind us now.

KeDengMS on 6 Jun 2018

Your reply makes no sense. You gave a wrong suggestion. Now you are adding no value to the thread. Moreover I can well use other software instead of changing my hardware. The word "now" is also misleading since I have already shown that CNTK dropped support for Fermi long ago already when CUDA 8.0 was running on GPU 2.1. And it is still viable and not too obsolete.

giuliohome on 6 Jun 2018

@giuliohome I understood your frustration when the support of your hardware is dropped. Unfortunately, as you pointed out, CNTK stopped supporting Fermi cards a long time ago.

The fact is that NVidia has stopped supporting capability 2.x in CUDA 9.0, and now most toolkits have upgraded to CUDA 9.x as CNTK. If you feel strongly against NVidia's decision, please send your comments to them directly.

KeDengMS on 6 Jun 2018

My cntk can not detect the GPU. It also returns (CPU, ). The following is the information of my CNTK and the CUDA? All the above cases do not seem to fall in my situation. I am running on Windows 2012 server. Help?

print(C.__version__)
2.6

from numba import cuda
cuda.detect()

Found 4 CUDA devices
id 0 b'Tesla K40c' [SUPPORTED]
compute capability: 3.5
pci device id: 0
pci bus id: 4
id 1 b'Tesla K40c' [SUPPORTED]
compute capability: 3.5
pci device id: 0
pci bus id: 2
id 2 b'Tesla K40c' [SUPPORTED]
compute capability: 3.5
pci device id: 0
pci bus id: 131
id 3 b'Tesla K40c' [SUPPORTED]
compute capability: 3.5
pci device id: 0
pci bus id: 132
Summary:
4/4 devices are supported

C.logging.set_trace_level(2);
C.all_devices()

(CPU,)

nimning on 26 Sep 2018

Did you notice any warning in import cntk? Maybe some cuda libs are not found.

KeDengMS on 26 Sep 2018

Did you notice any warning in import cntk? Maybe some cuda libs are not found.

C:\ProgramData\Anaconda3\lib\site-packages\cntk\cntk_py_init.py:32: UserWarning: Unsupported Windows version (2012serverr2). CNTK supports Windows 10 and above, only.
warnings.warn('Unsupported Windows version (%s). CNTK supports Windows 10 and above, only.' % __my_distro_ver__)

nimning on 27 Sep 2018

I had this issue.

I spent hours looking at the CUDA install, verifying all was working. In the end testing Python code:

from cntk.device import try_set_default_device, gpu
try_set_default_device(gpu(0))
import cntk

import cntk as C
print(C.device.all_devices())
print(C.device.try_set_default_device(cntk.device.gpu(0)))
print(C.device.use_default_device())

I resolved my problem by ensuring the package CNTK.CPUOnly was not installed

cntk cpu only

This somehow killed the GPU component:

cntk gpu with cpu only

I uninstalled the whole lot, reinstalled just: CNTK.GPU

cntk gpu fresh install

Note: The other components are CNTK.GPU Requirements.

Run:

var device = CNTK.DeviceDescriptor.GPUDevice(0);
Console.WriteLine($"Using:  {device.Type} Device info: {device.AsString()}");
Console.ReadLine();

and the problem was resolved!

Its worth noting, the issue I saw was on the CNTK Examples downloaded from the CNTK GitHub site. Restoring the packages is the only thing I can think of that may caused the issue.