Darkflow: Windows GPU not working. CUDNN_STATUS_BAD_PARAM

[strickon]

So I have been trying to run on windows 10 , gtx 1060, cudnn 5.1 cuda 8, and tensorflow-gpu and keep getting these errors. Tensforlow CPU works fine.

E c:\tf_jenkins\home\workspace\release-win\device\gpuos\windows\tensorflowstream_executor\cuda\cuda_dnn.cc:397] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E c:\tf_jenkins\home\workspace\release-win\device\gpuos\windows\tensorflowstream_executor\cuda\cuda_dnn.cc:404] error retrieving driver version: Permission denied: could not open driver version path for reading: /proc/driver/nvidia/version
E c:\tf_jenkins\home\workspace\release-win\device\gpuos\windows\tensorflowstream_executor\cuda\cuda_dnn.cc:364] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
F c:\tf_jenkins\home\workspace\release-win\device\gpuos\windows\tensorflow\core\kernels\conv_ops.cc:605] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)

I did a bunch of digging and testing. It seems that these errors stem from something related to conv2d and cudnn. Unfortunately you can't turn cudnn off which I tried as there is no gpu version of conv2d that doesn't use it.

I have also read that this error sometimes is masking another completely unrelated error. Some say it has to do with empty numpy arrays. I have been trying to track down this error. I think the CPU version may be more lenient in not crashing on this error so that is why we see it run there.

Anyone have any luck resolving this?

[strickon]

looks like if I adjust the gpu usage to .7 it works. I am not understanding what that actually means and how that other error should manifest itself.

[thtrieu]

Although you did not post the error message (only traceback is provided) I think this is solely a memory problem. When one runs out of memory, error messages can be very weird.
Close since the original issue is resolved using --gpu 0.7

[uptown]

@strickon There's a discussion similar with this issue.
https://devtalk.nvidia.com/default/topic/901646/-980-ti-windows-10-cuda-7-5-out-of-memory-after-allocating-4-5-out-of-6gb/?offset=4
https://social.technet.microsoft.com/Forums/windows/en-US/15b9654e-5da7-45b7-93de-e8b63faef064/windows-10-does-not-let-cuda-applications-to-use-all-vram-on-especially-secondary-graphics-cards?forum=win10itprohardware
In short, avoiding to use Windows 10 is the best solution!

BTW, using --gpu 0.85 worked for me in Windows 10.

[PaulHuynh489]

I'm running into the same issue. How do you set --gpu 0.8?

[janakptl00]

Running into simmilar problem, would love some explanation on how to set ''--gpu" parameter ..

[uptown]

hey guys, my computer has multiple gpus, so I can set --gpu 0.85 with "GPU NOT FOR DISPLAY" ( I thinks 0.15 loss is because of the Win10 issue).
If you use GPU which is also handling your display, you may reduce GPU usage rate.

[zmchang]

I am currently try to run object_detection demo on windows 10 and try to follow uptown's suggestion to set --gpu 0.85. But I don't know where and how to set this parameter.
I would appreciate it if you could point me to the right direction.
Thanks.

[Vozf]

For me the problem was using wrong cudnn lib
I used cudnn for cuda 9.1 when I had cuda 9.0. So i reinstalled cudnn for cuda 9.0 and everything worked.

[zmchang]

Thanks very much.I fixed it by changing cudnn from 9 to cudnn 8 and it worked.H  ave you tried to use different models to run this tutorial?I have issues, when I tried to use inception resnet V2 model.
William

  From: Alexander <[email protected]>

To: thtrieu/darkflow [e-mail]
Cc: zmchang [e-mail]; Comment [e-mail]
Sent: Sunday, April 22, 2018 10:08 AM
Subject: Re: [thtrieu/darkflow] Windows GPU not working. CUDNN_STATUS_BAD_PARAM (#98)

For me the problem was using wrong cudnn lib
I used cudnn for cuda 9.1 when I had cuda 9.0. So i reinstalled cudnn for cuda 9.0 and everything worked.—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[Vozf]

Nope. I just found this issue by googling. I had this problem in my project using inceptionV3. 24.04.2018, 05:06, "zmchang" [e-mail]:Thanks very much.I fixed it by changing cudnn from 9 to cudnn 8 and it worked.H  ave you tried to use different models to run this tutorial?I have issues, when I tried to use inception resnet V2 model.WilliamFrom: Alexander [e-mail]: thtrieu/darkflow [e-mail]: zmchang [e-mail]; Comment [e-mail]: Sunday, April 22, 2018 10:08 AMSubject: Re: [thtrieu/darkflow] Windows GPU not working. CUDNN_STATUS_BAD_PARAM (#98)For me the problem was using wrong cudnn libI used cudnn for cuda 9.1 when I had cuda 9.0. So i reinstalled cudnn for cuda 9.0 and everything worked.—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread. —You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread. 

[AlessandroRestagno]

I had the same error. I did a stupid mistake. I downloaded cuDNN for CUDA 9.2, when I had to use CUDA 9.0,
After downloading the right version, I fixed the problem.

[Saintat1]

I had the same error, fixed it by upgrading the graphics driver(1060).