Openpose: Issue running the examples on Ubuntu 16.04 : Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

[TheRobotStudio]

Hello,

After successful install of the openpose, when I try to run the first example :
$ ./build/examples/openpose/openpose.bin --video examples/media/video.avi --logging_level 0
it fails with this error message:

Starting pose estimation demo.
src/openpose/utilities/flagsToOpenPose.cpp:flagsToProducer():95
src/openpose/utilities/flagsToOpenPose.cpp:flagsToProducerType():66
src/openpose/utilities/flagsToOpenPose.cpp:flagsToPoseModel():14
src/openpose/utilities/flagsToOpenPose.cpp:flagsToScaleMode():38
examples/openpose/openpose.cpp:openPoseDemo():191
Configuring OpenPose wrapper. In examples/openpose/openpose.cpp:openPoseDemo():194
./include/openpose/wrapper/wrapper.hpp:configure():432
Auto-detecting GPUs... Detected 1 GPU(s), using them all.
./include/openpose/wrapper/wrapper.hpp:configure():612
./include/openpose/wrapper/wrapper.hpp:configure():838
Starting thread(s)
./include/openpose/wrapper/wrapper.hpp:configureThreadManager():1146
./include/openpose/thread/threadManager.hpp:exec():163
./include/openpose/thread/queueBase.hpp:addPusher():360
./include/openpose/thread/queueBase.hpp:addPusher():360
./include/openpose/thread/queueBase.hpp:addPusher():360
./include/openpose/thread/threadManager.hpp:exec():168
./include/openpose/thread/thread.hpp:startInThread():138
./include/openpose/thread/thread.hpp:startInThread():138
./include/openpose/thread/thread.hpp:startInThread():138
./include/openpose/thread/thread.hpp:threadFunction():182
./include/openpose/thread/thread.hpp:threadFunction():185
./include/openpose/thread/thread.hpp:threadFunction():182
./include/openpose/thread/thread.hpp:threadFunction():182
Starting initialization on thread. In src/openpose/pose/poseExtractorCaffe.cpp:netInitializationOnThread():44
./include/openpose/thread/thread.hpp:threadFunction():182
./include/openpose/thread/thread.hpp:threadFunction():185
./include/openpose/thread/thread.hpp:threadFunction():185
F0719 18:13:34.391757  1871 cudnn.hpp:122] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
    @     0x7f7dcb2225cd  google::LogMessage::Fail()
    @     0x7f7dcb224433  google::LogMessage::SendToLog()
    @     0x7f7dcb22215b  google::LogMessage::Flush()
    @     0x7f7dcb224e1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f7dc9d05388  caffe::CuDNNConvolutionLayer<>::Reshape()
    @     0x7f7dc9bfa0c2  caffe::Net<>::Init()
    @     0x7f7dc9bfcee0  caffe::Net<>::Net()
    @     0x7f7dcb4df74e  op::NetCaffe::initializationOnThread()
    @     0x7f7dcb58a64f  op::PoseExtractorCaffe::netInitializationOnThread()
    @     0x7f7dcb5749d1  op::SubThread<>::initializationOnThread()
    @     0x7f7dcb574fc0  op::Thread<>::initializationOnThread()
    @     0x7f7dcb579e1d  op::Thread<>::threadFunction()
    @     0x7f7dcad2cc80  (unknown)
    @     0x7f7dc83656ba  start_thread
    @     0x7f7dca79b3dd  clone
    @              (nil)  (unknown)
Aborted

It seems some people had this error while running other programs, but I didn't find a solution.
I use ubuntu 16.04, with CUDA 8.0 and cuDNN 5.1 with a nVIDIA GeForce GTX 1060 6GB (pascal)
Here are a few outputs that might help finding the issue :

$ uname -r
4.4.0-83-generic

$ nvidia-smi 
Wed Jul 19 17:53:20 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 0000:01:00.0      On |                  N/A |
|  0%   56C    P8    13W / 200W |    247MiB /  6069MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1613    G   /usr/lib/xorg/Xorg                             216MiB |
|    0      3842    G   compiz                                          28MiB |
+-----------------------------------------------------------------------------+

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

$ dpkg -l | grep cuDNN
ii  libcudnn6                                                   6.0.21-1+cuda8.0                                      amd64        cuDNN runtime libraries
ii  libcudnn6-dev                                               6.0.21-1+cuda8.0                                      amd64        cuDNN development libraries and headers
ii  libcudnn6-doc                                               6.0.21-1+cuda8.0                                      amd64        cuDNN documents and samples

$ ldconfig -p | grep libcudnn
    libcudnn.so.6 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudnn.so.6
    libcudnn.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudnn.so

$ ls -l /usr/local/cuda/lib64/libcud*
-rw-r--r-- 1 root root   556000 juil.  4 11:41 /usr/local/cuda/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root       16 juil.  4 11:41 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.8.0
lrwxrwxrwx 1 root root       19 juil.  4 11:41 /usr/local/cuda/lib64/libcudart.so.8.0 -> libcudart.so.8.0.61
-rwxr-xr-x 1 root root   415432 juil.  4 11:41 /usr/local/cuda/lib64/libcudart.so.8.0.61
-rw-r--r-- 1 root root   775162 juil.  4 11:41 /usr/local/cuda/lib64/libcudart_static.a
lrwxrwxrwx 1 root root       13 juil.  4 16:53 /usr/local/cuda/lib64/libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 root root       18 juil.  4 16:53 /usr/local/cuda/lib64/libcudnn.so.5 -> libcudnn.so.5.1.10
-rwxr-xr-x 1 root root 84163560 juil.  4 16:53 /usr/local/cuda/lib64/libcudnn.so.5.1.10
-rw-r--r-- 1 root root 70364814 juil.  4 16:53 /usr/local/cuda/lib64/libcudnn_static.a

$ cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR      5
#define CUDNN_MINOR      1
#define CUDNN_PATCHLEVEL 10

Any clue on how to fix that ?
Thanks !

Type of issue

• Execution error
• Help wanted

Your system configuration

Operating system (lsb_release -a on Ubuntu): Ubuntu 16.04.2 LTS
CUDA version (cat /usr/local/cuda/version.txt in most cases): CUDA Version 8.0.61
cuDNN version: 5.1
GPU model (nvidia-smi): see command result above
Caffe version: Default from OpenPose.
OpenCV version: installed with apt-get install libopencv-dev.
Generation mode (only for Ubuntu): Makefile + Makefile.config (default).
Compiler (gcc --version on Ubuntu): 5.4.0

[gineshidalgo99]

It is some kind of cuDNN problem. I'd go with the following steps:

• Re-install Nvidia graphic drivers and re-boot PC
• Re-install CUDA and re-boot PC
• Re-copy cuDNN and re-boot PC

[TheRobotStudio]

Thanks for your reply.
It seems I had cuDNN v6 installed through apt-get, and v5.1 installed manually by copying files in cuda folder.
When I run ldd on a bin file, it is linked to both:

$ ldd ./build/examples/openpose/openpose.bin | grep cudnn
    libcudnn.so.5 => /home/cyril/cuda-8.0/lib64/libcudnn.so.5 (0x00007f5dd266d000)
    libcudnn.so.6 => not found

Do I need to install both version or can I build it only against 5.1 ?

[gineshidalgo99]

Both cuDNN are interfering each other. Please, could you uninstall both of them and reinstall only one? (I recommend 5.1 for speed). Not sure whether installing a second one did affect the original one, that's why I recommend uninstalling both first.

[TheRobotStudio]

I've uninstalled both cuDNN version and reinstalled v5.1 and it's working, thanks!
The reason was that, even though I had the correct cuDNN files in my cuda folder, when I installed the v6, it has replaced the files in /usr/lib/x86_64-linux-gnu with v6.