Detectron: Error during test_spatial_narrow_as_op.py while installing

[PardoAlejo]

System information

• Operating system: Ubuntu 16.04.3 LTS
• Compiler version: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)
• CUDA version: CUDA 8.0
• cuDNN version: cuDNN 6.0
• NVIDIA driver version: nvidia-387
• GPU models (for all devices if they are not all the same): Titan X
• PYTHONPATH environment variable: /usr/local/lib/python2.7
• python --version output: Python 2.7.12

Hi,

I've been trying to install detectron in my machine. After following all steps required in install.md I check te installation by running python2 $DETECTRON/tests/test_spatial_narrow_as_op.py. And I get the following error:

ERROR: test_large_forward (__main__.SpatialNarrowAsOpTest)

Traceback (most recent call last):
File "test_spatial_narrow_as_op.py", line 68, in test_large_forward
self._run_test(A, B)
File "test_spatial_narrow_as_op.py", line 37, in _run_test
workspace.FeedBlob('A', A)
File "/usr/local/caffe2/python/workspace.py", line 322, in FeedBlob
return C.feed_blob(name, arr, StringifyProto(device_option))
RuntimeError: [enforce fail at common_cudnn.h:118] version_match. cuDNN compiled (7005) and runtime (7003) versions mismatch

Could anyone help me to figure out what is wrong with the installation?

[ir413]

Hi @PardoAlejo, it seems that your cuDNN versions are getting mixed. Please ensure that the same cuDNN version is used to compile Caffe2 and at runtime.

You may want to check that your $LD_LIBRARY_PATH does not point to a cuDNN version different from the one used to compile Caffe2 (if it does, try removing it).

[EP777]

Look for the compiled version, for 7005, use - Download cuDNN v7.0.5 (Dec 5, 2017), for CUDA 9.0 version

[ambigus9]

Hi, I install Caffe2 from Anaconda Pre-build Binaries with CUDA 8.0 and cuDNN 7.1.2 and got this error:

RuntimeError: [enforce fail at common_cudnn.h:118] version_match. cuDNN compiled (7101) and runtime (7102) versions mismatch

How can I "uninstall" or use the same version? I tried

sudo dpkg -i libcudnn7_7.1.1.5-1+cuda8.0_amd64.deb
sudo dpkg -i libcudnn7-dev_7.1.1.5-1+cuda8.0_amd64.deb

and also,

tar -xzvf cudnn-8.0-linux-x64-v7.1.tgz

sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

But doesn't work

[lupotto]

Hi,

I am having the exact problem as @ambigus9.

[jhonygiraldo]

Hi,

I have the same problem of @ambigus9 too.

[ambigus9]

@lupotto and @jhonygiraldo Please download the tar file wich this version cuDNN v7.1.1 Library for Linux from cuDNN using wget (click on the file to download it, and copy the download link an use wget) because for somereason download it from a browser doesn't download well, and make this steps:

tar -xzvf cudnn-8.0-linux-x64-v7.1.tgz

sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

After this you should check your version of cuDNN,

cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

Hope to help.

Note: It's better to build a Docker image using Dockerfile.

[lupotto]

Hi @ambigus9,

Thank you for sharing this. But I still stuck in the same problem, I will try with Docker.

[IgorKasianenko]

Hi, I'm having similar issue. I'm using env to run another framework.
I don't use conda or docker to install caffe2 and Detectron.
Error as follows version_match. cuDNN compiled (7102) and runtime (7005) versions mismatch

What is better solution - recompile caffe2 and Detectron with version 7005 that i need for another project or create virual env like conda for Detectron and use latest version of cuDNN?

[jhonygiraldo]

Hi @IgorKasianenko,

I think that the error message is a little bit confusing. You should uninstall caffe2, then install cuDNN v7.1.2 (Mar 21, 2018), for CUDA 9.0 or cuDNN v7.1.2 (Mar 21, 2018), for CUDA 9.1 (see your CUDA version). Finally, you should install again Caffe2.

[lupotto]

Hi guys,

As @ambigus9 said I solved the problem using Docker. I tried everything and I didn't find out any other solution with the mismatch of versions.

I explain the steps that I followed if someone is not familiar with docker

1. Install docker. (https://docs.docker.com/install/)
2. Install nvidia-docker (https://github.com/NVIDIA/nvidia-docker)
3. Follow the steps of (https://github.com/facebookresearch/Detectron/blob/master/INSTALL.md)
3.1 Clone repo
3.2 Install docker image & run the test image
4. After that, start the container associated with the image: sudo nvidia-docker container run --rm -it detectron:c2-cuda9-cudnn7 bash
5. Now, inside the container you can run the demo: python2 tools/infer_simple.py \
--cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml \
--output-dir /tmp/detectron-visualizations \
--image-ext jpg \
--wts https://s3-us-west-2.amazonaws.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl \
demo

I hope you find it helpful

[xingkongliang]

I find a cuDNN v7.1.2 in my an conda, so I uninstall this cudnn
conda uninstall cudnn
after that, I rebuid caffe2, I pass it.

[mbajaj01]

Facing same issue. Any solution for this without using docker?