Maskrcnn-benchmark: fatal error: cusparse.h: No such file or directory

[LiuPearl1]

🐛 Bug

When I run the command: python setup.py build develop, it has fatal error:

In file included from /home/ubuntu/yuan/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/ROIPool_cuda.cu:3:0:
/home/ubuntu/.conda/envs/pt10/lib/python3.6/site-packages/torch/include/ATen/cuda/CUDAContext.h:12:22: fatal error: cusparse.h: No such file or directory
compilation terminated.
error: command '/usr/local/cuda-9.0/bin/nvcc' failed with exit status 1

And I also check the enviroment:

(pt10) ubuntu@dgx1-84-1:~/yuan/maskrcnn-benchmark$ conda list
# packages in environment at /home/ubuntu/.conda/envs/pt10:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl  
ca-certificates           2019.1.23                     0  
certifi                   2019.3.9                 py36_0  
cffi                      1.12.2           py36h2e261b9_1  
cudatoolkit               9.0                  h13b8566_0  
cycler                    0.10.0                    <pip>
Cython                    0.29.7                    <pip>
intel-openmp              2019.3                      199  
kiwisolver                1.0.1                     <pip>
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
matplotlib                3.0.3                     <pip>
mkl                       2019.3                      199  
mkl_fft                   1.0.10           py36ha843d7b_0  
mkl_random                1.0.2            py36hd81dba3_0  
ncurses                   6.1                  he6710b0_1  
ninja                     1.9.0                     <pip>
ninja                     1.9.0            py36hfd86e86_0  
numpy                     1.16.2                    <pip>
numpy                     1.16.2           py36h7e9f1db_0  
numpy-base                1.16.2           py36hde5b4d6_0  
openssl                   1.1.1b               h7b6447c_1  
pip                       19.0.3                   py36_0  
pycocotools               2.0.0                     <pip>
pycparser                 2.19                     py36_0  
pyparsing                 2.4.0                     <pip>
python                    3.6.8                h0371630_0  
python-dateutil           2.8.0                     <pip>
pytorch-nightly           1.1.0.dev20190417 py3.6_cuda9.0.176_cudnn7.4.2_0    pytorch
PyYAML                    5.1                       <pip>
readline                  7.0                  h7b6447c_5  
setuptools                41.0.0                   py36_0  
six                       1.12.0                    <pip>
sqlite                    3.27.2               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
tqdm                      4.31.1                    <pip>
wheel                     0.33.1                   py36_0  
xz                        5.2.4                h14c3975_4  
yacs                      0.1.6                     <pip>
zlib                      1.2.11               h7b6447c_3  

I also check the gcc version:

(pt10) ubuntu@dgx1-84-1:~/yuan/maskrcnn-benchmark$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I don't know how to deal with it. What should I do now?

[LeviViana]

What is your PyTorch's version ?

[LiuPearl1]

@LeviViana You can see that, my torch version is 1.1.0, pytorch-nightly.

[laberpalaber]

I got the same problem as well and for me it was solved after reinstalling CUDA.

[Ranchentx]

You can add this file: cusparse.h in your computer: cuda-9.0/include. It is missed when you install CUDA

[Ranchentx]

You could find the file: cusparse.h from other computer which installed completely

[LiuPearl1]

I have solved this problem. Usepip to install pytorch environment and the problem will be solved.
Run this command to install the environment: pip install torch torchvision.

[mengxia1994]

when i tried to use pip, it tells me requirement already satisfied. I think my cuda is installed correctly, cause i have ran other projects successfully.

[vikiQiu]

I got the same problem.

In file included from /xxx/awesome-semantic-segmentation-pytorch/core/nn/csrc/cuda/syncbn_cuda.cu:5:0:
/usr/local/lib/python3.5/dist-packages/torch/lib/include/ATen/cuda/CUDAContext.h:12:22: fatal error: cusparse.h: No such file or directory
compilation terminated.
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

It's caused by missing the cusparse.h in cuda directory. I get into the directory /user/local/ and find 2 cuda directory: cuda and cuda-9.0. I move the directory cuda into another directory. The problem is then solved, for it use the right command '/usr/local/cuda-9.0/bin/nvcc'.

[rogerthat94]

I had this error running my code in the tensroflow/tensorflow:1.11.0-gpu-py3 Docker container

I was able to fix this issue by installing Cuda on my host OS using these instructions.

[JivanRoquet]

Same problem here. Manually placing the file only moves the problem to the next missing file.

Re-installing CUDA serves no purpose except create a huge mess with apt screaming everywhere.

This happens on a perfectly well-working installation with CUDA and TF2.0-GPU enabled (and working fine).

[ImJustAskingDude]

Hey,

I have struggled with the same issue, and the solution is really simple:
apt-get install cuda-cusparse-dev-XX-X
For me it was:
apt-get install cuda-cusparse-dev-10-1

-dev is the package that contains cusparse.h

For further reference, here is where I've found it:
http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/

Hope this helps

[zeusbrav]

I got the same problem.

In file included from /xxx/awesome-semantic-segmentation-pytorch/core/nn/csrc/cuda/syncbn_cuda.cu:5:0:
/usr/local/lib/python3.5/dist-packages/torch/lib/include/ATen/cuda/CUDAContext.h:12:22: fatal error: cusparse.h: No such file or directory
compilation terminated.
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

It's caused by missing the cusparse.h in cuda directory. I get into the directory /user/local/ and find 2 cuda directory: cuda and cuda-9.0. I move the directory cuda into another directory. The problem is then solved, for it use the right command '/usr/local/cuda-9.0/bin/nvcc'.

Can you show me the command you use to move the directory cuda into another directory?Thx

[zeusbrav]

In file included from csrc/multi_tensor_sgd_kernel.cu:3:0:
/usr/local/lib/python3.6/dist-packages/torch/include/ATen/cuda/CUDAContext.h:6:22: fatal error: cusparse.h: No such file or directory
compilation terminated.
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1

:/usr/local# ls
bin cuda cuda-10.0 etc games include lib man sbin share src

Q: what should i do next?