Centernet: bug .Compilc make.sh error

[Joincsx]

[Joincsx]

ubuntu 16.04;CUDA_v9.0;pytouch-v0.4.1
when ./make.sh
there is detail:
home/sxc/CenterNet/src/lib/models/networks/DCNv2
Including CUDA code.
/home/sxc/CenterNet/src/lib/models/networks/DCNv2
generating /tmp/tmpou481gf4/_dcn_v2.c
setting the current directory to '/tmp/tmpou481gf4'
running build_ext
building '_dcn_v2' extension
creating home
creating home/sxc
creating home/sxc/CenterNet
creating home/sxc/CenterNet/src
creating home/sxc/CenterNet/src/lib
creating home/sxc/CenterNet/src/lib/models
creating home/sxc/CenterNet/src/lib/models/networks
creating home/sxc/CenterNet/src/lib/models/networks/DCNv2
creating home/sxc/CenterNet/src/lib/models/networks/DCNv2/src
gcc -pthread -B /home/sxc/local_install/anaconda3/envs/CenterNet/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/sxc/local_install/anaconda3/envs/CenterNet/include/python3.6m -c _dcn_v2.c -o ./_dcn_v2.o -std=c99 -fopenmp -std=c99
In file included from /home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THC.h:4:0,
from _dcn_v2.c:493:
/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THCGeneral.h:12:18: fatal error: cuda.h: 没有那个文件或目录
compilation terminated.
Traceback (most recent call last):
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
depends=ext.depends)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "build.py", line 43, in
ffi.build()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 189, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 111, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.VerificationError: CompileError: command 'gcc' failed with exit status 1
Including CUDA code.
/home/sxc/CenterNet/src/lib/models/networks/DCNv2
generating /tmp/tmpw5egiwkr/_dcn_v2_double.c
setting the current directory to '/tmp/tmpw5egiwkr'
running build_ext
building '_dcn_v2_double' extension
creating home
creating home/sxc
creating home/sxc/CenterNet
creating home/sxc/CenterNet/src
creating home/sxc/CenterNet/src/lib
creating home/sxc/CenterNet/src/lib/models
creating home/sxc/CenterNet/src/lib/models/networks
creating home/sxc/CenterNet/src/lib/models/networks/DCNv2
creating home/sxc/CenterNet/src/lib/models/networks/DCNv2/src
gcc -pthread -B /home/sxc/local_install/anaconda3/envs/CenterNet/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/sxc/local_install/anaconda3/envs/CenterNet/include/python3.6m -c _dcn_v2_double.c -o ./_dcn_v2_double.o -std=c99 -fopenmp -std=c99
In file included from /home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THC.h:4:0,
from _dcn_v2_double.c:493:
/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THCGeneral.h:12:18: fatal error: cuda.h: 没有那个文件或目录
compilation terminated.
Traceback (most recent call last):
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
depends=ext.depends)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "build_double.py", line 43, in
ffi.build()
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 189, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 111, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/home/sxc/local_install/anaconda3/envs/CenterNet/lib/python3.6/site-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.VerificationError: CompileError: command 'gcc' failed with exit status 1

please help me,thanks.

[xingyizhou]

Hi,
I haven't met this before. It is more likely that you haven't set the cuda path correctly. Can you check this https://github.com/locuslab/pytorch_fft/issues/21 ? Also, can you make sure your nvcc --version works as expected and your pytorch version corresponds to your cuda versoin?

[Joincsx]

@xingyizhou
my cuda version is cuda8 and pytorch version is pytorch0.4.1. is it right?
And what is the right‘s cuda path?
Thank you!

[Joincsx]

@xingyizhou
I have add the cuda path to ./bashrc.
It is my ./bashrc about cuda.

cuda

export PATH=~/local_install/cuda8/bin:$PATH
export LD_LIBRARY_PATH=~/local_install/cuda8/lib64:$LD_LIBRARY_PATH

[xingyizhou]

Hi,
I am suspecting that the problem might come from CUDA 8.0. Can you try CUDA 9.0? CUDA 8.0 is also much slower than CUDA 9.0 as far as I know.

[Joincsx]

@xingyizhou I have try the CUDA 9.0,but it is faild.The error is the same.

[Joincsx]

@xingyizhou
Hello,I try update my pytorch to ptorch 1.0 and I use the DCNv2 from https://github.com/CharlesShang/DCNv2. Luckily，it work well.
I do not know why?But I have save my problem.Thanks!

[Ranchentx]

@Joincsx
Hello, I follow your instruction to install ptorch 1.0 and use DCNv2 from https://github.com/CharlesShang/DCNv2. But when I compile make.sh, I met a problem:

home/ubuntu/.conda/envs/CenterNet/compiler_compat/ld: build/temp.linux-x86_64-3.6/raid/jing/data/CenterNet/src/lib/models/networks/DCNv2/src/cuda/dcn_v2_im2col_cuda_double.o: in function modulated_deformable_col2im_coord_cuda': tmpxft_00009f12_00000000-5_dcn_v2_im2col_cuda_double.cudafe1.cpp:(.text+0x36b): multiple definition ofmodulated_deformable_col2im_coord_cuda'; build/temp.linux-x86_64-3.6/raid/jing/data/CenterNet/src/lib/models/networks/DCNv2/src/cuda/dcn_v2_im2col_cuda.o:tmpxft_00009edd_00000000-5_dcn_v2_im2col_cuda.cudafe1.cpp:(.text+0x36b): first defined here
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

[Joincsx]

@xingyizhou
Is it the problem of gcc version? my gcc version is gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11).

[Joincsx]

@Ranchentx
Hello.
My cuda version is CUDA 9 , python version is python 3.6,gcc version is gcc 5.4.
You can check you enveronment .

[xingyizhou]

My gcc is also 5.4.

[Hrhythm]

@xingyizhou
Hello,I try update my pytorch to ptorch 1.0 and I use the DCNv2 from https://github.com/CharlesShang/DCNv2. Luckily，it work well.
I do not know why?But I have save my problem.Thanks!

I follow your advice,but I met this error:
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:416

[xingyizhou]

I found this. It says For RTX 2080, you need to use CUDA10, not CUDA 9.2.

[Hrhythm]

I found this. It says For RTX 2080, you need to use CUDA10, not CUDA 9.2.
Thanks for your reply,actually I am using CUDA 10. But I still met this error.

[haoopan]

Add below on the head of make.sh,which can solve this problem!
export CUDA_PATH=/usr/local/cuda/
export CXXFLAGS="-std=c++11"
export CFLAGS="-std=c99"

export PATH=/usr/local/cuda-8.0/bin${PATH:+:${PATH}}
export CPATH=/usr/local/cuda-8.0/include${CPATH:+:${CPATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

[FranciscoReveriano]

I am using gcc 6 and CUDA 10.1 on Ubuntu 18.04
For some reason I keep on getting the:
command '/usr/local/cuda/bin/nvcc' failed with exit status 1 ?

Any ideas?

[fliman]

@Joincsx
Hello, I follow your instruction to install ptorch 1.0 and use DCNv2 from https://github.com/CharlesShang/DCNv2. But when I compile make.sh, I met a problem:

home/ubuntu/.conda/envs/CenterNet/compiler_compat/ld: build/temp.linux-x86_64-3.6/raid/jing/data/CenterNet/src/lib/models/networks/DCNv2/src/cuda/dcn_v2_im2col_cuda_double.o: in function modulated_deformable_col2im_coord_cuda': tmpxft_00009f12_00000000-5_dcn_v2_im2col_cuda_double.cudafe1.cpp:(.text+0x36b): multiple definition ofmodulated_deformable_col2im_coord_cuda'; build/temp.linux-x86_64-3.6/raid/jing/data/CenterNet/src/lib/models/networks/DCNv2/src/cuda/dcn_v2_im2col_cuda.o:tmpxft_00009edd_00000000-5_dcn_v2_im2col_cuda.cudafe1.cpp:(.text+0x36b): first defined here
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

how do you solve this problem?