Faster-rcnn.pytorch: Build lib/make.sh with python3 env. failed

Created on 24 Jan 2018 · 6Comments · Source: jwyang/faster-rcnn.pytorch

Hi Jianwei, I've tried to build this project in conda venv with python3 and cuda 9.0. But encountered compile failure while running lib/make.sh.

More build details: https://github.com/clement10601/faster-rcnn.pytorch

Env:

OS: Archlinux (4.14.8)
x86_64-conda_cos6-linux-gnu-cc: gcc version 7.2.0 (Thread model: posix)
nvcc: release 9.0, V9.0.176
Graphic card: Tesla k20m

Env var:

CPATH=/opt/cuda/include
CUDA_PATH=/opt/cuda/

Traceback

/home/kchwang/miniconda3/envs/frcnn3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -fPIC -DWITH_CUDA -I/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/kchwang/miniconda3/envs/frcnn3/include/python3.6m -c _roi_crop.c -o ./_roi_crop.o
In file included from /home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THC.h:4:0,
                 from _roi_crop.c:493:
/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC/THCGeneral.h:9:10: fatal error: cuda.h: No such file or directory
 #include "cuda.h"
          ^~~~~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
    extra_postargs)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command '/home/kchwang/miniconda3/envs/frcnn3/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 51, in _build
    dist.run_command('build_ext')
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
    1 #!/usr/bin/env bash
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command '/home/kchwang/miniconda3/envs/frcnn3/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build.py", line 36, in <module>
    ffi.build()
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 167, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 103, in _build_extension
    ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/cffi/api.py", line 690, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/cffi/recompiler.py", line 1515, in recompile
    compiler_verbose, debug)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/kchwang/miniconda3/envs/frcnn3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: CompileError: command '/home/kchwang/miniconda3/envs/frcnn3/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

Source

clement10601

Most helpful comment

@ShoufaChen
Test "compile one" first:

cd model/nms/src
echo "Compiling nms kernels by nvcc..."
nvcc -c -o nms_cuda_kernel.cu.o nms_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52
cd ../
python build.py

if it's compling correct~ you can change others like this. (Note: sm_xx depend to you gpu)

AceCoooool on 8 Mar 2018

👍3 🚀1 ❤1 🎉1

All 6 comments

Try force location of nvcc
$CUDA_PATH/bin/nvcc

#!/usr/bin/env bash

CUDA_PATH=/usr/local/cuda/

python setup.py build_ext --inplace
rm -rf build

CUDA_ARCH="-gencode arch=compute_30,code=sm_30 \
           -gencode arch=compute_35,code=sm_35 \
           -gencode arch=compute_50,code=sm_50 \
           -gencode arch=compute_52,code=sm_52 \
           -gencode arch=compute_60,code=sm_60 \
           -gencode arch=compute_61,code=sm_61"
       #-gencode=arch=compute_70,code=sm_70 "

# compile NMS
cd model/nms/src
echo "Compiling nms kernels by nvcc..."
$CUDA_PATH/bin/nvcc -c -o nms_cuda_kernel.cu.o nms_cuda_kernel.cu \
     -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CUDA_ARCH

cd ../
python build.py

# compile roi_pooling
cd ../../
cd model/roi_pooling/src
echo "Compiling roi pooling kernels by nvcc..."
$CUDA_PATH/bin/nvcc -c -o roi_pooling.cu.o roi_pooling_kernel.cu \
     -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CUDA_ARCH
cd ../
python build.py

# compile roi_align
cd ../../
cd model/roi_align/src
echo "Compiling roi align kernels by nvcc..."
$CUDA_PATH/bin/nvcc -c -o roi_align_kernel.cu.o roi_align_kernel.cu \
     -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CUDA_ARCH
cd ../
python build.py

# compile roi_crop
cd ../../
cd model/roi_crop/src
echo "Compiling roi crop kernels by nvcc..."
$CUDA_PATH/bin/nvcc -c -o roi_crop_cuda_kernel.cu.o roi_crop_cuda_kernel.cu \
     -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CUDA_ARCH
cd ../
python build.py

dereyly on 24 Jan 2018

😄1 👍1

@dereyly
Problem solved, using following command:
CPATH=/opt/cuda/include ./make.sh

clement10601 on 25 Jan 2018

👍2

Great! I am closing this issue.

jwyang on 25 Jan 2018

I come up with a similar error, but the solutions above cannot solve this problem.

/home/yongyi/anaconda3/compiler_compat/ld: cannot find -lpthread
/home/yongyi/anaconda3/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
File "/home/yongyi/anaconda3/lib/python3.6/distutils/unixccompiler.py", line 197, in link
self.spawn(linker + ld_args)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 49, in _build
dist.run_command('build_ext')
File "/home/yongyi/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/yongyi/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/home/yongyi/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/home/yongyi/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/home/yongyi/anaconda3/lib/python3.6/distutils/unixccompiler.py", line 199, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "build.py", line 35, in
ffi.build()
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 167, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 103, in _build_extension
ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/cffi/api.py", line 684, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/cffi/recompiler.py", line 1484, in recompile
compiler_verbose, debug)
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 20, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/home/yongyi/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 56, in _build
raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

YongyiTang92 on 10 Feb 2018

@YongyiTang92 I got the same problem, have you solved it

ShoufaChen on 3 Mar 2018

@ShoufaChen
Test "compile one" first:

cd model/nms/src
echo "Compiling nms kernels by nvcc..."
nvcc -c -o nms_cuda_kernel.cu.o nms_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52
cd ../
python build.py

if it's compling correct~ you can change others like this. (Note: sm_xx depend to you gpu)

AceCoooool on 8 Mar 2018

👍3 🚀1 ❤1 🎉1

Was this page helpful?

0 / 5 - 0 ratings