Detectron: make ops can't find header files

[PuyaYavari]

I solved the previous problem mentioned in https://github.com/facebookresearch/Detectron/issues/634 you can check there for details. But after I successfully sat up Python modules (sudo make), when I was trying to build custom operators library I faced some issues. I solved some but I couldn't find a solution for this problem. Basically Python can't find Caffe2 header files.

Error details:

ubuntu@ip-172-31-19-83:~/densepose$ make ops
mkdir -p build && cd build && cmake .. && make -j4
-- Caffe2: Found protobuf with old-style protobuf targets.
-- Caffe2: Protobuf version
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s): 3.7
-- Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37
-- Summary:
-- CMake version : 3.5.1
-- CMake command : /usr/bin/cmake
-- System name : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 0.8.2
-- Caffe2 include path : /home/include
-- Caffe2 found CUDA : True
-- CUDA version : 9.0
-- CuDNN version : 7.1.4
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/densepose/build
make[1]: Entering directory '/home/ubuntu/densepose/build'
make[2]: Entering directory '/home/ubuntu/densepose/build'
make[3]: Entering directory '/home/ubuntu/densepose/build'
make[3]: Entering directory '/home/ubuntu/densepose/build'
[ 50%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o
[ 50%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o
make[3]: Leaving directory '/home/ubuntu/densepose/build'
make[3]: Entering directory '/home/ubuntu/densepose/build'
[ 50%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o
[ 50%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o
In file included from /home/ubuntu/densepose/detectron/ops/pool_points_interp.cc:10:0:
/home/ubuntu/densepose/detectron/ops/pool_points_interp.h:13:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
In file included from /home/ubuntu/densepose/detectron/ops/zero_even_op.cc:9:0:
/home/ubuntu/densepose/detectron/ops/zero_even_op.h:12:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:86: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o' failed
make[3]: * [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o] Error 1
make[3]: Waiting for unfinished jobs....
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o' failed
make[3]: [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o] Error 1
make[3]: Leaving directory '/home/ubuntu/densepose/build'
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/all' failed
make[2]: [CMakeFiles/caffe2_detectron_custom_ops.dir/all] Error 2
make[2]: ** Waiting for unfinished jobs....
/home/ubuntu/densepose/detectron/ops/zero_even_op.cu:9:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
/home/ubuntu/densepose/detectron/ops/pool_points_interp.cu:11:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:219 (message):
Error generating
/home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

CMake Error at caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o.cmake:219 (message):
Error generating
/home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o' failed
make[3]: * [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o] Error 1
make[3]: Waiting for unfinished jobs....
CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:70: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o' failed
make[3]: [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o] Error 1
make[3]: Leaving directory '/home/ubuntu/densepose/build'
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all' failed
make[2]: [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all] Error 2
make[2]: Leaving directory '/home/ubuntu/densepose/build'
Makefile:127: recipe for target 'all' failed
make[1]: [all] Error 2
make[1]: Leaving directory '/home/ubuntu/densepose/build'
Makefile:13: recipe for target 'ops' failed
make: * [ops] Error 2

System information

• Operating system: Ubuntu 16.04
• Compiler version: ?
• CUDA version: CUDA 9
• cuDNN version: 7.1.2
• NVIDIA driver version:
• GPU models (for all devices if they are not all the same):
  ubuntu@ip-172-31-19-83:~$ nvidia-smi
  Sat Sep 8 17:14:30 2018
  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 396.37 Driver Version: 396.37 |
  |-------------------------------+----------------------+----------------------+
  | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
  | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
  |===============================+======================+======================|
  | 0 Tesla K80 On | 00000000:00:1E.0 Off | 0 |
  | N/A 36C P8 29W / 149W | 0MiB / 11441MiB | 0% Default |
  +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

• PYTHONPATH environment variable: /home/ubuntu/pytorch/build
• python --version output: Python 2.7.12

[vkhalidov]

Does Caffe2 include path look correct to you (mentioned in CMake output: Caffe2 include path : /home/include)? Are the mentioned Caffe2 header files located in this folder?

[PuyaYavari]

@vkhalidov yes I checked it and the include path is problematic, but how can I change it and which directory should I set it to?

[vkhalidov]

You should first determine which Caffe2 CMake module is found and used to configure Detectron. You can see that include path in Caffe2 is set relative to its CMake module location: see here, or directly in your Caffe2 CMake module which is found by CMake. It seems like your Caffe2 install is broken - it might contain traces of previous installs.

[PuyaYavari]

The caffe2 installation is on a newly opened instance, so it can't be a previous installs problem, I can't really understand what's wrong with it, will try reinstalling caffe2 and will inform you.
Thanks for your reply

[PuyaYavari]

I reinstalled everything on a new instance and when everything was ok untill make ops, when I did sudo make ops it asked for CMAKE_INCLUDE_PATH, I set that to where the Caffe2Config.cmake is and now this error occurred:

ubuntu@ip-172-31-30-108:~/densepose$ sudo make ops
mkdir -p build && cd build && cmake .. && make -j4
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
CMake Error at /usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2/public/utils.cmake:189 (cuda_select_nvcc_arch_flags):
Unknown CMake command "cuda_select_nvcc_arch_flags".
Call Stack (most recent call first):
/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2/public/cuda.cmake:332 (torch_cuda_get_nvcc_gencode_flag)
/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
CMakeLists.txt:8 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/ubuntu/densepose/CMakeFiles/CMakeOutput.log".
See also "/home/ubuntu/densepose/CMakeFiles/CMakeError.log".
Makefile:13: recipe for target 'ops' failed
make: * [ops] Error 1

[PuyaYavari]

I fixed the previous issue by upgrading cmake now a new error occurred (I should run make again I guess and that also gives an detectron error):

ubuntu@ip-172-31-30-108:~/densepose$ sudo make ops
mkdir -p build && cd build && cmake .. && make -j4
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "7.0")
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s): 3.7
-- Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37
-- Summary:
-- CMake version : 3.12.2
-- CMake command : /usr/local/bin/cmake
-- System name : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 0.8.2
-- Caffe2 include path : /usr/local/lib/python2.7/dist-packages/torch/include
-- Caffe2 found CUDA : True
-- CUDA version : 9.0
-- CuDNN version : 7.1.4
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/densepose
make[1]: Entering directory '/home/ubuntu/densepose/build'
make[1]: * No targets specified and no makefile found. Stop.
make[1]: Leaving directory '/home/ubuntu/densepose/build'
Makefile:13: recipe for target 'ops' failed
make: * [ops] Error 2

[vkhalidov]

Have you tried make ops without sudo? Your CMake log looks strange. It says

-- Build files have been written to: /home/ubuntu/densepose

However, build files should have been written to /home/ubuntu/densepose/build. No wonder the next line mentions that no makefile was found. I also wonder, if CMake has written build files to the source dir, what happened to the original make files.

[PuyaYavari]

I get permission errors without sudo, I know it can cause some PATH issues but not sure if it caused by that.
I removed and cloned densepose again, this is the entire process:

ubuntu@ip-172-31-30-108:~/densepose$ sudo make
python2 setup.py develop --user
Compiling detectron/utils/cython_bbox.pyx because it changed.
Compiling detectron/utils/cython_nms.pyx because it changed.
[1/2] Cythonizing detectron/utils/cython_bbox.pyx
[2/2] Cythonizing detectron/utils/cython_nms.pyx
running develop
running egg_info
creating Detectron.egg-info
writing Detectron.egg-info/PKG-INFO
writing top-level names to Detectron.egg-info/top_level.txt
writing dependency_links to Detectron.egg-info/dependency_links.txt
writing manifest file 'Detectron.egg-info/SOURCES.txt'
reading manifest file 'Detectron.egg-info/SOURCES.txt'
writing manifest file 'Detectron.egg-info/SOURCES.txt'
running build_ext
building 'detectron.utils.cython_bbox' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/detectron
creating build/temp.linux-x86_64-2.7/detectron/utils
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c detectron/utils/cython_bbox.c -o build/temp.linux-x86_64-2.7/detectron/utils/cython_bbox.o -Wno-cpp
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/detectron
creating build/lib.linux-x86_64-2.7/detectron/utils
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/detectron/utils/cython_bbox.o -o build/lib.linux-x86_64-2.7/detectron/utils/cython_bbox.so
building 'detectron.utils.cython_nms' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c detectron/utils/cython_nms.c -o build/temp.linux-x86_64-2.7/detectron/utils/cython_nms.o -Wno-cpp
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/detectron/utils/cython_nms.o -o build/lib.linux-x86_64-2.7/detectron/utils/cython_nms.so
copying build/lib.linux-x86_64-2.7/detectron/utils/cython_bbox.so -> detectron/utils
copying build/lib.linux-x86_64-2.7/detectron/utils/cython_nms.so -> detectron/utils
Creating /home/ubuntu/.local/lib/python2.7/site-packages/Detectron.egg-link (link to .)
Detectron 0.0.0 is already the active version in easy-install.pth

Installed /home/ubuntu/densepose
Processing dependencies for Detectron==0.0.0
Finished processing dependencies for Detectron==0.0.0
ubuntu@ip-172-31-30-108:~/densepose$ python detectron/tests/test_spatial_narrow_as_op.py
[E init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
[E init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
Found Detectron ops lib: /home/ubuntu/pytorch/build/lib/libcaffe2_detectron_ops_gpu.so

...

Ran 3 tests in 2.173s

OK
ubuntu@ip-172-31-30-108:~/densepose$ sudo make ops
mkdir -p build && cd build && cmake .. && make -j4
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (find_package):
By not providing "FindCaffe2.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Caffe2", but
CMake did not find one.

Could not find a package configuration file provided by "Caffe2" with any
of the following names:

Caffe2Config.cmake
caffe2-config.cmake

Add the installation prefix of "Caffe2" to CMAKE_PREFIX_PATH or set
"Caffe2_DIR" to a directory containing one of the above files. If "Caffe2"
provides a separate development package or SDK, be sure it has been
installed.

-- Configuring incomplete, errors occurred!
See also "/home/ubuntu/densepose/build/CMakeFiles/CMakeOutput.log".
Makefile:13: recipe for target 'ops' failed
make: * [ops] Error 1

ubuntu@ip-172-31-30-108:~/densepose$ cmake -DCMAKE_PREFIX_PATH=/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "7.0")
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found CUDNN: /usr/local/cuda/include
-- Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s): 3.7
-- Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37
-- Summary:
-- CMake version : 3.12.2
-- CMake command : /usr/local/bin/cmake
-- System name : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 0.8.2
-- Caffe2 include path : /usr/local/lib/python2.7/dist-packages/torch/include
-- Caffe2 found CUDA : True
-- CUDA version : 9.0
-- CuDNN version : 7.1.4
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/densepose

ubuntu@ip-172-31-30-108:~/densepose$ sudo make ops
make: * No rule to make target 'ops'. Stop.
ubuntu@ip-172-31-30-108:~/densepose$ sudo make
Scanning dependencies of target caffe2_detectron_custom_ops
[ 12%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o
In file included from /home/ubuntu/densepose/detectron/ops/pool_points_interp.cc:10:0:
/home/ubuntu/densepose/detectron/ops/pool_points_interp.h:13:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o' failed
make[2]: [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/all' failed
make[1]: [CMakeFiles/caffe2_detectron_custom_ops.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: * [all] Error 2

[vkhalidov]

OK, I see. I suggest that instead of doing sudo make ops, you perform the corresponding commands manually. First, you need to verify that your DensePose sources are clean. Then from your DensePose source directory ~/densepose you need to run the following commands

1. mkdir build
2. cd build
3. cmake .. -DCaffe2_DIR=/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2
4. make -j4

[PuyaYavari]

ubuntu@ip-172-31-30-108:~$ cd densepose/
ubuntu@ip-172-31-30-108:~/densepose$ mkdir build
ubuntu@ip-172-31-30-108:~/densepose$ cd build/
ubuntu@ip-172-31-30-108:~/densepose/build$ cmake .. -DCaffe2_DIR=/usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "7.0")
-- Caffe2: CUDA detected: 9.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 9.0
-- Found CUDNN: /usr/local/cuda/include
-- Found cuDNN: v7.1.4 (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn.so)
-- Autodetected CUDA architecture(s): 3.7
-- Added CUDA NVCC flags for: -gencode;arch=compute_37,code=sm_37
-- Summary:
-- CMake version : 3.12.2
-- CMake command : /usr/local/bin/cmake
-- System name : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 0.8.2
-- Caffe2 include path : /usr/local/lib/python2.7/dist-packages/torch/include
-- Caffe2 found CUDA : True
-- CUDA version : 9.0
-- CuDNN version : 7.1.4
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/densepose/build
ubuntu@ip-172-31-30-108:~/densepose/build$ make -j4
[ 12%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o
Scanning dependencies of target caffe2_detectron_custom_ops
[ 25%] Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o
[ 37%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o
[ 50%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o
In file included from /home/ubuntu/densepose/detectron/ops/zero_even_op.cc:9:0:
/home/ubuntu/densepose/detectron/ops/zero_even_op.h:12:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
In file included from /home/ubuntu/densepose/detectron/ops/pool_points_interp.cc:10:0:
/home/ubuntu/densepose/detectron/ops/pool_points_interp.h:13:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:75: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o' failed
make[2]: * [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o] Error 1
make[2]: Waiting for unfinished jobs....
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o' failed
make[2]: [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/all' failed
make[1]: [CMakeFiles/caffe2_detectron_custom_ops.dir/all] Error 2
make[1]: ** Waiting for unfinished jobs....
/home/ubuntu/densepose/detectron/ops/zero_even_op.cu:9:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:219 (message):
Error generating
/home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:70: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o' failed
make[2]: * [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o] Error 1
make[2]: * Waiting for unfinished jobs....
/home/ubuntu/densepose/detectron/ops/pool_points_interp.cu:11:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMake Error at caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o.cmake:219 (message):
Error generating
/home/ubuntu/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o' failed
make[2]: * [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o] Error 1
CMakeFiles/Makefile2:109: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all' failed
make[1]: [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: ** [all] Error 2

[PuyaYavari]

can there be a problem with my Caffe2_DIR?
my output for find -name Caffe2Config.cmake is:

ubuntu@ip-172-31-30-108:/$ sudo find -name Caffe2Config.cmake
./usr/local/lib/python2.7/dist-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake
./home/ubuntu/pytorch/torch/lib/tmp_install/share/cmake/Caffe2/Caffe2Config.cmake
./home/ubuntu/pytorch/torch/share/cmake/Caffe2/Caffe2Config.cmake
./home/ubuntu/pytorch/build/Caffe2Config.cmake
./home/ubuntu/pytorch/build/lib.linux-x86_64-2.7/torch/share/cmake/Caffe2/Caffe2Config.cmake

[JiangChenrui]

I have the same problem, do you have solve it?

[PuyaYavari]

@holyhond I don't

[boycehbz]

I have the same problem as your first error, and I change the caffe2 include path in Caffe2Config.cmake. But it seems useless.

[PuyaYavari]

I have been trying to install Densepose for 2-3 months and I tried it on a couple of different instances and this is as far as I got, every time sticking at places for weeks until I figure out that it's about something missing in installation guide or is some problem in development process (as in cmake issue where it only works with latest cmake). So I kind of gave up on this installation.

[boycehbz]

have you try to install it by docker?

[PuyaYavari]

I tried so many things that I don't even remember all of them, but guess I tried that one too if I'm not wrong.
If it worked for you inform me so I try it again

[zakdances]

Having the same issue but I'm just trying to make Detectron work, no densepose.

[ricardo-0x07]

Just started but stuck on the same issue:
mkdir -p build && cd build && cmake .. && make -j4
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "10.0", minimum required is "7.0")
-- Caffe2: CUDA detected: 10.0
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 10.0
-- Found CUDNN: /usr/include
-- Found cuDNN: v5.1.10 (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Autodetected CUDA architecture(s): 6.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60
-- Summary:
-- CMake version : 3.11.3
-- CMake command : /usr/bin/cmake
-- System name : Linux
-- C++ compiler : /usr/bin/c++
-- C++ compiler version : 5.4.0
-- CXX flags : -std=c++11 -O2 -fPIC -Wno-narrowing
-- Caffe2 version : 1.0.0
-- Caffe2 include path : /home/clivecadogan/anaconda2/lib/python2.7/site-packages/torch/lib/include
-- Caffe2 found CUDA : True
-- CUDA version : 10.0
-- CuDNN version : 5.1.10
-- Configuring done
-- Generating done
-- Build files have been written to: /home/clivecadogan/densepose/build
makeÆ1Å: Entering directory '/home/clivecadogan/densepose/build'
makeÆ2Å: Entering directory '/home/clivecadogan/densepose/build'
makeÆ3Å: Entering directory '/home/clivecadogan/densepose/build'
makeÆ3Å: Entering directory '/home/clivecadogan/densepose/build'
Scanning dependencies of target caffe2_detectron_custom_ops
Æ 12%Å Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o
Æ 25%Å Building NVCC (Device) object CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o
makeÆ3Å: Leaving directory '/home/clivecadogan/densepose/build'
makeÆ3Å: Entering directory '/home/clivecadogan/densepose/build'
Æ 37%Å Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o
Æ 50%Å Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o
In file included from /home/clivecadogan/densepose/detectron/ops/zero_even_op.cc:9:0:
/home/clivecadogan/densepose/detectron/ops/zero_even_op.h:12:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:75: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o' failed
makeÆ3Å: * ÆCMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.oÅ Error 1
makeÆ3Å: Waiting for unfinished jobs....
In file included from /home/clivecadogan/densepose/detectron/ops/pool_points_interp.cc:10:0:
/home/clivecadogan/densepose/detectron/ops/pool_points_interp.h:13:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.o' failed
makeÆ3Å: ÆCMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/pool_points_interp.cc.oÅ Error 1
makeÆ3Å: Leaving directory '/home/clivecadogan/densepose/build'
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/all' failed
makeÆ2Å: ÆCMakeFiles/caffe2_detectron_custom_ops.dir/allÅ Error 2
makeÆ2Å: ** Waiting for unfinished jobs....
/home/clivecadogan/densepose/detectron/ops/zero_even_op.cu:9:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:219 (message):
Error generating
/home/clivecadogan/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

/home/clivecadogan/densepose/detectron/ops/pool_points_interp.cu:11:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:70: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o' failed
makeÆ3Å: * ÆCMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.oÅ Error 1
makeÆ3Å: * Waiting for unfinished jobs....
CMake Error at caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o.cmake:219 (message):
Error generating
/home/clivecadogan/densepose/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.o' failed
makeÆ3Å: * ÆCMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_pool_points_interp.cu.oÅ Error 1
makeÆ3Å: Leaving directory '/home/clivecadogan/densepose/build'
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all' failed
makeÆ2Å: ÆCMakeFiles/caffe2_detectron_custom_ops_gpu.dir/allÅ Error 2
makeÆ2Å: Leaving directory '/home/clivecadogan/densepose/build'
Makefile:129: recipe for target 'all' failed
makeÆ1Å: ÆallÅ Error 2
makeÆ1Å: Leaving directory '/home/clivecadogan/densepose/build'
Makefile:13: recipe for target 'ops' failed
make: * ÆopsÅ Error 2

[htzheng]

stuck on the same thing for 1 week, couldn't find a solution.

[zaoqiweixaio]

First of all, thank facebook researchers for making their excellent work open.
I am new to linux and deep learning, so please forgive me for my unprofessional answer. I finally succeed to run densepose demo. Hope any point helps.
I installed caffe2 from source with the following link https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile (tried using anaconda to install at first, but failed for some later operations).

System: Ubuntu 16.04, Cuda 8.0, cuDNN 6.0, git the latest code and related things such as caffe2(pytorch).

Contents added at the end of ~/.bashrc:
export PYTHONPATH=/usr/local:$PYTHONPATH
export PYTHONPATH=$PYTHONPATH:/home/username/pytorch/build
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/:$LIBRARY_PATH

export C_INCLUDE_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/include/:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/local/lib/python2.7/dist-packages/torch/lib/include/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/home/username/pytorch/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/build/:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/home/username/pytorch/build/:$CPLUS_INCLUDE_PATH

export C_INCLUDE_PATH=/home/username/pytorch/torch/lib/include/:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/home/username/pytorch/torch/lib/include/:$CPLUS_INCLUDE_PATH

I know it is unprofessional to add so many paths for C and CPlus include path. I get a little crazy when stucking. Whenever cannot finding a .h file, I locate the exact filename (better using the output information, e.g., XX/XX/XX.h) and add a path.

After "make -j4" succeeds, python2 $DENSEPOSE/detectron/tests/test_zero_even_op.py failes. I encounter the error: densepose/build/libcaffe2_detectron_custom_ops_gpu.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl28AllocateAlignedAndAddCleanupEmPFvPvE.
Solved by "hyounsamk" in this link https://github.com/facebookresearch/DensePose/issues/119

[hanabi7]

stuck on the same thing for a month，couldn.t really find a solution

[shaljam]

@hanabi7 Have you tried https://github.com/facebookresearch/detectron2?