Detectron: CUDA_ARCH_NAME error with conda version of caffe2

[dhpollack]

Expected results

cd lib
make
make ops

This should work

Actual results

# make ops

mkdir -p build && cd build && cmake .. && make -j4
-- Caffe2: Found gflags with new-style gflags target.
-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Caffe2: Found glog (include: /opt/conda/include, library: /opt/conda/lib/libglog.so)
-- Caffe2: Found protobuf with old-style protobuf targets.
-- Caffe2: Protobuf version 
-- Caffe2: CUDA detected: 9.0
-- Found cuDNN: v7.0.5  (include: /usr/include, library: /usr/lib/x86_64-linux-gnu/libcudnn.so)
-- Automatic GPU detection returned 3.7.
-- Added CUDA NVCC flags for: sm_37
-- CUDA detected: 9.0
CMake Error at cmake/Cuda.cmake:32 (message):
  Invalid CUDA_ARCH_NAME, supported values: Kepler, Maxwell, Pascal, Volta,
  All, Manual.  Got Auto
Call Stack (most recent call first):
  cmake/Cuda.cmake:183 (caffe2_select_nvcc_arch_flags)
  cmake/Dependencies.cmake:11 (include)
  CMakeLists.txt:14 (include)


-- Configuring incomplete, errors occurred!
See also "/opt/detectron/lib/build/CMakeFiles/CMakeOutput.log".
See also "/opt/detectron/lib/build/CMakeFiles/CMakeError.log".
Makefile:13: recipe for target 'ops' failed
make: *** [ops] Error 1

Detailed steps to reproduce

Install caffe2 via conda as recommended on the caffe2 website and then follow the install instructions for Detectron

System information

• Operating system: ubuntu 16.04 (nvidia/cuda:9.0-devel-ubuntu16.04 docker image to be exact)
• Compiler version: gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
• CUDA version: 9.0
• cuDNN version: 7.0.5.15
• NVIDIA driver version: 390.30
• GPU models (for all devices if they are not all the same): Kepler (did not try others)
• PYTHONPATH environment variable: /opt/conda/bin/python (also tried /opt/conda and /opt/conda/lib/python2.7/site-packages/caffe2 and combinations of these)
• python --version output: 2.7
• Anything else that seems relevant: Seems like caffe2 allows for the "Auto" arch name which is not taken into account by Detectron. See cuda.cmake in the caffe2 repo.

[rbgirshick]

@ir413 @pietern @Yangqing: seems like the detectron cuda related cmake files may have gotten out of sync with caffe2's. IIRC, the C2 side was going to provide some way for external projects to make use of their cuda cmake files to make compiling custom ops easier. Is that still the plan? Thanks.

[manyaafonso]

So what is the solution? I have the exact same problem.

[dhpollack]

I used the Dockerfile in this repo. I also tried compiling caffe2 manually setting the cuda arch name but the docket was easier.

[orionr]

We're investigating, but for those that need to get unblocked you can explicitly set the CUDA_ARCH_NAME during cmake as outlined at https://github.com/caffe2/caffe2/issues/2191

[Yangqing]

@rbgirshick - sent a PR to address this issue. Basically, I had to keep the existing legacy path to not disrupt current users who are still on an earlier version of Caffe2. For whoever that is installing Caffe2 current head version (or any version post 0.8.2), they will be on the new path. Maybe we can remove the legacy part in a few months when more users are upgrading C2.

[sdalxn]

I have once installed detectron under caffe2 0.8.1 successfully. But when I delete it and want to use caffe2 0.8.2. I have met this exact issue on the same machine.

Do I need to reinstall caffe2 0,8,1?

[walter040422]

Hi, I met with the same problem. What is the solution please?

[twmht]

Any update?

[arasharchor]

@walter040422 @twmht
simply add the following line in your cmake
-DCUDA_ARCH_NAME=Pascal

[ir413]

Addressed by 8f2f5cf.