Maskrcnn-benchmark: is cuda 10.2 is suported ?

[mrtechnoo]

[xiayanqian]

So...is CUDA 10.2 supported?
I run the code and encounter the problem:
.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 416, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
I add add torch.backends.cudnn.enabled = False to disable cudnn,
then another problem:
.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 416, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling 'cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)'

It seems that CUDA 10.2 doesn't work...Can someone give me some advice?

[xiayanqian]

Sorry for disturbing, the maskrcnn-benchmark is ok in the envoriment cuda 10.2, I ran the maskrcnn code on GPU successfuly.
It seems that ZED camera SDK(which include maskrcnn-benchmark) causes the problem, I'll check it...

[DangerousY]

@xiayanqian could you tell me how to run maskrcnn with Cuda 10.2, I encounter some problems too.

[xiayanqian]

@xiayanqian could you tell me how to run maskrcnn with Cuda 10.2, I encounter some problems too.

I think the problem is that hardware platform(GPU) could not afford the computation cost. Nothing to do with the CUDA version.
I ran the maskrcnn-benchmark on Jetson AGX xavier, the program is ok.

[AndreV84]

@xiayanqian could you extend the installation steps for XAvier AGX, please?
I am failing with the output
`idia/.local/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c /home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu -o build/temp.linux-aarch64-3.6/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_72,code=sm_72 -std=c++14
/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu(42): error: identifier "AT_CHECK" is undefined

/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu(68): error: identifier "AT_CHECK" is undefined

2 errors detected in the compilation of "/tmp/tmpxft_00003557_00000000-6_deform_pool_cuda.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
`

[xiayanqian]

@xiayanqian could you extend the installation steps for XAvier AGX, please?
I am failing with the output
`idia/.local/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.6m -c /home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu -o build/temp.linux-aarch64-3.6/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=1 -gencode=arch=compute_72,code=sm_72 -std=c++14
/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu(42): error: identifier "AT_CHECK" is undefined

/home/nvidia/maskrcnn-benchmark/maskrcnn_benchmark/csrc/cuda/deform_pool_cuda.cu(68): error: identifier "AT_CHECK" is undefined

2 errors detected in the compilation of "/tmp/tmpxft_00003557_00000000-6_deform_pool_cuda.cpp1.ii".
error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
`

try replacing AT_CHECK with TORCH_CHECK

[AndreV84]

@xiayanqian it worked
thank you