Maskrcnn-benchmark: maskrcnn benchmark error

Created on 9 Mar 2019 · 13Comments · Source: facebookresearch/maskrcnn-benchmark

win10+anaconda3+python3.7.2+cuda9.0

ROIAlign_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (__imp_??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@XZ)
ROIPool_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (__imp_??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@XZ)
SigmoidFocalLoss_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) public: __cdecl c10::cuda::CUDAStream::operator struct CUstream_st *(void)const " (__imp_??BCUDAStream@cuda@c10@@QEBAPEAUCUstream_st@@XZ)
ROIAlign_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (__imp_?getCurrentCUDAStream@cuda@c10@@YA?AVCUDAStream@12@F@Z)
ROIPool_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (__imp_?getCurrentCUDAStream@cuda@c10@@YA?AVCUDAStream@12@F@Z)
SigmoidFocalLoss_cuda.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) class c10::cuda::CUDAStream __cdecl c10::cuda::getCurrentCUDAStream(short)" (__imp_?getCurrentCUDAStream@cuda@c10@@YA?AVCUDAStream@12@F@Z)
buildlib.win-amd64-3.7maskrcnn_benchmark_C.cp37-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe' failed with exit status 1120

have someone ever met? I would appreciate it if you could give me some advice.

windows

Source

paotato

Most helpful comment

Full guide for install

Windows 10 laptop + GTX 1050 (2019 March 23)

CUDA

Install CUDA 10.1, Custom install, Development and Runtime, nothing else, it's likely your graphics driver will be updated even when you unclick it (congrats). When you select the components and the installer hangs and all checkboxes disappear, simply click back & next again.
Open up device manager and if the driver fails, install your video card's latest driver from NVIDIA website. (I installed that also from Cuda setup first and it messed up the driver).
Download cudnn-10.1-windows10-x64-v7.5.0.56 (requires registration). Extract to a temp folder. As putting it into a folder and adding to %PATH% won't work anymore for some reason, do the following:

Copy \cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin.
Copy \cuda\include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include.
Copy \cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64.

Visual Studio and friends

Install 2017 Community, C++ dev at least. Make certain you click 'VC++ 2017 version 15.4 v14.11 toolset' below the 'Individual components' tab.
Go to https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017 and download "Build Tools For Visual Studio" under "Tools for "Visual Studio", run it. Select: 'Workloads - Visual C++ build tools' and select from individual components also 'Windows 10 SDK' and 'VC++ 2015.3 v14.00 (v140) toolset for desktop'.
Add this to your PATH environment variables: C:Program Files (x86)Windows Kits10binx64
Copy rc.exe & rcdll.dll from C:Program Files (x86)Windows Kits8.1binx86 to C:Program Files (x86)Microsoft Visual Studio 14.0VCbin

Restart your computer at this point in any case.

Anaconda
Install Anaconda if you didn't do it yet. We will work with a new environment called 'seg' with python 3.6 (just that we can also install Tensorflow, but maybe it's not needed any more for TF and 3.7 is fine, I don't know, go ahead with default/3.7 if you don't care). We'll also work with pytorch-nightly as even 1.0.1 doesn't contain what we need. Execute the following commands:

conda create -n seg python=3.6
activate seg
conda install ipython
pip install ninja yacs cython matplotlib tqdm requests
conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0
conda install -c anaconda git
conda install nb_conda
pip install opencv-python opencv-contrib-python
pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"

PyTorch solution

Download/unzip or git this solution, open up Anaconda prompt and navigate into the working folder.
Navigate to maskrcnn_benchmarkcsrccuda and modify ROIAlign_cuda.cu, ROIPool_cuda.cu and SigmoidFocalLoss_cuda.cu in the following manner: replace the call for THCCeilDiv to ceil_div1,ceil_div2,ceil_div3 respectively (replace all two calls in each of these files) and add the following function (in all of these files, so for the first file it is like below, just the name changes because of proper linking later):

long ceil_div1(long a, long b) { 
    return (a + b - 1) / b; 
}

Navigate back to the root folder of the solution, open up setup.py and modify the following two parts like this:

    extra_compile_args = {"cxx": ['/MD']}
    extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']

    ext_modules = [
        extension(
            "maskrcnn_benchmark._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
            extra_link_args = extra_link_args,
        )
    ]

Go back to Anaconda command prompt and execute:

set "VS150COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
set DISTUTILS_USE_SDK=1
call "%VS150COMNTOOLS%\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build develop

Time for a beer. It works on 2019.03.23, if you are from the future, you will have another battle, but hopefully this will make it easier.
Kudos to all authors who gave hints, was too busy stealing their ideas to record their names. One beer for septentriones above for certain!
If something is not 100%, PM me, so I can update this. Cheers!

Notes:
In case you run into an empty page when launching Jupyter, don't panic. Execute:

pip uninstall tornado
pip install tornado==5.1.1

pip uninstall notebook
pip install notebook==5.7.4

Re-launch and press CTRL+Shift+R in browser. Clear cache, and repeat relaunching with CTRL+Shift+R if still empty.

bpfliegel on 23 Mar 2019

👍10

All 13 comments

Sorry, I have no experience with Windows.

fmassa on 12 Mar 2019

I had the same issue with the only difference being that I'm using Cuda 10.0

After some digging I discovered that c10_cuda.lib is missing from the linking call and the symbols needed are present in c10_cuda.dll. I was able to add c10_cuda.lib as an extra linker argument in setup.py and build successfully.

septentriones on 18 Mar 2019

I had the same issue with the only difference being that I'm using Cuda 10.0

After some digging I discovered that c10_cuda.lib is missing from the linking call and the symbols needed are present in c10_cuda.dll. I was able to add c10_cuda.lib as an extra linker argument in setup.py and build successfully.

@septentriones Would you let us know exactly how did you do it? I tried for an hour, but had no luck so far. Any dirty hack is welcome :) Thanks.

bpfliegel on 23 Mar 2019

Sure. I was also working through some other compile issues (among other changes I had to cast to long in SigmoidFocalLuss_cuda.cu as in this previous commit) and tried a number of fixes. Currently my setup.py includes the changes suggested here and additionally has 'c10_cuda.lib' as an argument for extra_link_args. In summary I added:

extra_compile_args = {"cxx": ['/MD']}
extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']

and modified the ext_modules as:

ext_modules = [
    extension(
        "maskrcnn_benchmark._C",
        sources,
        include_dirs=include_dirs,
        define_macros=define_macros,
        extra_compile_args=extra_compile_args,
        extra_link_args = extra_link_args,
    )
]

I hope that works for you too!

Note: What led me to figuring this out was looking for the missing symbols in c10.lib and eventually c10_cuda.lib using dumpbin and undname from the VS console.

septentriones on 23 Mar 2019

Thanks really a lot for the extensive response. It doesn't work for me yet, but I think I messed up something, because I could get the linker working earlier with modifying the command manually - and that doesn't work now either :) Will be back with the results later :)

bpfliegel on 23 Mar 2019

Reinstalled conda environment with pytorch nightly and philferriere's coco and finally it compiles, thanks!!! Messed up my files somehow.

One thing to mention is that besides those links you mentioned (they guided me also), is that at the moment all three *.cu files needs to be changed, with long declarations:

long ceil_div(long a, long b) {
return (a + b - 1) / b;
}

I'll try to post a more extensive guide.

bpfliegel on 23 Mar 2019

I'm glad you finally got it working! As you mention, I did also use pytorch nightly and philferrier's coco. I only needed to change the one *.cu file I mentioned earlier though because the other two *.cu files already had the change (but is important to make sure that all three are modified similarly). Cheers!

septentriones on 23 Mar 2019

Hmm, I don't see that there are changed in the master repo. I need to change all three of them. Thanks a lot again!

bpfliegel on 23 Mar 2019

Full guide for install

Windows 10 laptop + GTX 1050 (2019 March 23)

CUDA

Install CUDA 10.1, Custom install, Development and Runtime, nothing else, it's likely your graphics driver will be updated even when you unclick it (congrats). When you select the components and the installer hangs and all checkboxes disappear, simply click back & next again.
Open up device manager and if the driver fails, install your video card's latest driver from NVIDIA website. (I installed that also from Cuda setup first and it messed up the driver).
Download cudnn-10.1-windows10-x64-v7.5.0.56 (requires registration). Extract to a temp folder. As putting it into a folder and adding to %PATH% won't work anymore for some reason, do the following:

Copy \cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin.
Copy \cuda\include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include.
Copy \cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64.

Visual Studio and friends

Install 2017 Community, C++ dev at least. Make certain you click 'VC++ 2017 version 15.4 v14.11 toolset' below the 'Individual components' tab.
Go to https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017 and download "Build Tools For Visual Studio" under "Tools for "Visual Studio", run it. Select: 'Workloads - Visual C++ build tools' and select from individual components also 'Windows 10 SDK' and 'VC++ 2015.3 v14.00 (v140) toolset for desktop'.
Add this to your PATH environment variables: C:Program Files (x86)Windows Kits10binx64
Copy rc.exe & rcdll.dll from C:Program Files (x86)Windows Kits8.1binx86 to C:Program Files (x86)Microsoft Visual Studio 14.0VCbin

Restart your computer at this point in any case.

conda create -n seg python=3.6
activate seg
conda install ipython
pip install ninja yacs cython matplotlib tqdm requests
conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0
conda install -c anaconda git
conda install nb_conda
pip install opencv-python opencv-contrib-python
pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"

PyTorch solution

Download/unzip or git this solution, open up Anaconda prompt and navigate into the working folder.
Navigate to maskrcnn_benchmarkcsrccuda and modify ROIAlign_cuda.cu, ROIPool_cuda.cu and SigmoidFocalLoss_cuda.cu in the following manner: replace the call for THCCeilDiv to ceil_div1,ceil_div2,ceil_div3 respectively (replace all two calls in each of these files) and add the following function (in all of these files, so for the first file it is like below, just the name changes because of proper linking later):

long ceil_div1(long a, long b) { 
    return (a + b - 1) / b; 
}

Navigate back to the root folder of the solution, open up setup.py and modify the following two parts like this:

    extra_compile_args = {"cxx": ['/MD']}
    extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']

    ext_modules = [
        extension(
            "maskrcnn_benchmark._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
            extra_link_args = extra_link_args,
        )
    ]

Go back to Anaconda command prompt and execute:

set "VS150COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
set DISTUTILS_USE_SDK=1
call "%VS150COMNTOOLS%\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build develop

Notes:
In case you run into an empty page when launching Jupyter, don't panic. Execute:

pip uninstall tornado
pip install tornado==5.1.1

pip uninstall notebook
pip install notebook==5.7.4

Re-launch and press CTRL+Shift+R in browser. Clear cache, and repeat relaunching with CTRL+Shift+R if still empty.

bpfliegel on 23 Mar 2019

👍10

@fmassa ^^^ Francisco, this might be interesting to you aswell. Thanks a lot to you guys for the great job!

bpfliegel on 23 Mar 2019

Updated the guide, CuDNN was not working as per the initial description.

bpfliegel on 26 Mar 2019

@bpfliegel Hi, I follow your steps and got the error as below:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\nvcc.exe -c d:\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc\cuda\deform_conv_cuda.cu -o build\temp.win-amd64-3.6\Release\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc\cuda\deform_conv_cuda.obj -Xcompiler /wd4819 -Xcompiler /MD -Id:\workspace\maskrcnn-benchmark\maskrcnn_benchmark\csrc -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\TH -IC:\Users\User\Anaconda3\envs\maskrcnn\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include" -IC:\Users\User\Anaconda3\envs\maskrcnn\include -IC:\Users\User\Anaconda3\envs\maskrcnn\include "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.14.26428\ATLMFC\include" "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.14.26428\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" "-Ic:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0
deform_conv_cuda.cu
c:\program files\nvidia gpu computing toolkit\cuda\v9.0\include\crt/host_config.h(133): fatal error C1189: #error:  -- unsupported Microsoft Visual Studio version! Only the versions 2012, 2013, 2015 and 2017 are supported!
error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v9.0\\bin\\nvcc.exe' failed with exit status 2

How to fix it?
Thanks a lot!

peterhsu2018 on 15 May 2019

In case anyone else is trying this in 2021: https://github.com/facebookresearch/maskrcnn-benchmark/issues/547#issuecomment-475903282

Full guide for install

Windows 10 laptop + GTX 1050 (2019 March 23)

CUDA

Install CUDA 10.1, Custom install, Development and Runtime, nothing else, it's likely your graphics driver will be updated even when you unclick it (congrats). When you select the components and the installer hangs and all checkboxes disappear, simply click back & next again.

Open up device manager and if the driver fails, install your video card's latest driver from NVIDIA website. (I installed that also from Cuda setup first and it messed up the driver).

Download cudnn-10.1-windows10-x64-v7.5.0.56 (requires registration). Extract to a temp folder. As putting it into a folder and adding to %PATH% won't work anymore for some reason, do the following:
Copy \cuda\bin\cudnn64_7.dll to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin.
Copy \cuda\include\cudnn.h to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include.
Copy \cuda\lib\x64\cudnn.lib to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64.
Visual Studio and friends

Install 2017 Community, C++ dev at least. Make certain you click 'VC++ 2017 version 15.4 v14.11 toolset' below the 'Individual components' tab.

Go to https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2017 and download "Build Tools For Visual Studio" under "Tools for "Visual Studio", run it. Select: 'Workloads - Visual C++ build tools' and select from individual components also 'Windows 10 SDK' and 'VC++ 2015.3 v14.00 (v140) toolset for desktop'.

Add this to your PATH environment variables: C:Program Files (x86)Windows Kits10binx64

Copy rc.exe & rcdll.dll from C:Program Files (x86)Windows Kits8.1binx86 to C:Program Files (x86)Microsoft Visual Studio 14.0VCbin

Restart your computer at this point in any case.

Anaconda
Install Anaconda if you didn't do it yet. We will work with a new environment called 'seg' with python 3.6 (just that we can also install Tensorflow, but maybe it's not needed any more for TF and 3.7 is fine, I don't know, go ahead with default/3.7 if you don't care). We'll also work with pytorch-nightly as even 1.0.1 doesn't contain what we need. Execute the following commands:
conda create -n seg python=3.6
activate seg
conda install ipython
pip install ninja yacs cython matplotlib tqdm requests
conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0
conda install -c anaconda git
conda install nb_conda
pip install opencv-python opencv-contrib-python
pip install "git+https://github.com/philferriere/cocoapi.git#egg=pycocotools&subdirectory=PythonAPI"
PyTorch solution

Download/unzip or git this solution, open up Anaconda prompt and navigate into the working folder.

Navigate to maskrcnn_benchmarkcsrccuda and modify ROIAlign_cuda.cu, ROIPool_cuda.cu and SigmoidFocalLoss_cuda.cu in the following manner: replace the call for THCCeilDiv to ceil_div1,ceil_div2,ceil_div3 respectively (replace all two calls in each of these files) and add the following function (in all of these files, so for the first file it is like below, just the name changes because of proper linking later):
long ceil_div1(long a, long b) { 
  return (a + b - 1) / b; 
}
Navigate back to the root folder of the solution, open up setup.py and modify the following two parts like this:
    extra_compile_args = {"cxx": ['/MD']}
    extra_link_args = ['c10_cuda.lib','/NODEFAULTLIB:LIBCMT.LIB']
    ext_modules = [
        extension(
            "maskrcnn_benchmark._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
            extra_link_args = extra_link_args,
        )
    ]
Go back to Anaconda command prompt and execute:
set "VS150COMNTOOLS=c:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build"
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
set DISTUTILS_USE_SDK=1
call "%VS150COMNTOOLS%\vcvarsall.bat" x64 -vcvars_ver=14.11
python setup.py build develop
Time for a beer. It works on 2019.03.23, if you are from the future, you will have another battle, but hopefully this will make it easier.
Kudos to all authors who gave hints, was too busy stealing their ideas to record their names. One beer for septentriones above for certain!
If something is not 100%, PM me, so I can update this. Cheers!

Notes:
In case you run into an empty page when launching Jupyter, don't panic. Execute:
pip uninstall tornado
pip install tornado==5.1.1

pip uninstall notebook
pip install notebook==5.7.4
Re-launch and press CTRL+Shift+R in browser. Clear cache, and repeat relaunching with CTRL+Shift+R if still empty.

In case anyone else is trying this in 2021:

Change this line

conda install -c pytorch pytorch-nightly torchvision cudatoolkit=10.0

With this:

conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch

Install apex:

git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install

Make sure you have the following versions:

Pytorch: 1.4.0
torchvision: 0.5.0
apex: 0.1.0

This worked for me on 2021.03.16 on my Windows 10.

nathalia-kim on 16 Mar 2021

Was this page helpful?

0 / 5 - 0 ratings

Related issues

?? What's the problem

kaaier · 3Comments

Why the large batchsize cause training slow?

auroua · 3Comments

Error when trying to train: RuntimeError: cuda runtime error (59) : device-side assert triggered

Nacho114 · 4Comments

How to Improve?

jbitton · 4Comments

Run coco panoptic dataset

YuShen1116 · 4Comments