Vision: RuntimeError: No such operator torchvision::nms

Created on 25 Feb 2020 · 23Comments · Source: pytorch/vision

🐛 Bug

Ran into this issue when trying to use Faster R-CCC example on a Windows machine.
RuntimeError: No such operator torchvision::nms

To Reproduce

Steps to reproduce the behavior:

conda create -n pytorch
conda activate pytorch
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python

and then

>>> import torch
>>> torch.ops.torchvision.nms
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\david\Miniconda3\envs\pytorch\lib\site-packages\torch\_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

Expected behavior

To be able to use torchvision.nms ...

Environment

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

and

nvidia-smi
Tue Feb 25 15:39:48 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 436.48       Driver Version: 436.48       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+

bug help wanted binaries windows

Source

davidsteinar

👍3

Most helpful comment

It seems that when I do

conda create -n torch
conda activate torch
python -V

I get Python 3.6.8

Then when I do

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I get Python 3.8.1. This environment produces the RuntimeError: No such operator torchvision::nms

But

if I specifically create a python 3.7 environment

conda create -n torch37 python=3.7
conda activate torch37
python -V

I get Python 3.7.6

Then

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I also get Python 3.7.6. In this environment I get no error

import torch
import torchvision
torch.ops.torchvision.nms
>>> <built-in method nms of PyCapsule object at 0x00000201B0AC9FC0>

So the issue seems to be with conda install pytorch torchvision cudatoolkit=10.1 -c pytorch choosing python 3.8 and causing this issue...

davidsteinar on 25 Feb 2020

👍5 🚀1

All 23 comments

Hi,

Thanks for the bug report!

There seems to be an issue with our binaries, at least for Windows.

@peterjc123 do you know if this is a known problem?

fmassa on 25 Feb 2020

I updated my CUDA Toolkit to 10.1 update2 and matching cuDNN to v7.6.5

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

but I still get the same error.

davidsteinar on 25 Feb 2020

Sorry, I could not reproduce this error with torch==1.4.0 and torchvision==0.5.0 with python=3.7 and cudatoolkit=10.1.

Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
>>> import torch
>>> torch.ops.torchvision.nms
<built-in method nms of PyCapsule object at 0x0000029BF9F74150>

My env:

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: (Rev2, Built by MSYS2 project) 7.1.0
CMake version: version 3.16.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 442.19
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.3
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl    defaults
[conda] mkl                       2019.4                      245    defaults
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.14           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.4.0           py3.7_cuda101_cudnn7_0    pytorch
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.5.0                py37_cu101    pytorch

peterjc123 on 25 Feb 2020

It seems that when I do

conda create -n torch
conda activate torch
python -V

I get Python 3.6.8

Then when I do

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I get Python 3.8.1. This environment produces the RuntimeError: No such operator torchvision::nms

But

if I specifically create a python 3.7 environment

conda create -n torch37 python=3.7
conda activate torch37
python -V

I get Python 3.7.6

Then

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I also get Python 3.7.6. In this environment I get no error

import torch
import torchvision
torch.ops.torchvision.nms
>>> <built-in method nms of PyCapsule object at 0x00000201B0AC9FC0>

So the issue seems to be with conda install pytorch torchvision cudatoolkit=10.1 -c pytorch choosing python 3.8 and causing this issue...

davidsteinar on 25 Feb 2020

👍5 🚀1

You mean it's Python 3.8 only issue, right? I'll check later.

peterjc123 on 25 Feb 2020

👍3

Yes it seems so.

davidsteinar on 25 Feb 2020

I have the same issue. I'll try @davidsteinar solution. I'll remove the environment and create a new one for that.

Thanks @davidsteinar and @peterjc123

amrotork on 26 Feb 2020

I tried and it is actually a DLL loading problem for Python 3.8. The workground is to add the directory of cuda before importing torchvision using the following code.

import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.add_dll_directory(py_dll_path)

peterjc123 on 27 Feb 2020

In Python 3.6 the solution will give the error:

AttributeError: module 'os' has no attribute 'add_dll_directory'

So I solved my issue adding this before importing torch and torchvision:

import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.environ['PATH'] += py_dll_path

nicolas-gervais on 29 Mar 2020

🎉1 👍1

@nicolas-gervais This shouldn't be needed for Python 3.6. Could you please tell me something about your env, like how did you install python, pytorch and torchvision?

peterjc123 on 30 Mar 2020

I updated my CUDA Toolkit to 10.1 update2 and matching cuDNN to v7.6.5

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

but I still get the same error.

How to show the Environment?

ckqsars on 14 Apr 2020

python -m torch.utils.collect_env

peterjc123 on 14 Apr 2020

Hello, I also encountered this issue with python3.6, here is my environment:
Collecting environment information...
PyTorch version: 1.4.0+cu100
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla P100-PCIE-16GB

Nvidia driver version: 384.130
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.18.2
[pip3] torch==1.4.0+cu100
[pip3] torchvision==0.5.0
[conda] Could not collect

linghu8812 on 22 Apr 2020

I encountered the same issue in pytorch 1.5
I installed pytorch from source (tag v1.5.0), torchvison form source(master), python version is 3.6

torch.ops.torchvision.nms
op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

plumtus on 23 Apr 2020

I also encountered the same error with PyTorch-1.5.0 and torchvision-0.5.0. Below are the details of my environment.

Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: CentOS Linux release 7.7.1908 (Core)
GCC version: (GCC) 7.3.0
CMake version: version 3.8.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P100-PCIE-12GB
Nvidia driver version: 440.33.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.5.0
[pip3] torchvision==0.5.0
[conda] blas 1.0 mkl
[conda] cuda80 1.0 h205658b_0 pytorch
[conda] mkl 2019.4 243
[conda] mkl-service 2.3.0 py36he904b0f_0
[conda] mkl_fft 1.0.1 py36h3010b51_0
[conda] mkl_random 1.0.1 py36h629b387_0
[conda] pytorch 0.4.1 py36_py35_py27__9.0.176_7.1.2_2 pytorch
[conda] torchvision 0.2.1 py_2 pytorch

yaw91 on 23 Apr 2020

@yaw91 @plumtus you need to use PyTorch 1.5 together with torchvision 0.6.0, otherwise this won't work.

fmassa on 27 Apr 2020

I am also having the same issue on PyTorch nightly build.

Collecting environment information...
PyTorch version: 1.6.0.dev20200424
Is debug build: No
CUDA used to build PyTorch: 9.2

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 418.116.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.6.0.dev20200424
[pip] torchvision==0.7.0a0+c031e28
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               9.2                           0
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] numpy                     1.18.1           py37h4f9e942_0
[conda] numpy-base                1.18.1           py37hde5b4d6_1
[conda] pytorch                   1.6.0.dev20200424 py3.7_cuda9.2.148_cudnn7.6.3_0    pytorch-nightly
[conda] torchvision               0.7.0a0+c031e28           <pip>

mthrok on 27 Apr 2020

Solved. In my case the error occurred because I used different machines with different GPUs for building torchvision and running training.

Specifying CUDA compute capability such as TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" when building torchvision resolved this.

mthrok on 27 Apr 2020

👍1

Could you explain how do we can Specifying CUDA compute capability such as TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" when building torchvision resolved this?

ebrahim31 on 18 Jun 2020

@ebrahim31 just pass

TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" python setup.py install

when compiling torchvision.

fmassa on 7 Jul 2020

I tried and it is actually a DLL loading problem for Python 3.8. The workground is to add the directory of cuda before importing torchvision using the following code.
import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.add_dll_directory(py_dll_path)

I get another error that add_dll_directory is not an implemented function of os module.

yustiks on 30 Oct 2020

@yustiks add_dll_directory is only present in Python 3.8+.

In torchvision we have a guard for this see https://github.com/pytorch/vision/blob/b93d5ee2ddcf2a9876b5871cbb958016e263336b/torchvision/extension.py#L27-L34

Please open a new issue if you hit an issue with torchvision

fmassa on 30 Oct 2020

I also encounter the same issue. In my case, nvidia-smi: CUDA 11.0 and I installed pytorch using torchtoolkit 10.1.
So in order to solve, use this command:
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
Hope this helps