Xla: pytorch-nightly torchvision error

[brianhhu]

I just updated my pytorch-nightly conda environment, and tried to run a script to train a model that was working previously. The error I received is pasted below:

Traceback (most recent call last): File "test_train_cifar.py", line 20, in <module> import torchvision File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/__init__.py", line 1, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/__init__.py", line 11, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/__init__.py", line 1, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/models/detection/faster_rcnn.py", line 7, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/ops/__init__.py", line 1, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/ops/boxes.py", line 2, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/_C.py", line 7, in <module> File "/anaconda3/envs/pytorch-nightly/lib/python3.6/site-packages/torchvision-0.3.0-py3.6-linux-x86_64.egg/torchvision/_C.py", line 6, in __bootstrap__ File "/anaconda3/envs/pytorch-nightly/lib/python3.6/imp.py", line 343, in load_dynamic return _load(spec) ImportError: /home/jupyter/.cache/Python-Eggs/torchvision-0.3.0-py3.6-linux-x86_64.egg-tmp/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: THPVariable_Wrap

[ailzhang]

cc: @jysohn23 when does our nightly get built everyday? AFAIK there was a breakage from pytorch repo last night(it's fixed now), so if it was built in that time it might be broken?
Maybe we should not push if the nightly didn't pass test.

[jysohn23]

Nightly gets built around 1am every day if that's what you're asking. Yeah whenever there's a build / CPU test breakage we fail the nightly release workflow so nothing gets published to nightly, so tonight nothing was published. CPU tests are the first line of defense that prevents publishing if entirely broken.

@brianhhu What do you mean by?

I just updated my pytorch-nightly conda environment

We currently do this to install torchvision as part of our builds.

[brianhhu]

I used the recommended steps in the README to update the wheels in the pytorch-nightly conda env:

(vm)$ cd /usr/share/torch-xla-nightly/pytorch/xla
(vm)$ ./scripts/update_nightly_torch_wheels.sh

I haven't tried downgrading torchvision to 0.2.0, that may fix the problem.

[jysohn23]

Can you try rebuilding torchvision following this within the conda environment after updating the nightly torch wheels?

[brianhhu]

Ok, that seems to work! Do I have to repeat this step each time I update the nightly torch wheels? Thanks~

[jysohn23]

I'll update the update_nightly_torch_wheels.sh script to do this, but in the mean time please do! Thanks :)

[tweetyone]

I have the same problem. I run the build_torch_wheels.sh, but it doesn't work, the error still exist. And during the process of running build_torch_wheels.sh, also appear some errors.

[jysohn23]

I have the same problem. I run the build_torch_wheels.sh, but it doesn't work, the error still exist. And during the process of running build_torch_wheels.sh, also appear some errors.

Can you give more detail? Where are you running this? Are you trying to build everything from scratch? Are you seeing torchvision error too?

[tweetyone]

I use prebuilt PyTorch/XLA Image on pytorch-nightly environment. I updated pytorch-nightly and run test_mnist.py, then saw the torchvision error: undefined symbol: THPVariable_Wrap.
After saw this issue, I run ./scripts/build_torch_wheels.sh, it has some problems like the picture shows, also, the THPVariable_Wrap error still exist.

[jysohn23]

Sorry, I lost track of this. Was this resolved? Thanks.

[tweetyone]

Sorry, I lost track of this. Was this resolved? Thanks.

no, so I am using pytorch-0.1 now. :)

[jysohn23]

no, so I am using pytorch-0.1 now. :)

You should be able to use pytorch-nightly just fine with torchvision. Did you try rebuilding torchvision after updating the wheels?

vision$ python setup.py install from the vision directory (which corresponds to torchvision?

[tweetyone]

can't rebuild, permission denied

[jysohn23]

Oh that's because it's under /usr/share. Try either prepending the command with sudo or cloning the repo somewhere else (home?) and building.

[tweetyone]

Oh that's because it's under /usr/share. Try either prepending the command with sudo or cloning the repo somewhere else (home?) and building.

I cloned the repo and rebuild the torchvision. It works! thanks!