Text: pytorch minimum version isn't specified

[avacariu]

🐛 Bug

Describe the bug
Pytorch minimum version isn't specified in the setup.py, but it should be because (e.g) torchtext 0.7 cannot be used with pytorch 1.4 nor 1.5 (it crashes on import).

To Reproduce

(venv) $ pip install torch==1.4 torchtext==0.7
...
(venv) $ python -c 'import torchtext'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "venv/lib/python3.7/site-packages/torchtext/__init__.py", line 42, in <module>
    _init_extension()
  File "venv/lib/python3.7/site-packages/torchtext/__init__.py", line 38, in _init_extension
    torch.ops.load_library(ext_specs.origin)
  File "venv/lib/python3.7/site-packages/torch/_ops.py", line 106, in load_library
    ctypes.CDLL(path)
  File "/home/andrei/.pyenv/versions/3.7.5/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libtorch_cpu.so: cannot open shared object file: No such file or directory

and with torch 1.5:

(venv) $ pip install torch==1.5 torchtext==0.7
...
(venv) $ python3 -c 'import torchtext'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "venv/lib/python3.7/site-packages/torchtext/__init__.py", line 42, in <module>
    _init_extension()
  File "venv/lib/python3.7/site-packages/torchtext/__init__.py", line 38, in _init_extension
    torch.ops.load_library(ext_specs.origin)
  File "venv/lib/python3.7/site-packages/torch/_ops.py", line 105, in load_library
    ctypes.CDLL(path)
  File "/home/andrei/.pyenv/versions/3.7.5/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: venv/lib/python3.7/site-packages/torchtext/_torchtext.so: undefined symbol: _ZN3c1023_fastEqualsForContainerERKNS_6IValueES2_

It does work fine with torch 1.6:

(venv) $ pip install torch==1.6 torchtext==0.7
...
(venv) $ python3 -c 'import torchtext'

Expected behavior
Either the torchtext package should specify a minimum version of pytorch so that it doesn't get accidentally installed when I'm using an older torch version, or it should work with older versions of pytorch.

Screenshots
n/a

Environment

• PyTorch Version (e.g., 1.0): 1.4.0 and 1.5.0
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source): n/a
• Python version: 3.7
• CUDA/cuDNN version: n/a
• GPU models and configuration: n/a
• Any other relevant information:

Additional context
Add any other context about the problem here.

[zhangguanheng66]

Thanks for bringing up the issue. In v0.7.0 release, we add cpp extension to torchtext library and the binary package via pypi were compiled with the latest version of torch (a.k.a. v1.6).

[wconstab]

I'm seeing a related issue, with pytorch nightly + torchtext==0.7.0 where torchtext crashes on import. It's been repro'd both in our torch/hub CI and locally.

Hub CI log: https://app.circleci.com/pipelines/github/pytorch/hub/462/workflows/88e8875f-1745-4b11-99d4-8752a39347b9/jobs/651

Local Repro:
This works:

conda install pytorch torchvision cudatoolkit=9.2 -c pytorch
pip install torchvision==0.7.0
python -c 'import torchtext'

This crashes (segfault):

conda install pytorch torchvision cudatoolkit=9.2 -c pytorch-nightly
python -c 'import torchtext' 

The nightly version I tested locally is: pytorch-nightly/linux-64::pytorch-1.7.0.dev20200727-py3.7_cuda10.2.89_cudnn7.6.5_0

note: I did the steps in the exact order above, and I used the particular conda command bc i copied it from pytorch.org, but I don't think the order is important and I don't think torchvision is related even though it's included in the command.

[zhangguanheng66]

I'm seeing a related issue, with pytorch nightly + torchtext==0.7.0 where torchtext crashes on import. It's been repro'd both in our torch/hub CI and locally.

Hub CI log: https://app.circleci.com/pipelines/github/pytorch/hub/462/workflows/88e8875f-1745-4b11-99d4-8752a39347b9/jobs/651

Local Repro:
This works:

conda install pytorch torchvision cudatoolkit=9.2 -c pytorch
pip install torchvision==0.7.0
python -c 'import torchtext'

This crashes (segfault):

conda install pytorch torchvision cudatoolkit=9.2 -c pytorch-nightly
python -c 'import torchtext' 

The nightly version I tested locally is: pytorch-nightly/linux-64::pytorch-1.7.0.dev20200727-py3.7_cuda10.2.89_cudnn7.6.5_0

note: I did the steps in the exact order above, and I used the particular conda command bc i copied it from pytorch.org, but I don't think the order is important and I don't think torchvision is related even though it's included in the command.

Since we added the cpp extension and compiled the binary release package separately for pip/conda/nightlies, I would suggest not to mix them. Please install the packages from same channels because our release tests cover them accordingly. cc @seemethere

[Traeyee]

I use torch==1.3.0 torchtext==0.7.0 and encountered the same issue. I downgraded torchtext to 0.6.0 and the problem disappeared. I don't know whether it will work with torch>=1.4 and it really seems to be a compatibility problem.

[wconstab]

FWIW I believe it is 'unsupported' to mix and match conda and pip packages from within the torch ecosytstem. To be clear, it wouldn't be an issue if you pip installed something else like 'pytest' while using conda for torch- but because of how the conda and pip packages of torch components are built, the binary (lib) parts of torch, torchtext, torchvision need to be all compiled and packaged the same way in order to interoperate correctly. If you're using conda to install torch and torchtext and still having issues, let us know.

[avacariu]

I'll open a PR for this, because I notice there's a "version compatibility" table in the README, but that should really be encoded in the setup.py.

@wconstab This issue isn't really about pip vs conda, it's more that new versions of torchtext are compiled against specific torch versions and you have to use compatible versions.

[zhangguanheng66]

Since cpp extension was added to torchtext, the 0.7.0 release was compiled against torch 1.6.0 release. Unlike the earlier version of torchtext with python code only, users have to use torchtext 0.7.0 release strictly with torch 1.6.0.

@avacariu yeap. A PR to update the README file is a good idea.

[avacariu]

@zhangguanheng66 The README isn't the problem; it already says what you said. The problem is the setup.py that doesn't specify what you just said. If my dependencies are defined as install_requires = ['torch==1.4.0', 'torchtext'] in my project, then pip will pull in the latest torchtext which is incompatible because of this line:

https://github.com/pytorch/text/blob/e9b711b67d83ca5ad575508f730e897c4ace9926/setup.py#L85

That torch dependency should say torch==1.6.0 if that's the version torchtext is compatible with.

[zhangguanheng66]

@zhangguanheng66 The README isn't the problem; it already says what you said. The problem is the setup.py that doesn't specify what you just said. If my dependencies are defined as install_requires = ['torch==1.4.0', 'torchtext'] in my project, then pip will pull in the latest torchtext which is incompatible because of this line:

https://github.com/pytorch/text/blob/e9b711b67d83ca5ad575508f730e897c4ace9926/setup.py#L85

That torch dependency should say torch==1.6.0 if that's the version torchtext is compatible with.

But if you set install_requires = ['torch', 'torchtext'], pip should be able to pull the right wheels which are compatible with each other, right?

[avacariu]

Yes, but I can't because my code specifically depends on torch 1.4.0 since the latest torch has backward-incompatible changes.

[avacariu]

And actually, you're not guaranteed to get compatible wheels anyways because sometimes pip will just grab a dependency from cache rather than installing the latest version.

And also suppose torch releases a new version, but there's no new torchtext wheel that is built against it yet. Then you'll also end up with a segfault.

[zhangguanheng66]

And actually, you're not guaranteed to get compatible wheels anyways because sometimes pip will just grab a dependency from cache rather than installing the latest version.

And also suppose torch releases a new version, but there's no new torchtext wheel that is built against it yet. Then you'll also end up with a segfault.

Since we add the cpp extension, we are committed to add the compatible torchtext package whenever there is a new release for torch. IMO, we should update the installation table in the README file here

[avacariu]

How come you're unwilling to encode that information in setup.py?

[da03]

How come you're unwilling to encode that information in setup.py?

I agree. Dependency information should be encoded setup.py, which is the case for most well-managed python packages. Not encoding the dependency information in setup.py would cause lots of compatibility issues for legacy code relying on older versions of torchtext, and by "older version" I'm talking about torchtext 0.7.0, which was released only a few months ago and lots of users are still using that due to the overhaul of code in later versions (such as the retiring of Field and Batch).

[dsokeng]

Seeing the same issue with the versions that look compatible according to the table in README.md:

# python = 3.7.7
torch==1.7.1
torchtext==0.8.1

Error:
```
OSError: /home/.../.pyenv/versions/3.7.7/envs/.../lib/python3.7/site-packages/torchtext/_torchtext.so: undefined symbol: _ZN3c1023_fastEqualsForContainerERKNS_6IValueES2_