Smart_open: Cannot install 'smart-open' using Pipenv - incompatible dependencies

[pranasziaukas]

I am using Python 3.7 and Pipenv to manage the dependencies.

I am starting from scratch in a new folder with

pipenv install smart-open --three

This is supposed to create a new virtual environment using Python3, install smart-open and... that's it. However, it cannot lock the dependencies and returns

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches botocore<1.11.0,<1.13.0,<2.0.0,==1.10.84,>=1.10.65,>=1.12.0,>=1.3.0

Further inspection

pipenv graph

returns

smart-open==1.6.0
  - boto [required: >=2.32, installed: 2.49.0]
  - boto3 [required: Any, installed: 1.9.0]
    - botocore [required: >=1.12.0,<1.13.0, installed: 1.12.0]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
      - urllib3 [required: >=1.20,<1.24, installed: 1.23]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
    - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
      - botocore [required: >=1.3.0,<2.0.0, installed: 1.12.0]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
        - urllib3 [required: >=1.20,<1.24, installed: 1.23]
  - bz2file [required: Any, installed: 0.98]
  - requests [required: Any, installed: 2.19.1]
    - certifi [required: >=2017.4.17, installed: 2018.8.24]
    - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
    - idna [required: >=2.5,<2.8, installed: 2.7]
    - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

[pranasziaukas]

Sorry, I'm moving this to the pipenv issue tracker.

[whyboris]

I too have the problem with pipenv installing the package. Is it not an issue with smart-open though? I am trying to install _gensim_ and get this tree:

  - smart-open [required: >=1.2.1, installed: 1.7.0]
    - boto [required: >=2.32, installed: 2.49.0]
    - boto3 [required: <1.8.0, installed: 1.7.84]
      - botocore [required: >=1.10.84,<1.11.0, installed: 1.10.84]
        - docutils [required: >=0.10, installed: 0.14]
        - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
        - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
          - six [required: >=1.5, installed: 1.11.0]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - s3transfer [required: >=0.1.10,<0.2.0, installed: 0.1.13]
        - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.84]
          - docutils [required: >=0.10, installed: 0.14]
          - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
          - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
            - six [required: >=1.5, installed: 1.11.0]
    - botocore [required: <1.11.0, installed: 1.10.84]
      - docutils [required: >=0.10, installed: 0.14]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.3]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.7.3]
        - six [required: >=1.5, installed: 1.11.0]
    - bz2file [required: Any, installed: 0.98]
    - requests [required: Any, installed: 2.19.1]
      - certifi [required: >=2017.4.17, installed: 2018.8.24]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.8, installed: 2.7]
      - urllib3 [required: >=1.21.1,<1.24, installed: 1.23]

Specifically, notice that we have incompatible versions required:

- botocore [required: >=1.10.84,<1.11.0,  installed: 1.10.84]
- botocore [required: >=1.3.0,<2.0.0,     installed: 1.10.84]
- botocore [required: <1.11.0,            installed: 1.10.84]

I'm looking on pipenv issue tracker as well, but why are the required fields different? I imagine that comes from the package, not the _pipenv_ installer 😕

[whyboris]

Related ticket for tracking on pipenv side: https://github.com/pypa/pipenv/issues/2867

[whyboris]

We're in trouble ... @menshikh-iv ?

It looks like pipenv is doing its best, but the versions are simply incompatible -- two dependencies want two different versions of sub-dependencies:

smart_open wants

        'boto3 < 1.8.0',
        'botocore < 1.11.0'

https://github.com/RaRe-Technologies/smart_open/blob/master/setup.py#L54

While boto3 wants

    'botocore>=1.12.6,<1.13.0',

https://github.com/boto/boto3/blob/develop/setup.py#L17

[menshikh-iv]

@whyboris yes, you are right, looks critical

I can drop limitations, but in this case, we can't use tests, hmm
we'll try to fix it ASAP, thanks.

CC: @mpenkov

[menshikh-iv]

Fixed in #232, let me release this bugfix as 1.7.1

[menshikh-iv]

Version 1.7.1 (with unpinned boto3 & botocore) released @whyboris @petergaultney

[petergaultney]

Thank you for this!!

For anyone else watching this issue, I had to run curl -X PURGE https://pypi.org/simple/smart_open to get pip to actually find the new 1.7.1 version on PyPi. YMMV :)

[whyboris]

Confirming the problem is solved. I no longer get the error I got before I opened this issue. 🎉

Thank you for the swift work 🙌

[piskvorky]

Ivan is an unstoppable release tornado this week 🌪 Thanks @menshikh-iv!

[shubhangi-bajpai]

hello everyone. I tried to import gensim in python and am faced with the following error.. please help me. It's urgent as I am working on my internship project.

AttributeError: module 'smart_open' has no attribute 'local_file'

Thanks in advance

[mpenkov]

@shubhangi-bajpai Please make a new ticket and fill in the template, including the version you're using, etc.