Pip-tools: Problems resolving Sphinx 1.8.5+

[jmbowman]

pip-compile seems to be unable to find any versions of Sphinx greater than 1.8.5, even though the newest 2.0.1 release installs normally via pip (at least under Python 3.5+) Additionally, it fails to find any dependencies for Sphinx 1.8.5, even though they seem unchanged from the previous 1.8.4 release. Specifying Sphinx==1.8.4 in the input file correctly enumerates and pins all of the Sphinx dependencies. Very little seems to have changed in Sphinx 1.8.5, the only vaguely relevant difference I've found so far is that the downloaded wheel reports having been built with wheel 0.33.1 instead of 0.32.3. Sphinx 2.0.0 was a major release which dropped Python 2.7 support, but pip-compile is failing to find it even when run under Python 3.7 where it installs correctly.

Environment Versions

1. OS Type: macOS and Linux
2. Python version: 3.7.2 and 2.7.12
3. pip version: 9.1.1 and 9.0.3
4. pip-tools version: 3.6.1, master, 3.6.0, 3.5.0

Steps to replicate

1. Create sphinx.in containing just Sphinx
2. Run pip-compile sphinx.in

Expected result

Something like:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile sphinx.in
#
alabaster==0.7.12         # via sphinx
babel==2.6.0              # via sphinx
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
docutils==0.14            # via sphinx
idna==2.8                 # via requests
imagesize==1.1.0          # via sphinx
jinja2==2.10.1            # via sphinx
markupsafe==1.1.1         # via jinja2
packaging==19.0           # via sphinx
pygments==2.3.1           # via sphinx
pyparsing==2.4.0          # via packaging
pytz==2019.1              # via babel
requests==2.21.0          # via sphinx
six==1.12.0               # via packaging, sphinx
snowballstemmer==1.2.1    # via sphinx
sphinx==2.0.1
sphinxcontrib-websupport==1.1.0  # via sphinx
typing==3.6.6             # via sphinx
urllib3==1.24.3           # via requests

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile sphinx.in
#
sphinx==1.8.5

If Sphinx==2.0.1 (or 2.0.0) is specified in the input file, pip-compile exits with an error instead:

No handlers could be found for logger "pip._internal.index"
Traceback (most recent call last):
  File "/Users/jbowman/.virtualenvs/edx-proctoring/bin/pip-compile", line 10, in <module>
    sys.exit(cli())
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/scripts/compile.py", line 352, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/resolver.py", line 119, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/resolver.py", line 225, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/resolver.py", line 331, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/repositories/local.py", line 66, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 254, in get_dependencies
    download_dir, ireq, wheel_cache
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/piptools/repositories/pypi.py", line 217, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/pip/_internal/resolve.py", line 242, in _get_abstract_dist_for
    self.require_hashes
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/pip/_internal/operations/prepare.py", line 282, in prepare_linked_requirement
    req.populate_link(finder, upgrade_allowed, require_hashes)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 198, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/python2.7/site-packages/pip/_internal/index.py", line 792, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for sphinx==2.0.1

[atugushev]

Hello @jmbowman,

Thanks for the report! According to your stacktrace the pip used from py27 environment. See:

  File "/Users/jbowman/.virtualenvs/edx-proctoring/lib/ ---> python2.7 <--- /site-packages/pip/_internal/index.py", line 792, in find_requirement
    'No matching distribution found for %s' % req

That's why pip couldn't find 2.0.1 version. I've checked locally with py37 and it works like a charm:

$ python -V
Python 3.7.3

$ pip --version
pip 19.1.1 from /Projects/pip-tools/.venv/lib/python3.7/site-packages/pip (python 3.7)

$ pip-compile --version
pip-compile, version 3.6.1

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
alabaster==0.7.12         # via sphinx
babel==2.6.0              # via sphinx
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
docutils==0.14            # via sphinx
idna==2.8                 # via requests
imagesize==1.1.0          # via sphinx
jinja2==2.10.1            # via sphinx
markupsafe==1.1.1         # via jinja2
packaging==19.0           # via sphinx
pygments==2.3.1           # via sphinx
pyparsing==2.4.0          # via packaging
pytz==2019.1              # via babel
requests==2.21.0          # via sphinx
six==1.12.0               # via packaging
snowballstemmer==1.2.1    # via sphinx
sphinx==2.0.1
sphinxcontrib-applehelp==1.0.1  # via sphinx
sphinxcontrib-devhelp==1.0.1  # via sphinx
sphinxcontrib-htmlhelp==1.0.2  # via sphinx
sphinxcontrib-jsmath==1.0.1  # via sphinx
sphinxcontrib-qthelp==1.0.2  # via sphinx
sphinxcontrib-serializinghtml==1.1.3  # via sphinx
urllib3==1.24.3           # via requests

Could you recheck your env where you've been running pip-tools?

[jmbowman]

Thanks for the nudge in the right direction, you've helped clear up some of the problems:

• I had somehow ended up with pip-tools 3.5.0 installed in my Mac's system Python 2.7 installation.
• That copy of pip-compile was taking precedence over the 3.6.1 installation in my Python 3.7 virtualenv, even though the virtualenv's version of pip-sync was correctly being used (I have no idea how that was happening; I've since uninstalled the offending 3.5.0 installation).

So you can ignore the part about not being able to find Sphinx>1.8.5. And I've confirmed that under Python 3.7, pip-compile 3.6.1 correctly enumerates the dependencies of Sphinx 1.8.5. But I'm still reproducing the failure to get those dependencies in the output when running pip-compile under Python 2.7 (even though it works for Sphinx 1.8.4). And that's also reproducing on a remote Jenkins installation running Linux workers.

We're actively porting all our code to Python 3, but it's going to take a few months to wrap that up, and I'm a little concerned that this failure to find some dependencies under 2.7 might also be happening for other packages that I haven't noticed yet. The Sphinx case was just blatantly obvious that almost a dozen packages were suddenly being removed from the generated file after a minor point release.

[atugushev]

But I'm still reproducing the failure to get those dependencies in the output when running pip-compile under Python 2.7

I've got this output for py27 and latest pip-tools:

$ cat requirements.in
sphinx

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
alabaster==0.7.12         # via sphinx
babel==2.6.0              # via sphinx
certifi==2019.3.9         # via requests
chardet==3.0.4            # via requests
docutils==0.14            # via sphinx
idna==2.8                 # via requests
imagesize==1.1.0          # via sphinx
jinja2==2.10.1            # via sphinx
markupsafe==1.1.1         # via jinja2
packaging==19.0           # via sphinx
pygments==2.4.0           # via sphinx
pyparsing==2.4.0          # via packaging
pytz==2019.1              # via babel
requests==2.21.0          # via sphinx
six==1.12.0               # via packaging, sphinx
snowballstemmer==1.2.1    # via sphinx
sphinx==1.8.5
sphinxcontrib-websupport==1.1.0  # via sphinx
typing==3.6.6             # via sphinx
urllib3==1.24.3           # via requests

Sorry, I'm a little bit confused what is don't work in your case. Do you have only pinned sphinx==1.8.5 in your requirements.txt without the other dependencies for py27, right? Could it be confirmed in a docker?

[jmbowman]

I've reproduced your results in a Docker container, but continuing to get the problem in a fresh macOS Python 2.7.15 virtualenv and those remote Jenkins environments. So it doesn't seem to be categorically broken under 2.7, but there is definitely some environment characteristic which causes things to break; still trying to isolate what that could be. I've tried both pip 9.0.3 and 9.1.1, I'll do some more experiments tonight. But here are some relevant setup and conditions where the problem is manifesting:

(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ cat sphinx.in
Sphinx<2.0.0
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ pip --version
pip 19.1.1 from /Users/jbowman/.virtualenvs/sphinx-compile/lib/python2.7/site-packages/pip (python 2.7)
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ easy_install --version
setuptools 41.0.1 from /Users/jbowman/.virtualenvs/sphinx-compile/lib/python2.7/site-packages (Python 2.7)
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ pip-compile --version
pip-compile, version 3.6.1
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ which pip-compile
/Users/jbowman/.virtualenvs/sphinx-compile/bin/pip-compile
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.4
BuildVersion:   18E226
(sphinx-compile) C02XN6P9JHD2:Downloads jbowman$ pip-compile -v sphinx.in
Using indexes:
  https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  Sphinx<2.0.0

Finding the best candidates:
  found candidate sphinx==1.8.5 (constraint was <2.0.0)

Finding secondary dependencies:
  sphinx==1.8.5             requires -
------------------------------------------------------------
Result of round 1: stable, done

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile -v sphinx.in
#
sphinx==1.8.5

Any suggestions for diagnosing further before I start spelunking through the pip-compile code?

[vphilippon]

@jmbowman Just in case, I would suggest to run pip-compile --rebuild, both on your local machine and your jenkins runner, which will clear the pip-tools dependency cache (yes, we cache the resolved dependency for faster resolution later).
With your "corrupted" setup from earlier, it might have corrupted the cache.

[jmbowman]

Ah, that fixed it locally for me. I'll try adding that to the Jenkins jobs also.