Pip: Respect python-requires in offline mode (--find-link --no-index)

[gaborbernat]

One way (and before the rewrite release the only way) to bootstrap a virtual environment in an offline environment is to point pip to some bundled wheels via the --find-links parameter. The problem is that if this folder contains both wheels that support the target environment and wheels that don't (e.g. with setuptools 45 dropping python 2), pip does not pick the last supported version of a package, but instead picks the highest version, and then fails at install time.

Triggered by https://github.com/pypa/virtualenv/issues/1496.

[chrahunt]

As a workaround, you can set --index-url to a local directory that looks like a PEP 503 repo e.g. contains directories with names matching expected package names, with each containing a minimal index.html that can have anchor elements pointing to the available packages.

The current pip resolver doesn't handle any situations like this gracefully. As @pfmoore mentioned in the linked issue, this is related to the new resolver work - we need to determine what situations are cause for failure vs backtracking during dependency resolution.

[sbidoul]

[side note, specifically for the setuptools issue]

I did a quick test with a setuptools 45 wheel renamed to setuptools-45.0.0-py3-none-any.whl (py3 tag instead of py2.py3) and it seems to work fine. IMHO the setuptools 45 wheel has a wrong tag and should be pulled from PyPI and replaced with one with the py3 tag.

[pfmoore]

@sbidoul Good idea, As long as there's no local sdist, and the PyPI files have the correct data-requires-python tag (which it does) then this should be a reasonable fix.

[sbidoul]

I opened setuptools issue https://github.com/pypa/setuptools/issues/1974.

[pganssle]

I don't think that fixes the wider problem. You could easily have a similar situation with a wheel that doesn't support Python 3.4 or 3.8 or something where the tags aren't sufficient.

[pfmoore]

I don't think that fixes the wider problem

It doesn't. And it doesn't make any difference when a sdist is available, as the fact that the tag doesn't cover py2 just means that the sdist will be used (at which point Python-Requires kicks in). But none of that is IMO any reason not to tag the wheel correctly.

IMO, Python-Requires should be handled by the resolver, as this issue suggests, so that only versions that support the Python version being used are considered. But doing so has a cost, as checking Python-Requires can need anything up to a full build. So having as many other indications that a file isn't relevant for the current interpreter is still useful.

[brainwane]

We need to check whether this is reproducible in git master - we think it might be already fixed.

[brainwane]

@pfmoore Now that your contract is finishing up, should we un-assign this from you?

[pfmoore]

I've unassigned myself. This may already be fixed in the new resolver (we do respect Python-Requires, so it may be that all that is needed is for someone to confirm that.

@gaborbernat can you confirm if this is still an issue with the new resolver (--unstable-feature=resolver)?

[gaborbernat]

I cannot seem to reproduce it now, but the condition to close this I'd expect would be a test to pips test suite to validate this use case is working as expected.

[pradyunsg]

https://github.com/pypa/pip/blob/e1d489fce65fc7d908b5807b2e58c0c073237fec/tests/functional/test_new_resolver.py#L336-L385

There is a corresponding test for this now.