Pip: `--unstable-feature=resolver` fails to recognize dependencies between explicit dist URLs

[webknjaz]

I have a CI that pulls in ansible and ansible-base from their
respective unreleased "devel" locations. ansible depends on
ansible-base. Both have target versions 2.10.0 I was using the
following form for pip's arguments (originally):
ansible @ https://github.com/ansible/ansible.git@devel and
ansible-base @ https://toshio.fedorapeople.org/ansible/acd/ansible/ansible-2.10.0.tar.gz.

ansible has a dependency ansible-base<2.11.0,>=2.10.0.

At some point, this started failing so I started experimenting on
how to fix it. Here's the forms I've tried:

ansible @ https://github.com/ansible/ansible.git@devel#egg=ansible==2.10
ansible @ https://github.com/ansible/ansible/archive/devel.zip#egg=ansible==2.10
https://github.com/ansible/ansible/archive/devel.zip#egg=ansible==2.10
ansible-base @ https://toshio.fedorapeople.org/ansible/acd/ansible/ansible-2.10.0.tar.gz#egg=ansible-base==2.10
https://toshio.fedorapeople.org/ansible/acd/ansible/ansible-2.10.0.tar.gz#egg=ansible-base==2.10

And I've also enabled --unstable-feature=resolver which happens to
have the same issue. And the issue is that pip install says that it
cannot fulfill these two deps (even though the log says that it has
downloaded both tarballs and recognized their deps).

Failure example: https://github.com/ansible/ansible-lint/runs/762485001#step:8:581

Here's a verbose log output: https://gist.github.com/webknjaz/4516cf3f6b6984f91501c1789826ff60

The most interesting part extracted:

ERROR: Could not find a version that satisfies the requirement ExplicitRequirement(LinkCandidate('https://github.com/ansible/ansible/archive/devel.zip#egg=ansible-base==2.10.0'))
ERROR: Could not find a version that satisfies the requirement ansible-base<2.11,>=2.10.0 (from ansible)
ERROR: No matching distribution found for ansible-base, ansible-base
Exception information:
Traceback (most recent call last):
  File "~/src/github/ansible/ansible-lint/.tox/ansibledevel-py38/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 80, in resolve
    self._result = resolver.resolve(requirements)
  File "~/src/github/ansible/ansible-lint/.tox/ansibledevel-py38/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 413, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "~/src/github/ansible/ansible-lint/.tox/ansibledevel-py38/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 319, in resolve
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=ExplicitRequirement(LinkCandidate('https://github.com/ansible/ansible/archive/devel.zip#egg=ansible-base==2.10.0')), parent=None), RequirementInformation(requirement=SpecifierRequirement('ansible-base<2.11,>=2.10.0'), parent=LinkCandidate('https://toshio.fedorapeople.org/ansible/acd/ansible/ansible-2.10.0.tar.gz#egg=ansible==2.10.0'))]

[pradyunsg]

The "easiest" case to reproduce this seems to be:

pip install "https://github.com/ansible/ansible/archive/devel.zip" "https://toshio.fedorapeople.org/ansible/acd/ansible/ansible-2.10.0.tar.gz" --unstable-feature=resolver

This seems to be an issue due to the interaction of ExplicitRequirement and SpecifierRequirement that we hadn't anticipated. :)

@pfmoore Could you try this out with your PR for #8377, and see what that generates? I wonder if we would've generated a useful error message to hint that something was wrong in this case.

[pradyunsg]

I'm not 100% sure if this should block "stable" or beta, so I've put it in the "prod" pool.

[uranusjr]

Does this work on master? This is why we did the resolvelib 0.4 rewrite, if I read it correctly.

[webknjaz]

What is the procedure for testing against master?

[uranusjr]

Just install pip from GitHub:

~
python -m pip install -U "pip @ https://github.com/pypa/pip/archive/master.zip"
~

[webknjaz]

I'll try. Installing from the local Git repo exploded... Oh, it's not pip that explodes. It's Ansible's setup.py.

[pfmoore]

Hi @webknjaz - can I check, is the above comment confirming that there's not a pip issue here, but rather an issue with ansible? Or am I misunderstanding?

[webknjaz]

Yeah, it looks like it, sorry for the noise.

[pfmoore]

No worries, I appreciate you putting the time in to help test :-)