Pip-tools: pip-compile doesn't support the new pip resolver

[rouge8]

pip-compile doesn't support using the 2020 resolver.

Environment Versions

1. OS Type macOS
2. Python version: $ python -V: Python 3.7.8
3. pip version: $ pip --version: pip 20.2
4. pip-tools version: $ pip-compile --version: pip-compile, version 5.3.0

Steps to replicate

1. Create t.txt:
   --use-feature=2020-resolver boto3
2. Run pip-compile t.txt:

Traceback (most recent call last):
  File ".tox/pip-compile/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/scripts/compile.py", line 458, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 169, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 274, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/resolver.py", line 384, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/local.py", line 75, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 232, in get_dependencies
    download_dir, ireq, wheel_cache
  File "/Users/andy/gr/gold/.tox/pip-compile/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 184, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
AttributeError: 'Resolver' object has no attribute '_resolve_one'

Expected result

pip-compile completes successfully.

Actual result

pip-compile errors out.

[atugushev]

Hello @rouge8,

Thanks for the issue! Yes, currently, pip-tools doesn't support new pip's resolver. I think it makes sense to adopt it after the resolver is being stable so that we can use the internal API.

[rouge8]

Well pip never considers the internal API stable right? :)

For my purposes I don’t actually need the new resolver logic except to deal with https://github.com/pypa/pip/issues/5780. If you don’t think the new resolver internals are stable enough for pip-tools yet, I’m sure I can come up with some hack to work around this issue. :)

[salotz]

I got a different error (but I suspect related to this issue) and I didn't specify the --use-feature=2020-resolver flag in any file and using pip 20.2.1 :

pip-compile  --output-file=requirements.in ./requirements.in ../dirA/requirements.in ../dirB/requirements.in

Traceback (most recent call last):
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/scripts/compile.py", line 392, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/resolver.py", line 169, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/resolver.py", line 255, in _resolve_one_round
    constraints = sorted(self.constraints, key=key_from_ireq)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/resolver.py", line 128, in constraints
    self._group_constraints(chain(self.our_constraints, self.their_constraints))
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/resolver.py", line 231, in _group_constraints
    self.repository.get_dependencies(ireq)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/repositories/local.py", line 74, in get_dependencies
    return self.repository.get_dependencies(ireq)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 228, in get_dependencies
    download_dir, ireq, wheel_cache
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/piptools/repositories/pypi.py", line 180, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/salotz/tree/lab/devel/wepy/_conda_envs/dev/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 425, in _resolve_one
    assert req_to_install.user_supplied
AssertionError

[atugushev]

@salotz which pip-tools version is used?

[salotz]

apologies, that was 5.1.2, works with 5.3.1 :+1: @atugushev

[groodt]

I'm hitting this error too.

The new resolver is going to be the default from middle or end of October if I understand correctly.
https://github.com/pypa/pip/issues/8937
(This will be very exciting)

It probably makes sense to ensure pip-tools works with both --use-feature=2020-resolver and --use-deprecated-feature=legacy-resolver

[AndydeCleyre]

pypa/pip#988 (comment):

We are aiming to release pip 20.3 about a week from now, on Wednesday or Thursday, Oct 28 or 29. We are preparing to change the default dependency resolution behavior and make the new resolver the default in pip 20.3.

@atugushev @graingert How much do you already know about what needs to be done here in pip-tools?

[ssbarnea]

This issue is critical as the new resolver is days away from becoming default. If you was one of those that added use-feature = 2020-resolver to your ~/.config/pip/pip.conf in order to benefit from extra conflict resolution in newer resolver you are unable to use pip-compile at all.

[hynek]

So weirdly the 5.4 release seems to have fixed this for Python 3. But it still breaks for Python 2 (I know, I know). Can someone confirm this behavior, or is it just me?

[atugushev]

@hynek pip-tools==5.4.0 adds only pip==20.3 support. I've tried locally py27, but works fine. Could you create an issue with a traceback?

[hynek]

I…uh…played a bit with the settings hence and forth and suddenly it started working and I can't break it anymore. 😳 It was 100% 5.4.0/20.3 before. I guess my computer is just haunted.

[webknjaz]

Well pip never considers the internal API stable right? :)

Can't we just use resolvelib directly instead of the pip's vendored copy? This way pip-tools wouldn't need to rely on the internal pip's resolver wrapper but use it directly. Its API is quite simple (I'm in the progress of integrating resolvelib into ansible-galaxy CLI, for example) — using it would mean having a stable public API to rely on but the behavior may differ from what a given version of pip does, of course.

[ssbarnea]

This bus is still a big PITA because even latest version will choke if you happen to define:

export PIP_USE_FEATURE=2020-resolver

This used to be a popular method to enable the new resolver before it was released, so we would detect issues at CI level.

[hynek]

I believe you can work around that for now using pip check?

[ssbarnea]

Well,.. I removed that line from that CI pipeline that was running pip-compile, but having to do this on multiple projects is no joy. We should still have to fix it. Sadly lots of people interested but nobody willing to do it.

[brycedrennan]

Is anyone working on this? Thinking about taking a look but don't want to duplicate work.

[groodt]

This should actually be working now. I think this issue could be closed.

Which version of pip are you seeing errors with? I use pip==21.0.1 (new resolver enabled by default) with the latest pip-compile and things work correctly.

[brycedrennan]

okay well I have something that works with pip install but not pip compile. I'll see if I can boil it down to the base case."

python 3.7.10
pip 21.0.1
pip-tools 6.0.1

[pohmelie]

I have simplest example:
req.txt:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0

pip-compile fails with:

Could not find a version that matches asyncpg>=0.22.0,~=0.15.0 (from -r req.txt (line 1))
Tried: 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.1, 0.6.3, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.9.0, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.20.0, 0.20.0, 0.20.1, 0.20.1, 0.21.0, 0.21.0, 0.22.0, 0.22.0
There are incompatible versions in the resolved dependencies:
  asyncpg~=0.15.0 (from -r req.txt (line 1))
  asyncpg>=0.22.0 (from asyncpgsa==0.27.1->-r req.txt (line 2))

but pip install succeed with:

...
Successfully installed asyncpg-0.15.0 asyncpgsa-0.27.0 greenlet-1.0.0 sqlalchemy-1.4.6

Tested on both pypi and github-master versions of pip-tools. Latest pip (since pip-tools required it).

If I change req.txt as:

asyncpg ~= 0.15.0
asyncpgsa >= 0.22.0, < 0.27.1

Then I get pip-compile succeed with:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile req.txt
#
asyncpg==0.15.0
    # via
    #   -r req.txt
    #   asyncpgsa
asyncpgsa==0.27.0
    # via -r req.txt
greenlet==1.0.0
    # via sqlalchemy
sqlalchemy==1.4.6
    # via asyncpgsa

[brycedrennan]

confirmed pip-compile can't handle,

botocore==1.20.20

but pip finds valid solution:

boto3==1.17.20
botocore==1.20.20

So sounds like no ones working on this?