Pip: extra is not honored with the dependency resolver

[gaborbernat]

Steps to reproduce:

env VIRTUALENV_PIP=20.2b1 virtualenv venv --clear; env PIP_UNSTABLE_FEATURE="resolver" venv/bin/pip install 'setuptools_scm >= 3.5[toml]'                                                                           created virtual environment CPython3.8.3.final.0-64 in 1969ms
  creator CPython3Posix(dest=/Users/bgabor8/git/dqc/ecobre/venv, clear=True, global=False)
  seeder FromAppData(download=False, pip=20.2b1, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bgabor8/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Looking in indexes: http://localhost:3141/bernat/bb
Collecting setuptools_scm>=3.5[toml]
  Downloading http://localhost:3141/bernat/bb/%2Bf/0d2/3db3d43e0a43e/setuptools_scm-3.5.0-py2.py3-none-any.whl (26 kB)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-3.5.0

without the resolver:

env VIRTUALENV_PIP=20.2b1 virtualenv venv --clear; env venv/bin/pip install 'setuptools_scm >= 3.5[toml]'                                                                                                            created virtual environment CPython3.8.3.final.0-64 in 2233ms
  creator CPython3Posix(dest=/Users/bgabor8/git/dqc/ecobre/venv, clear=True, global=False)
  seeder FromAppData(download=False, pip=20.2b1, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bgabor8/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Looking in indexes: http://localhost:3141/bernat/bb
Collecting setuptools_scm>=3.5[toml]
  Downloading http://localhost:3141/bernat/bb/%2Bf/0d2/3db3d43e0a43e/setuptools_scm-3.5.0-py2.py3-none-any.whl (26 kB)
Collecting toml; extra == "toml"
  Downloading http://localhost:3141/bernat/bb/%2Bf/bda/89d5935c2eac5/toml-0.10.1-py2.py3-none-any.whl (19 kB)
Installing collected packages: toml, setuptools-scm

This sadly is blocking me from doing more testing, as does not allow me to build a package I'll install afterwards.

[uranusjr]

Hmm. We read extras here:

https://github.com/pypa/pip/blob/3ec6b824ddbdef2128238858473bc80c7bf502e8/src/pip/_internal/resolution/resolvelib/requirements.py#L53-L58

But weirdly ireq.req.extras does not contain the extras, only ireq.extra does. I don’t really understand why, but will file a PR to change this anyway. We are already using ireq.extras in many other places in the resolver anyway, and I guess it makes sense to always do the same?

(The weird thing about this is that extras listed in a package’s Requires-Dist are correctly listed in ireq.req.extras, only the root requirements’ aren’t.)

[uranusjr]

Ah, I think I know what’s going on. setuptools_scm >= 3.5[toml] is actually not valid PEP 508 syntax, but pip accepts it anyway. The parser augments packaging’s Requirement to parse this additional part, and attaches that onto InstallRequirement to go along with the extras in Requirement.

So:

• setuptools_scm>=3.5[toml] will have ireq.extras == {'toml'} but ireq.req.extras == set().
• setuptools_scm[toml]>=3.5 will have ireq.extras == ireq.req.extras == {'toml'}.

We currently only tests the valid PEP 508 syntax variant, which is why this slipped. I’ll file the PR to switch to ireq.extras, but @gaborbernat you should be able to use the PEP 508 syntax to work around the issue.

[pradyunsg]

pip accepts it anyway.

Gah. This needs to be fixed. I'm OK to switch us to ireq.extras in the new resolver for now, but we should add a tracking issue to deprecate and remove the code that results in pip accepting the non-PEP 508 format.

[pfmoore]

but we should add a tracking issue

Done. #8288