Pip: Add support for outputting a list of extras and their requirements.

[software-opal]

• Pip version: 9.0.1
• Python version: 3.6
• Operating system: Ubuntu 17.10

Description:

I'm trying to write a script to automatically generate constraints files(with hashes) and I'd like a way to allow this script to detect any extras, and generate the constraints file for those too.

I have no preference to the command; but it might be nice to put it in pip show as another section(depending on compatibility)

What I've run:

$ pip show requests
Name: requests
Version: 2.18.4
...
Requires: idna, urllib3, chardet, certifi
ExtraRequires[security]: cryptography, pyOpenSSL
ExtraRequires[socks]: PySocks

[software-opal]

An alternative would be to create a pip metadata <package> command to output the <...>.dist-info/METADATA or <...>.dist-info/metadata.json files so that they can be parsed by downstream tooling.

[pradyunsg]

This seems to be deeply related and a better alternative to #3797.

[jaraco]

An even better alternative would be to use importlib_metadata, which has an API.

>>> import importlib_metadata
>>> importlib_metadata.metadata('xonsh').get_all('Provides-Extra')
['linux', 'mac', 'proctitle', 'ptk', 'pygments', 'win']
>>> importlib_metadata.metadata('xonsh').get_all('Requires-Dist')
["distro; extra == 'linux'", "gnureadline; extra == 'mac'", "setproctitle; extra == 'proctitle'", "prompt-toolkit; extra == 'ptk'", "pygments (>=2.2); extra == 'pygments'", "win-unicode-console; extra == 'win'"]

[jaraco]

And use packaging to parse them:

>>> req = next(map(packaging.requirements.Requirement, importlib_metadata('xonsh').get_all('Requires-Dist')))
>>> req.name
'distro'
>>> req.specifier
<SpecifierSet('')>
>>> req.extras
set()
>>> req.marker
<Marker('extra == "linux"')>

[glyph]

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one means.

Thanks for reading!

[pfmoore]

Given that, as @jaraco mentioned, the data is available via importlib_metadata and packaging, maybe someone could write and publish a standalone script to extract this data?

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip. (If nothing else, a standalone script could be the basis of a future PR for pip).

[exarkun]

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip.

The OP posted the issue a year and a half ago. I hope they already found a way to not be blocked in their endeavours. :) This is the pip issue tracker though. It seems appropriate to focus on how to support this use-case in pip here.

[exarkun]

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one _means_.

Thanks for reading!

One thing to note is that the OP seems to have been discussing gathering this information for programmatic consumption ("I'm trying to write a script to automatically ...") but this comment from @glyph is firmly focused on user education and so probably manual usage by a human. These are both good and interesting use-cases but they may not have the same solution.

[tsvikas]

while it might be a slight different enhancement than what OP meant, I think we all would be glad to have a pip search-extras <package>, that will output the available extras for an (uninstalled) package.
pip is a tool that allows us to list (search) packages & than install them, it allows us to install extras, but how can we discover them?
I believe it's more similar to what @glyph wanted?

Using the
import importlib_metadata; importlib_metadata.metadata('twisted').get_all('Provides-Extra')
requires a venv with importlib_metadata installed, and running python code - not an easy interaction when you are just trying to remember a name of an extra to install.

[pradyunsg]

Okay, this feature makes sense and we'd want to include it in pip. We welcome folks to submit a PR for implementing this functionality. Note that this does not mean that the said PR would be accepted - it would still be subject to our regular code reviews as with every other PR in pip.

The relevant bit of code is likely in src/pip/_internal/commands/show.py. Currently, pip uses pkg_resources and the main task is to figure out how to correctly compute the extras a package supports and getting the dependencies that are specified in that extra.

[uranusjr]

There are actually two features this thread talks about.

The top post describes showing showing extras in pip show. The functionality is simple to implement (as demostrated multiple times here), but the interface would require some thought. Personally I would prefer pip to ditch the current pip show formatand introduce something that shows the actual metadata (instead of a transformed version it currently outputs), which would greatly simplies things. Maybe a process like the previously happened pip list migration would be applicable here. If pip decides to stick with the current show format, it would be another problem to come up with a good format to display extras.

But that does not really help the discovery problem mentioned by @glyph, since show only works for installed packages. Currently there are essentially no ways packages could advertise their extras (except document them manually). The first solution that comes to mind is to show that on PyPI like Crates.io (at the bottom in the right-hand column). The problem though is that Python package requirements are declared per artifact, not per version, so thoeratically a package can declare different extras for its Windows and Linux wheels of the same version. This is a very weird things to do in almost all cases, but still it is a thing, and PyPI needs to accomodate that, making the feature difficult to design.

A more workable solution would be to let pip show work with non-installed wheels. The simplese addition would be to allow pip show [path-to-wheel] so people can combine it with pip download to achieve what they want. There’s another possible step pip can take to implement something like pip show --download [name], but that’s probably worth its own discussion.

I hope this thought dump would be marginally helpful for anyone interesting in implementing this feature.

[deveshks]

Personally I would prefer pip to ditch the current pip show format and introduce something that shows the actual metadata (instead of a transformed version it currently outputs), which would greatly simplies things.

The main suggestions provided in https://github.com/pypa/pip/pull/8008#pullrequestreview-391133428 which is the documentation PR for https://github.com/pypa/pip/pull/7967 (Adding json format display to pip show) is to add a metadata key in the json output which will have a dictionary containing the fields in https://packaging.python.org/specifications/core-metadata/ . That should be able to take care of listing extras and their requirements atleast for _installed packages_ .

[fabiencelier]

+1 for listing the extras of a package

We want to split our package into smaller packages linked as extras and it would be nice to see what are all the available extras.

Also I have open a stackoverflow question about that before finding this thread