Pip: Command to list all available versions pip finds for a package

[uranusjr]

What's the problem this feature will solve?
This is a feature suggestion I got during a user interview conducted by @ei8fdb. Currently pip does not seem to have a way to invoke PackageFinder without also select a “best match”. A common operation for users is however to look at the list of versions, in order to decide how they want to specify the version range.

Describe the solution you'd like
A command in pip to discover and output available versions for a package (or packages). It’d be also useful if the command can also include extra information, e.g. what dists are available for each version, and which version is currently installed, in a table form.

Something like:

$ py -3.8 -m pip discover numpy
I? version  sdist    wheel
   1.18.2   zip      cp38-cp38m-win_amd64 ...
v  1.18.1   zip      cp38-cp38m-win_amd64 ...
(more lines ...)

Each line represents a version. The sdist column shows what format the source dist is in (another common output would be tar.gz), and wheel shows compatible wheels. The dot dot dot at the ends indicates there are more wheels, but incompatible with the host. The I? column shows whether this version is installed. All fields except version can be blank if there’s not a match found.

Similar to install and download, the command would accept index and format options, and filter the output accordingly.

Alternative Solutions
The best solution currently is to manually look at the index directly, but this can be especially cumbersome if the package is not on pypi.org. And even for pypi.org, it is cumbersome to either fire up a browser, or directly use the JSON or simple API.

One other hack the user reported using is to tell pip to install a non-existent version (pip>1000000). pip would output available versions in the error message 😆

Additional context
I didn’t think of any.

[McSinyx]

I think I can work on this. Is there any option we should consider for this new subcommand?

[uranusjr]

I think first we need to reach some concensus among the maintainers this is actually a good idea 🙂

[pfmoore]

+1 from me. This is something I've wanted on a number of occasions.

The layout as described sounds reasonable to me. I can imagine further feature requests asking for extra information in the "I?" column, like "this version would be chosen for installation/upgrade", or "this version is incompatible with what's already installed". Those ideas may be impractical to add, though, and we should be driven by actual use cases, not just by what sounds nice to have (both of those ideas are things I think I'd use in practice, but I have lived without them till now, so they are far from essential).

@uranusjr Do you have any means of involving the user who suggested this, to get their direct input?

[xavfernandez]

+1 from me also on the general idea.

[uranusjr]

Do you have any means of involving the user who suggested this, to get their direct input?

Let me ask Bernard on Zulip, I believe he has ways to contact the interviewee.

[uranusjr]

I played around with this and put together a POC. Current output is like this:

> py src\pip find --python-version=3.6 "numpy>=1.15,<1.17"
version sdist wheel
------- ----- --------------------
1.16.6  zip   cp36-cp36m-win_amd64
1.16.5  zip   cp36-cp36m-win_amd64
1.16.4  zip   cp36-cp36m-win_amd64
1.16.3  zip   cp36-cp36m-win_amd64
1.16.2  zip   cp36-cp36m-win_amd64
1.16.1  zip   cp36-cp36m-win_amd64
1.16.0  zip   cp36-cp36m-win_amd64
1.15.4  zip   cp36-none-win_amd64
1.15.3  zip   cp36-none-win_amd64
1.15.2  zip   cp36-none-win_amd64
1.15.1  zip   cp36-none-win_amd64
1.15.0  zip   cp36-none-win_amd64

Code is available at https://github.com/uranusjr/pip/tree/pip-find-versions

[McSinyx]

What do you think about adding a column for package name, so that if multiple packages are requested, the output can be easily processed by common CLI utilities?

[gutsytechster]

One other hack the user reported using is to tell pip to install a non-existent version (pip>1000000). pip would output available versions in the error message

I actually do this often. I also think it'd be great to have this feature.
+1 from my side as well(as a pip user)

[McSinyx]

@uranusjr, may I continue the feature on top of your branch so you can focus on other works?

[uranusjr]

Sure. Regarding the package name, I don’t like the idea since the output would be very repetitive. In the POC I opted to disallow supplying multiple packages instead. An alternative would be to put the name above the table instead. A JSON format can be added for machine-readable output.

One decision I’m pondering on is how to order the versions. The POC puts the latest on top, but I’m thinking maybe it makes more sense to put the latest the the bottom instead, since it’s easy to forget to head the output, and the current ordering requires the user to scroll all the way back for relevant output. OTOH though latest-on-top does make more sense visually (the reverent rows are closer to the header). One alternative would be to introduce a “display x rows at most” flag and set it to a reasonable default, since it is very uncommon for the user to want to see all versions.

[McSinyx]

disallow supplying multiple packages

This seems fine to me, until we want to speed up by parallelization (GNU parallel doesnt work since we still need synchronized outputs).

An alternative would be to put the name above the table instead. A JSON format can be added for machine-readable output.

At this point it might be a good idea to provide both human readable and machine-friendly one. JSON IMHO is not something many'd prefer for quick and dirty scraping though.

aptitude's take

$ aptitude versions '~npython3-aiohttp' | cat # for more compact and GitHub-friendly output
Package python3-aiohttp:
p  3.6.2-1+b1 testing 500

Package python3-aiohttp-cors:
p  0.7.0-1 testing 500

Package python3-aiohttp-dbg:
p  3.6.2-1+b1 testing 500

Package python3-aiohttp-jinja2:
p  1.2.0-1 testing 500

Package python3-aiohttp-mako:
p  0.4.0-1 testing 500

Package python3-aiohttp-security:
p  0.4.0-2 testing 500

Package python3-aiohttp-session:
p  2.9.0-2 testing 500

Package python3-aiohttp-socks:
p  0.3.4-1 testing 500

Package python3-aiohttp-wsgi:
p  0.8.2-2 testing 500
$ aptitude versions --group-by=none '~npython3-aiohttp' | cat
p  python3-aiohttp 3.6.2-1+b1 testing 500
p  python3-aiohttp-cors 0.7.0-1 testing 500
p  python3-aiohttp-dbg 3.6.2-1+b1 testing 500
p  python3-aiohttp-jinja2 1.2.0-1 testing 500
p  python3-aiohttp-mako 0.4.0-1 testing 500
p  python3-aiohttp-security 0.4.0-2 testing 500
p  python3-aiohttp-session 2.9.0-2 testing 500
p  python3-aiohttp-socks 0.3.4-1 testing 500
p  python3-aiohttp-wsgi 0.8.2-2 testing 500

latest-on-top does make more sense visually (the reverent rows are closer to the header). One alternative would be to introduce a “display x rows at most” flag and set it to a reasonable default, since it is very uncommon for the user to want to see all versions.

I agreed with this, although it'd be better if we can have more input from the UX research.

[pradyunsg]

I feel like this makes sense, and we should likely look into aggregating all of our "search, introspection and discovery" commands (like pip search, pip list, pip show, this proposal, proposed pip find and I'm sure there's more) into a better set of not-one-trick-pony commands.

I feel like it's possible to come up with something that's easier to mentally model, and the lower "CLI surface area" will also be easier to maintain as well. Yes, it will require changing more things but I feel that such a fix would improve the overall situation enough to justify the costs.

OTOH, there's no hurry here and seeing what other "gaps" are found during the user research interviews... would be useful. :)

[McSinyx]

About the command name, I'm not quite favoring either discover or find for

list all available versions [of] a package

The reason is simply because they seem to imply some discovery, i.e. something more than what the package name can cover (also search, find and discover are synonyms/result of each other). My suggestion as an user of many *nix package managers would be to either

1. Incorporate the feature as part of pip list, given the similarity of the output format
2. Use a IMHO more intuitive name like pip versions

I feel like it's possible to come up with something that's easier to mentally model, and the lower "CLI surface area" will also be easier to maintain as well.

IIUC, this also has to do with the growing complexity of pip subcommands. Although admittedly it is convenient (and less to be maintained) to be able to share options between subcommands if we parse them altogether, in longer run it might be beneficial to split them into independent executables, i.e. in the way git does it. I cannot estimate how much work (research and implementation) that would cost though.

[sbidoul]

@uranusjr I your example you have a --python-version. Is it the intent to add options to filter on other compatibility tags? Or is the command expected to return only files that are compatible with the current python?

[uranusjr]

@sbidoul My intention was to use it to filter on wheel compatiblity tags and data-requires-python attributes on indexes.

[nlhkabu]

I've added this ticket to the functionality research epic (#8516)
I'll look into how users understand the terms discover and find vs the current search command. My gut feeling is that there will be some confusion.
I'll also research whether this functionality could sit under @pradyunsg's suggestion of creating a pip inspect command (maybe pip inspect --available-versions could work)