pip show --json

[anntzer]

• Pip version: 10
• Python version: any
• Operating system: any

Description:

This is a feature request, related to #3568 and 4824, but made more necessary by the privatization of pip's internals: let pip show gain a --json flag (or --format=(humanreadable|json)) (also handling pip show --files) that outputs the result as json (or whatever other format you deem suitable, but pip list already has --format=json (hence the suggestion of --format=json instead of --json above). Otherwise, it becomes necessary to either parse the output of pip show, or to just copy-paste pip's implementation, in order to access the same metadata.

What I've run:

e.g.

pip show pip

[pradyunsg]

Why not utilize PyPI's APIs directly?

See: https://pypi.org/pypi/pip/json and https://warehouse.pypa.io/api-reference/json/

If that resolves your concern, feel free to close this issue. :)

[anntzer]

pip show doesn't need internet access, and works with packages that only exist locally.

[pfmoore]

IMO, parsing the output of pip show isn't that hard (it was designed to be both human readable and machine parseable). It's standard RFC-compliant mail header format, so you can use the stdlib email.parser.BytesHeaderParser class to parse the output of a subprocess call to pip show --files without needing external dependencies or any code of your own:

>>> import subprocess
>>> from email.parser import BytesHeaderParser
>>> p = subprocess.run([sys.executable, '-m', 'pip', 'show', 'pip', '--files'], stdout=subprocess.PIPE)
>>> h = BytesHeaderParser().parsebytes(p.stdout)
>>> print(h['Name'])
pip

But I don't see any reason why we would reject a PR that added a --json argument to pip show. I just doubt it's something we'll get to ourselves particularly soon.

[pradyunsg]

pip show doesn't need internet access, and works with packages that only exist locally.

Indeed. Silly me.

That said, I second everything @pfmoore said above.

[anntzer]

Fair enough. I don't think the format is document though (is it intended to be stable?).

[pfmoore]

It's shown in the docs, but I take your point.

[pradyunsg]

Happy to accept a PR that adds this to the documentation.

[kunalJa]

Hi! I was wondering, because solving this problem requires adding an option to show.py, how exactly does the _dest_ work in _self.cmd_opts.add_option()_? Does it need to be an attribute pre-existing in options in order to get the value given to it by the action?

[minggli]

Hi, it appears that the PR related to this issue has been merged already. Can we consider close this issue?

[MarckK]

Hi @minggli, the PR improves documentation on existing functionality. @kynan and I are working on fixing the issue. :)

[kynan]

@pradyunsg Please move to in progress

[weakcamel]

I admit I'm hearing - perhaps wrongly - mixed messages on this issue and would appreciate clarification:

• the referenced https://github.com/pypa/pip/pull/5514 PR has been closed with conclusion that
  the output of pip show is already machine readable, therefore --json flag is not needed (see @pfmoore comments)
• commits from @MarckK indicate that this work is to be continued after all

What's the official stance then?

Personally, I would find --json output very useful.

I'm just in the middle of "slurping" output of pip show and have discovered, that it is not in RFC-compliant format after all, if you choose/need to use it with more than 1 package at a time. Namely, while the headers are fine, the separator --- between the packages is actually not, hence the standard email.parser.Bytesparser is unable to handle it on its own. Example:

$ pip show pip six
Name: pip
Version: 18.1
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [email protected]
License: MIT
Location: .../lib/python3.7/site-packages
Requires: 
Required-by: 
---
Name: six
Version: 1.12.0
Summary: Python 2 and 3 compatibility utilities
Home-page: https://github.com/benjaminp/six
Author: Benjamin Peterson
Author-email: [email protected]
License: MIT
Location: .../lib/python3.7/site-packages
Requires: 
Required-by: pytest, more-itertools, astroid

Of course it's easy to work aroud, but it means either implementing your own parser, or splitting the (file) content into parts before feeding them into the email.parser.BytesParser which is suboptimal.

[pradyunsg]

@weakcamel Thanks for flagging this. FYI @MarckK opened a new PR at #5954, which is showing up in this issue.

/cc @xavfernandez @cjerdonek

[xavfernandez]

Since --format=json has already been added to pip list I'm in favor of adding the same to pip show.

[kunalJa]

This pull request that we initially created adds --json to pip show, but it was deemed unnecessary. The code is already created for an older version of pip (summer 2018) https://github.com/kunalJa/pip
but again- it was deemed unnecessary.

[achillesrasquinha]

Is this still considered? I don't see pip show RFC compliant for multiple packages.

[uranusjr]

@achillesrasquinha Would you mind provide some examples? We’ll need to decide

• Whether the non-compliance is a result of pip not rendering the fields correctly (and fix it if so)
• Whether the email header format is fundamentally unsuitable for pip show, and try to move to more structured formats (including JSON)

[chrahunt]

One note

Otherwise, it becomes necessary to either parse the output of pip show, or to just copy-paste pip's implementation, in order to access the same metadata.

This has never been necessary - pip itself uses pkg_resources (see docs) and Python 3.8 has a standard library module importlib.metadata for retrieving the same information. The latter also has a backport available which is compatible with Python 2.7 and 3.5+ here.

For example:

python -V
python -m pip show pip
python -c "from importlib.metadata import metadata; print(metadata('pip'))"
python -c "import pkg_resources; print(pkg_resources.get_distribution('pip').get_metadata('METADATA'))"

On my machine:

Output

Python 3.8.0
+ python -m pip show pip
Name: pip
Version: 19.2.3
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [email protected]
License: MIT
Location: /home/chris/.pyenv/versions/3.8.0/lib/python3.8/site-packages
Requires:
Required-by:
+ python -c from importlib.metadata import metadata; print(metadata('pip'))
Metadata-Version: 2.1
Name: pip
Version: 19.2.3
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [email protected]
License: MIT
Keywords: distutils easy_install egg setuptools wheel virtualenv
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*

pip - The Python Package Installer
==================================

.. image:: https://img.shields.io/pypi/v/pip.svg
   :target: https://pypi.org/project/pip/

.. image:: https://readthedocs.org/projects/pip/badge/?version=latest
   :target: https://pip.pypa.io/en/latest

pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.

Please take a look at our documentation for how to install and use pip:

* `Installation`_
* `Usage`_

Updates are released regularly, with a new version every 3 months. More details can be found in our documentation:

* `Release notes`_
* `Release process`_

If you find bugs, need help, or want to talk to the developers please use our mailing lists or chat rooms:

* `Issue tracking`_
* `Discourse channel`_
* `User IRC`_

If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:

* `GitHub page`_
* `Dev documentation`_
* `Dev mailing list`_
* `Dev IRC`_

Code of Conduct
---------------

Everyone interacting in the pip project's codebases, issue trackers, chat
rooms, and mailing lists is expected to follow the `PyPA Code of Conduct`_.

.. _package installer: https://packaging.python.org/en/latest/current/
.. _Python Package Index: https://pypi.org
.. _Installation: https://pip.pypa.io/en/stable/installing.html
.. _Usage: https://pip.pypa.io/en/stable/
.. _Release notes: https://pip.pypa.io/en/stable/news.html
.. _Release process: https://pip.pypa.io/en/latest/development/release-process/
.. _GitHub page: https://github.com/pypa/pip
.. _Dev documentation: https://pip.pypa.io/en/latest/development
.. _Issue tracking: https://github.com/pypa/pip/issues
.. _Discourse channel: https://discuss.python.org/c/packaging
.. _Dev mailing list: https://groups.google.com/forum/#!forum/pypa-dev
.. _User IRC: https://webchat.freenode.net/?channels=%23pypa
.. _Dev IRC: https://webchat.freenode.net/?channels=%23pypa-dev
.. _PyPA Code of Conduct: https://www.pypa.io/en/latest/code-of-conduct/



+ python -c import pkg_resources; print(pkg_resources.get_distribution('pip').get_metadata('METADATA'))
Metadata-Version: 2.1
Name: pip
Version: 19.2.3
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: [email protected]
License: MIT
Keywords: distutils easy_install egg setuptools wheel virtualenv
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*

pip - The Python Package Installer
==================================

.. image:: https://img.shields.io/pypi/v/pip.svg
   :target: https://pypi.org/project/pip/

.. image:: https://readthedocs.org/projects/pip/badge/?version=latest
   :target: https://pip.pypa.io/en/latest

pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.

Please take a look at our documentation for how to install and use pip:

* `Installation`_
* `Usage`_

Updates are released regularly, with a new version every 3 months. More details can be found in our documentation:

* `Release notes`_
* `Release process`_

If you find bugs, need help, or want to talk to the developers please use our mailing lists or chat rooms:

* `Issue tracking`_
* `Discourse channel`_
* `User IRC`_

If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:

* `GitHub page`_
* `Dev documentation`_
* `Dev mailing list`_
* `Dev IRC`_

Code of Conduct
---------------

Everyone interacting in the pip project's codebases, issue trackers, chat
rooms, and mailing lists is expected to follow the `PyPA Code of Conduct`_.

.. _package installer: https://packaging.python.org/en/latest/current/
.. _Python Package Index: https://pypi.org
.. _Installation: https://pip.pypa.io/en/stable/installing.html
.. _Usage: https://pip.pypa.io/en/stable/
.. _Release notes: https://pip.pypa.io/en/stable/news.html
.. _Release process: https://pip.pypa.io/en/latest/development/release-process/
.. _GitHub page: https://github.com/pypa/pip
.. _Dev documentation: https://pip.pypa.io/en/latest/development
.. _Issue tracking: https://github.com/pypa/pip/issues
.. _Discourse channel: https://discuss.python.org/c/packaging
.. _Dev mailing list: https://groups.google.com/forum/#!forum/pypa-dev
.. _User IRC: https://webchat.freenode.net/?channels=%23pypa
.. _Dev IRC: https://webchat.freenode.net/?channels=%23pypa-dev
.. _PyPA Code of Conduct: https://www.pypa.io/en/latest/code-of-conduct/

This has several upsides vs implementing pip show --format=json:

• available right now
• potentially hundreds of times more efficient
• doesn't pin an application or script to requiring a specific pip version
• much cleaner interface than a subprocess call and parsing json output

[uranusjr]

One scenario pip show --json would be useful is when you want to inspect another environment (not the one you’re running in). pkg_resources is probably still a good enough alternative right now since it is basically ubiquitous, but writing a compatibility layer going forward may be a chore compared to simply ask pip about it.

[chrahunt]

What does "compatibility layer" mean in this context?

[uranusjr]

I was kind of speculating into the far future (probably too far). With PEP 517 gaining usage, I assume setuptools could be less required in an environment, and some people will start trying to remove it in the future (and just have pip). pkg_resources wouldn’t be available in such scenario, so a tool would need to write code that works with both pkg_resources and importlib.metadata so it can inspect the most environments.

[weakcamel]

@chrahunt

This has never been necessary - pip itself uses pkg_resources (see docs) and Python 3.8 has a standard library module importlib.metadata for retrieving the same information.

The example you're showing isn't doing what pip show would, it doesn't support all the options pip does.

Sure, it can be extended to emulate what pip show --json would do, but then it would be longer (and as such, would re-implement pip partially).

Why not have support the output format in pip show itself, especially that pip list does that?

much cleaner interface than a subprocess call and parsing json output

Are you saying that it's a cleaner interface to call Python interpreter from shell/Perl/C++ code and feed it with an inline script to execute? ;-) I beg to differ.

[weakcamel]

@uranusjr

Whether the non-compliance is a result of pip not rendering the fields correctly (and fix it if so)

See my comment above: https://github.com/pypa/pip/issues/5261#issuecomment-453141518

The RFC non-compliant part is the separator between the packages. IMO the email header format is a bit unfortunate choice in case of >1 packages; to make it RFC compliant, pip would need to implement "multipart" Content-Type and what would it even mean in context of pip?

IMO the easiest way forward would be to leave the existing legacy format as-is and just provide a format flag to be automation-friendly.

[xavfernandez]

A case where pip show --json is IMHO simpler/more natural than pkg_resources/importlib.metadata is for non-python programmers that just want an easy way to retrieve package information.

[deveshks]

Hi Folks,

I have taken an initial stab at adding json format to pip show at https://github.com/pypa/pip/pull/7967 by taking out the part of constructing info needed by the default formatter from the default printing logic. The idea is to plug in the json printing logic in another PR.

Please feel free to take a look at the implementation and critique it :)