Pip: Output of "pip freeze" and "pip list --format=freeze" differ for packages installed via Direct URLs

[deveshks]

Originally found while triaging https://github.com/pypa/pip/issues/8174

Environment

• pip version: 20.1
• Python version: 3.8.2
• OS: OSX Catalina

Description

On performing an install using a direct URL in a virtualenv with wheel present, the output of pip freeze and pip list --format=freeze differ in the way the package is listed

Expected behavior

pip list --format=freezefollows the output format style of pip freeze, and so if the output of pip freeze changes in the case of installations with direct URL, pip list --format=freeze should follow suit and do the same.

How to Reproduce

1. Create a virtualenv
2. Upgrade pip in the virtualenv, and install wheel
3. Install a package using the direct URL (git VCS URL in this case)
4. See that the output of pip freeze and pip list --format=freeze differ

Output

$ python -m venv venv
$ source venv/bin/activate

(venv)$ pip install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.1

(venv) $ pip install wheel
Collecting wheel
  Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
Installing collected packages: wheel
Successfully installed wheel-0.34.2

(venv) $ pip install git+https://github.com/pypa/twine
Collecting git+https://github.com/pypa/twine
  Cloning https://github.com/pypa/twine to /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-req-build-shjhv6hl
  Running command git clone -q https://github.com/pypa/twine /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-req-build-shjhv6hl
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pkginfo>=1.4.2
  Using cached pkginfo-1.5.0.1-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: setuptools>=0.7.0 in ./venv/lib/python3.8/site-packages (from twine==3.1.2.dev58+gda8d62f) (41.2.0)
Collecting readme-renderer>=21.0
  Using cached readme_renderer-26.0-py2.py3-none-any.whl (15 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
  Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting requests>=2.20
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting keyring>=15.1
  Using cached keyring-21.2.0-py3-none-any.whl (30 kB)
Collecting tqdm>=4.14
  Using cached tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
Collecting bleach>=2.1.0
  Downloading bleach-3.1.5-py2.py3-none-any.whl (151 kB)
     |████████████████████████████████| 151 kB 252 kB/s 
Collecting docutils>=0.13.1
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting Pygments>=2.5.1
  Using cached Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting packaging
  Using cached packaging-20.3-py2.py3-none-any.whl (37 kB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Building wheels for collected packages: twine
  Building wheel for twine (PEP 517) ... done
  Created wheel for twine: filename=twine-3.1.2.dev58+gda8d62f-py3-none-any.whl size=36764 sha256=4d4a011ba94cc61d89b661977d66816de6b2dd9c23c189a4a77a68eda6997476
  Stored in directory: /private/var/folders/xg/blp845_s0xn093dyrtgy936h0000gp/T/pip-ephem-wheel-cache-_wbhcsnm/wheels/73/dd/fb/3039711df8645ab0ccc3e7d5f834289bf9ee7569c222c1fbc4
Successfully built twine
Installing collected packages: pkginfo, six, pyparsing, packaging, webencodings, bleach, docutils, Pygments, readme-renderer, chardet, certifi, urllib3, idna, requests, requests-toolbelt, keyring, tqdm, twine
Successfully installed Pygments-2.6.1 bleach-3.1.5 certifi-2020.4.5.1 chardet-3.0.4 docutils-0.16 idna-2.9 keyring-21.2.0 packaging-20.3 pkginfo-1.5.0.1 pyparsing-2.4.7 readme-renderer-26.0 requests-2.23.0 requests-toolbelt-0.9.1 six-1.14.0 tqdm-4.45.0 twine-3.1.2.dev58+gda8d62f urllib3-1.25.9 webencodings-0.5.1

(venv) $ pip freeze

....
twine @ git+https://github.com/pypa/twine@da8d62f6e8878900267a8499d4e3cfe49ac11002
....

(venv) $ pip list --format=freeze
....
twine==3.1.2.dev58+gda8d62f
....

[sbidoul]

pip list --format=freeze should follow suit and do the same.

@deveshks I'm personally not entirely sure both should do the same, or if the difference should be documented (and tested).

[uranusjr]

Maybe they should do different things, given that there’s indeed people wanting the old behaviour (i.e. #8174). But it is so entirely not obvious which one should do which. Maybe we should get the new inspect command going soon to replace them both. Is there a writeup for that somewhere @pradyunsg?

[xavfernandez]

Somewhat related to #7157.
And in this context, where pip freeze would be an alias to pip list --format=freeze (or pip inspect --format=freeze ?), this would be a bug :)
Ideally we should be able to output both format. Maybe a --format=freeze & --format=freeze-version ?

[deveshks]

I think if we want to start looking for a pip inspect to replace list and freeze, now along with supported options to keep relevant outputs from both, that should be able to make the 20.2 train in July if we come up with a documentation, and an implementation in the next 3 months?

After that, we can decide on the deprecation cycle to remove list and freeze.

Do we want to switch to https://github.com/pypa/pip/issues/7157 to start discussing on that, and include the issue here as part of that discussion? Or do we want to fix this in the next release, along with the inspect command?

[dragoljub]

Is there any way to prevent pip freeze from writing the direct path to the local wheel used to install from? Before Pip 20.1 and PEP 610 I could just pip freeze > requirements.txt after upgrading packages and not worry which directory I happen to be installing wheels from. This helps decouple local install paths from generic version requirements.

pip list --format=freeze seems to be one way that works to avoid the above issue.

[uranusjr]

pip wouldn’t record the package’s source URL if you don’t install the package by its path/URL:

pip install --no-index --find-links directory/containing/wheel package-name

[dragoljub]

@uranusjr that works if you only have one or two packages to install or already have a requirements.txt ready.

What about installing 30 new packages from wheels? I can easily script it to ‘’’pip install —no-index -f . [List of all wheels names in current directory]’’’. However, with new pip, freezing such an install brings the complete wheel path into the requirements.txt. :(

To follow your suggestion I would have to somehow strip out all the wheel file names down to just package names.

Is there another easier way to install all wheels In a directory regardless what wheels are in there?

[sbidoul]

Using bash, this is an easy way to get the package names:

pip install --no-index --find-links wheelsdir $(ls wheelsdir/*.whl | cut -d '-' -f 1)

At this stage, pip list --format=freeze also does what you want, as you noted. It is further being discussed in #8176. Ow, sorry, this is #8176.

[dragoljub]

I’m doing this in Windows 10, but the install is a python script. I’ll see about just stripping out the package names.

Yes, pip list --format=freeze covers this corner case for now. It would be good if this option remains if you want to build a pure requirements.txt regardless of installation from wheel or pypi.