pip freeze does not list repo url

[DataGreed]

Here's the detailed problem: http://stackoverflow.com/questions/11560056/pip-freeze-does-not-show-repository-paths-for-requirements-file

[qwcode]

pip's ability to discern where the install came from is limited after the fact using pip freeze.
pip internally relies upon setuptools to find all the installed distributions.
pip then checks to see if any of them are "editable"/-e vcs installations.
if editable, then it will print out the vcs url.
if not editable, then just the name=version spec.

btw, since you asked about "-e" in the stack link, "-e" or "editable" is an enhancement of what setuptools called a "develop" install. this is an "installation" that does not occur via copying packages into site-packages.
but rather a link of sorts is made in site-packages back to the source location of the project.
that way you can "develop" or "edit" the project live while it's installed.
for "-e" vcs installs, pip uses a "src" subdirectory of the virtualenv or sys.prefix to clone the project.
if you install from a vcs url w/o "-e", pip builds a distribution locally and installs from that, and at that point pip freeze (i.e. setuptools) doesn't know it came from vcs.

[xavfernandez]

A typical example is given in #4005.

$ pip install git+https://github.com/chadmiller/textbot_metapriv.git#egg=textbot_metapriv   # a private package that will never be in PyPI
$ pip freeze |grep bot
textbot-metapriv==1.0

[pradyunsg]

@xavfernandez is this still an issue?

[pradyunsg]

is this still an issue?

Yes.

[sbidoul]

What's the problem this feature will solve?

Abstract: this issue is becoming more acute now that pep517 is a thing and it does not support editable installs.

When installing vcs dependencies without --editable, the exact installed version is lost when subsequently doing a pip freeze. When installing with --editable, it is preserved.

Preserving the installed vcs reference is actually my main (if not only) use case for using --editable.

My problem today is that pep517 does not support --editable, so if I want to pip install from vcs a pep517 project which has no setup.py, I loose the ability to do a reliable pip freeze.

Here is an example installing a pep517 project in a virtualenv (flit):

$ mktmpenv -p python3
$ pip install git+https://github.com/takluyver/flit.git
$ pip freeze
certifi==2019.3.9
chardet==3.0.4
docutils==0.14
flit==1.3
idna==2.8
pytoml==0.1.20
requests==2.21.0
urllib3==1.24.1

So with this method, I loose the information about the exact flit commit installed, and pip freeze does not give me something reproducible.

If I try with -e, it fails because pip install -e assumes setuptools is present:

$ mktmpenv -p python3
$ pip install -e git+https://github.com/takluyver/flit.git#egg=flit
[...]
Installing collected packages: certifi, chardet, urllib3, idna, requests, pytoml, docutils, flit
  Running setup.py develop for flit
    Complete output from command /home/sbi/.virtualenvs/tempenv-04f816243221f/bin/python3 -c "import setuptools, tokenize;__file__='/home/sbi/.virtualenvs/tempenv-04f816243221f/src/flit/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'

Describe the solution you'd like

I understand adding editable support to pep517 is a hard problem. Perhaps it is not needed for this use case, and a simpler approach would be to have pip preserve the installed vcs reference somewhere to restitute it in pip freeze?

I'm willing attempt a PR, but I'm not sure where pip could store this information, assuming such a place exists at all today? Could someone give me some guidance as to where to start?

[cjerdonek]

Could someone give me some guidance as to where to start?

I don't think what you're asking has a simple or clear solution, so I'd suggest beginning by gathering / brainstorming ideas. Also, is this a feature that you're thinking would be limited to things installed with pip?

[sbidoul]

@cjerdonek Thanks for jumping in. Here are some thoughts.

So we'd need some sort of metadata in dist-info. Perhaps a VCSORIGIN field?

• it would be specified to be a frozen VCS reference (a git commit, not branch or tag)
• it would supersede the version for the purpose of pip freeze
• it would supersede the version as wheel cache entry key

Could work, but I dot not feel very comfortable with it and when thinking about this I always come back to editable installs.

I played a bit with poetry and flit.

In presence of a git dependency, poetry always does an "editable" install, in the sense it creates a .egg-link, so pip freeze works.

Flit, like pip install without -e, does nothing special and loses the VCS origin after installation so pip freeze does not work.

OTOH, flit has an interesting 'install --symlink' which generates a regular .dist-info, but does not copy the source code and symlinks it instead (or plays with .pth files in a variant that supports platforms without symlinks). pip freeze does not work in this case either because it exclusively relies on the presence of .egg-link to find the source location.

I like what flit does here very much, as it uses dist-info metadata and is very flexible

So a better approach could be going all the way to implement "editable" installs in pep 517? I could not find the discussion that led to its temporary exclusion from pep 517. From the use cases I can imagine it could be relatively simple along these lines:

• backend provides prepare_metadata_for_install_editable() and install_editable()
• backend has the choice to install with the .egg-link method, or .dist-info + symlink, or .dist-info + any other mechanism it wants (.pth, etc)
• when doing an editable install, frontends (or should it be backend?) add a LOCATION metadata to record the source path of the install
• pip (or pkg_resources?) uses the new LOCATION metadata to detect editable installs so pip freeze can continue working normally
• when editable installs create a .dist-info, pip uninstall would work normally (no need to special case, since the RECORD would simply contain the symlinks to be removed)

If we can go with the editable install approach, we could then close this issue as not supported and direct people who need it to use --editable.

Perhaps I'm opening too big a can of worms? Yet I think editable installs must become part of pep 517 sooner than later, and this could be a good motivation to move it forward.

Is it the right place to brainstorm this? Should we involve other people here or open the discussion somewhere else?

[cjerdonek]

Should we involve other people here or open the discussion somewhere else?

If you wanted a wider discussion, the best place is probably to start a new topic under the "Packaging" category of the Python Discourse instance: https://discuss.python.org/c/packaging

I'm not sure how much traction you'll get though because PEP 517 was complex and highly contentious and drawn out (one of the reasons for removing editable), and there was an update even as recently as a month or so ago. So I'm guessing the participants will need a breather before picking up a new, hard topic like this that has long-term implications.

But it could help to get people thinking and get some initial thoughts.

[sbidoul]

The discussion on the spec which is a prerequisite to implement this is at https://discuss.python.org/t/draft-pep-recording-the-origin-of-distributions-installed-from-direct-url-references/1535

[DataGreed]

Wow, never expected that my feature request from seven years ago would still be discussed and even had a spec written on it.

You have my respect.

[chrahunt]

Related PEP: python/peps#1145

[sbidoul]

Chris being apparently unavailable, it would be really cool if another PyPA member could help moving the PEP forward. https://github.com/python/peps/pull/1145#issuecomment-553635782

[sbidoul]

Ok, it's now PEP 610 :tada:

So question to pip maintainers: do you think all lights are green to start implementation, and the blocked label can be changed to awaiting PR? I which case I'll start working on this in the coming weeks with the hope it can land in pip 20.0.

[pfmoore]

The PEP needs to be accepted before I'd worry about implementing it. Look at PEP 1 for information on the lifecycle of a PEP - you're basically at the "Submitting a PEP" phase now.

[sbidoul]

Ah, right, looks like I'm trying to skip steps... Any suggestion about what to do next? Can we consider the "Submitting a PEP" phase is done? Should it be sent to python-dev for review? Or should the review take place in the discourse topic?

[pfmoore]

If @cjerdonek isn't available, you'll need to find a sponsor to help you through the next steps. They'd be able to answer the questions you ask here - I'm afraid I don't have the free time at the moment to take on the role of sponsor for this PEP, sorry.

[pradyunsg]

Or should the review take place in the discourse topic?

This -- you'd want to start a new topic on Discourse, for discussing the PEP. My personal preference would be to start the topic in the Packaging category, though you can do it in the PEPs category too.

(FWIW, I can't be the sponsor for the PEP -- not a core dev so there's that)

[sbidoul]

@pradyunsg @pfmoore a few more questions.

I note the Discussion-To field points to the existing discourse topic, so is it not better to do the review discussion there?
Is there a list of core devs that are also active in packaging available somewhere so that I'd know who to ping to find a sponsor and BDFL delegate?
Do these two need to be different persons?
Is the sponsor still required since the editorial work is done and a core dev co-authored?

[pfmoore]

Don't worry about BDFL-Delegate, as I have that role for all packaging interop PEPs.

A sponsor is a requirement for the PEP process, so I think you do need one. Also, the fact that you have all these questions is precisely what having a sponsor should address. It's difficult to argue that you don't need one because @cjerdonek co-authored, while still pointing out that he's currently not available and hence you need all of these questions answered ;-)

Honestly, I think you may be better just waiting for @cjerdonek to be available to assist - I'm not clear why this PEP is so urgent that it couldn't wait for that.

[sbidoul]

I'm not clear why this PEP is so urgent that it couldn't wait for that.

In good open source tradition, I have an itch that I can scratch by implementing this issue (which was tagged awaiting PR back then).

In the grand scheme of things, nothing is urgent of course. This issue is 7 years old. I proposed to work on it 7 months ago, did all the PEP writing and editorial work with Chris (4 months), then waited 3 more months after opening the PEP PR, before concluding that Chris was not available and kindly asking for an alternative to get it moving.

I'm not complaining nor anything, yet I must say I feel more and more like Asterix in his 12 tasks, in the part when he has to navigate a kafkaian administration ;-)

[pfmoore]

I sympathise. Maybe the best approach would be to ask on the original thread for this PEP for someone who can act as a sponsor. That's the most likely place to find someone who is a core dev, who is also interested in the proposal.

[bmartinn]

Hey @sbidoul, I'm not sure if it helps expedite the process, but you have my vote as well :)

[sbidoul]

@bmartinn thanks for the support.

There were a few comments on PEP 610 since it's up for review (thanks @uranusjr and @chrahunt), none of which raising blocking issue on the concept. However the PEP has not raised interest from python core devs, so that path seems to be closed for now.

That said I'm still very much motivated to implement this pip freeze improvement.

@pypa/pip-committers, would you see an objection if I make a PR that solves this issue, by implementing PEP 610, except it would name the file pip_direct_url.json and make it clear the file is reserved for pip private use? I don't think there is anything that prevents tools to add their own files into .dist-info directories.

This could serve as a real world implementation for the PEP, to discover and weed out any issue in it.

We would not document the file format and we'd make it clear that the file is not intended to be interoperable, and that if other tools want to use it people need to go and review PEP 610. This way we are not growing the pip public surface area, we are just making pip freeze work the way it should in more cases.

If and when the PEP get approved, it will be easy to introduce a transition path.

[bmartinn]

@sbidoul for what it's worth, I think that's a good compromise. It will definitely expand the transparency of the installed packages on the system, with not a lot strings attached (in terms of committing to a strict standard).

[sbidoul]

@bmartinn the PEP got a new sponsor, so thing are moving again. I'm currently writing a new version of the PEP that integrates all the review comments.

[sbidoul]

A proposed implementation is in #7612. People interested in testing this can do it by installing pip from that branch: python -m pip install -U git+https://github.com/pypa/pip@refs/pull/7612/head.

[bmartinn]

Thanks @sbidoul ! I'll check the PR

[josephcappadona]

A proposed implementation is in #7612. People interested in testing this can do it by installing pip from that branch: python -m pip install -U git+https://github.com/pypa/pip@refs/pull/7612/head.

Just wanted to chime in saying that this worked for me!

[thomasgilgenast]

A proposed implementation is in #7612. People interested in testing this can do it by installing pip from that branch: python -m pip install -U git+https://github.com/pypa/pip@refs/pull/7612/head.

Also chiming in to say I've used this PR to create "nested" dependencies (i.e., packageA depends on packageB depends on packageC, etc.) by using PEP 440 direct references to git repositories in setup.py's setup_requires in such a way that virtual environments survive a "round trip" of pip install packageA -> pip freeze > requirements.txt -> pip install -r requirements.txt without any manual editing of the requirements.txt file. I don't think this was possible to do with pip+setup.py at all before this PR, because the repo URL was only written to requirements.txt if the package was installed in editable mode, and I don't think there's a way to indicate that a dependency should be installed in editable mode in setup.py. With this PR, everything "just works" as a naive user might expect.

In short, this PR (and the Herculean effort being put in to create PEP 610 and address this issue) helps users like me who are still running into the pre-PEP-517/518 variant of this issue too.

[sbidoul]

@pradyunsg is this feature worth calling out in the release notes?

[pradyunsg]

Wouldn't the entry for #609 cover this?

[pradyunsg]

We have now (per #7951) published a beta release of pip, pip 20.1b1. This release includes #7612, which implemented a solution for this issue.

I hope participants in this issue will help us by testing the beta and checking for new bugs. We'd like to identify and iron out any potential issues before the main 20.1 release on Tuesday.

I also welcome positive feedback along the lines of "yay, it works better now!" as well, since the issue tracker is usually full of "issues". :)

[bmartinn]

@pradyunsg It works!!! It's so awesome!

Minor caveat though :)

If you do

sudo pip install git+https://github.com/...

Then when you run pip freeze (notice not sudo)
You get the following exception:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/commands/freeze.py", line 98, in run
    for line in freeze(**freeze_kwargs):
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 68, in freeze
    req = FrozenRequirement.from_dist(dist)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 256, in from_dist
    direct_url = dist_get_direct_url(dist)
  File "/usr/local/lib/python3.6/site-packages/pip/_internal/utils/direct_url_helpers.py", line 118, in dist_get_direct_url
    return DirectUrl.from_json(dist.get_metadata(DIRECT_URL_METADATA_NAME))
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1420, in get_metadata
    value = self._get(path)
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1616, in _get
    with open(path, 'rb') as stream:
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/direct_url.json'

As you can see the direct_url.json file is missing the r flag for group / others , which raises the exception.

-rw------- 1 root root 135 Apr 25 23:22 /usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/direct_url.json

I tested running chmod 644 on this file, and after that pip freeze works like a charm.

Other than this minor issue, great job guys! :+1:

[pradyunsg]

@bmartinn That's expected. I think that's a permissions situation with your system. Any files created by a sudo'ed command would be owned by root, and it might be the case that these files aren't readable. To verify this -- check if anything in /usr/local/lib/python3.6/site-packages/trains-0.14.1.dist-info/ is readable by group/others. :)

Other than that, in general, we strongly recommend users to not use sudo pip, to avoid conflicting with the system package manager (see https://stackoverflow.com/questions/21055859/what-are-the-risks-of-running-sudo-pip/22517157#22517157).

[bmartinn]

@pradyunsg I verified, the top_level.txt file next to the direct_url.json file has the correct permission...
I also opened an issue, so it doesn't get lost #8139

Other than that, in general, we strongly recommend users to not use sudo pip, to avoid conflicting with the system package manager (see https://stackoverflow.com/questions/21055859/what-are-the-risks-of-running-sudo-pip/22517157#22517157).

Definitely agree on this one, but you asked to test the new feature, so I did :)

[pradyunsg]

Ah yes, I was wrong about why this is happening. :)

Thanks for filing that separate issue @bmartinn and your report is much appreciated! ^>^