Pip: local development and pip's 2020 resolver

[gaborbernat]

Without the resolver pip always reinstalled source distributions that it got as a direct command-line argument. Such is the case of tox, that builds the sdist, and then invokes pip to install it. In a pip with resolver world now, pip seems to no longer do this but rather first check the version number to see if it's not already installed:

  Getting requirements to build wheel ... done
    Created temporary directory: /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j
    Running command /tmp/magic/.tox/test/bin/python /tmp/magic/.tox/test/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/tmpqp55wn5c
    running dist_info
    creating /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info
    writing /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/PKG-INFO
    writing dependency_links to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/dependency_links.txt
    writing entry points to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/entry_points.txt
    writing requirements to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/requires.txt
    writing top-level names to /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/top_level.txt
    writing manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    reading manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    writing manifest file '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.egg-info/SOURCES.txt'
    creating '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-modern-metadata-v95a7j6j/magic.dist-info'
    Preparing wheel metadata ... done
  Source in /private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-build-h298ahlt has version 1.4.1, which satisfies requirement magic==1.4.1 from file:///tmp/magic/.tox/.tmp/package/1/magic-1.4.1.tar.gz
  Removed magic==1.4.1 from file:///tmp/magic/.tox/.tmp/package/1/magic-1.4.1.tar.gz from build tracker '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-tracker-5awzmelt'
Removed build tracker: '/private/var/folders/kt/btg285ds5kx4l1k398lb7df80000gr/T/pip-req-tracker-5awzmelt'

For local development projects, the version number though is highly untrustworthy, and the records/their contents are different even if the version number is the same. You generally only change versions at releases. We should either always re-install sdists/wheels passed in as paths or ensure that the content of the installed and passed are byte identical...

PS. tox/users could pass --force-reinstall to force the operation I guess, but this would be a breaking feature from POV of pip users.

[sbidoul]

This relates to #5780 and the upgrade strategies for direct URLs.

[uranusjr]

Agreed. More specifically, the old resolver uses different logic for PEP 508 and legacy requirements on whether it should re-install a URL requirement. The new one tries to be consistent, but that consistent logic needs refining. I think the conclusion in #5780 is to

• Always re-install if the URL changes (possible with PEP 610)
• Use the existing installation if the URL does not change, and the package version matches
• Re-install if the package version changes, even if the URL is the same

I’m not sure what parts of the URL PEP 610 records; if it contains the query string part, tools like tox should be able to force package re-installation with general cache-busting techniques.

[sbidoul]

And let's not forget editables, which are always re-installed :)

[gaborbernat]

I’m not sure what parts of the URL PEP 610 records; if it contains the query string part, tools like tox should be able to force package re-installation with general cache-busting techniques.

Can you detail how this would look?

Use the existing installation if the URL does not change, and the package version matches

Note this use falls under this section which is not a solution for this issue and users will be surprised by this change in behaviour.

[uranusjr]

The most widely-used generic cache busting approach simply appends a hash to the query string, so you’d do something like

pip install "package_name @ file://path/to/file.whl?tox=1234abcd"

where the tox= value is machine generated and changes every time (e.g. build timestamp, hash, or UUID). This would make the URL change, without actually changing the thing it points to.

[gaborbernat]

That sounds something a tool could do (though all tooling would need to change how to install packages), but not that useful for humans 🤔

[uranusjr]

My understanding is that pip doesn’t (never did?) position itself as a development tool, and is expected to be used in conjuction of other tooling to work well in such settings.

---

pip’s current behaviour (always re-installs bare URLs, but never re-installs PEP 508 URL specs) is entirely arbitrary without justification. My understanding is that the situation is caused by historic oversight, and should be classified as a bug. But it is also not obvious which behaviour is “corerct”, and there are two camps of people wanting the opposite approach. The resolution in #5780 is most reasonable middle ground (which actually makes sense!) that I know. I am fully aware that it would break someone’s workflow no matter what we do, but I still believe it’s something that should be done.

[gaborbernat]

How would maintaining status quo break peoples workflow? 😊

[pradyunsg]

always re-installs bare URLs

I've not read this issue yet, but this behaviour is definitely intentional: https://github.com/pypa/pip/pull/4764 + https://github.com/pypa/pip/issues/536

never re-installs PEP 508 URL specs

:shrug:

[gregcouch]

This a big change for local development. During local development, we build the wheel, then pip install --upgrade it. The version number does not change during the local development cycle. Adding --use-feature=2020-resolver prevents the "upgrade" from happening. --force-reinstall is overkill because it reinstalls all of the dependencies (and actually doesn't work when some of the dependencies are not (yet) on pypi). --no-deps only works if all of the dependencies have already been installed, ie., don't change during development.

If --force-reinstall could be limited to the packages on the command line, which is what I would expect, then that would be a reasonable solution.

[uranusjr]

If --force-reinstall could be limited to the packages on the command line, which is what I would expect, then that would be a reasonable solution.

This is currently possible with --force-reinstall --no-deps (with the caveat that it also won’t fix your dependencies if they need upgrading). Maybe we should have something like --upgrade-strategy for this.

[pradyunsg]

I think local directories/files passed directly via the CLI (like pip install . or "pip install file) are a special case where we should unconditionally reinstall those.

I do agree that all other URL and package @ URL should be treated consistently - the fact that they differ was a quality of implementation issue.

[uranusjr]

I believe both the old and new resolvers do directory and VCS installations consistently. The inconsistent part is when they’re passed a URL to sdist or wheel, so #4764 and #536 don’t apply.

[uranusjr]

I thought about this a bit more; maybe we should just always reinstall a URL, no matter if it’s PEP 508, no matter whether it’s editable, and no matter whether it points to a directory, sdist, wheel, or VCS. The not-reinstall-URL thing is an optimisation anyway, and by always reinstalling a URL-specified requirements, we also give the user looking for optimisations use indexes instead, with which pip can offer even more optimisations than reinstallation.

[sbidoul]

always reinstall a URL

That can't work, it will raise a load of (performance) issues for people who actually need them.
I for one spent time, e.g. implementing caching of immutable VCS URLs for a reason.

Asking people to use to indexes won't make it either. One reason is that using indexes requires generating unique version numbers which is a burden when, e.g. you want to install an unmerged upstream PR for some dependency.

Always reinstalling local directories sounds better to me.

Perhaps always reinstalling local files too, or not.

A more targeted --force-reinstall variant would be interesting to explore too.

What is actually the use case for installing from a wheel or sdist in a development workflow?
Would there be an issue with "pip install localdir" (which builds a wheel anyway before installing)?

[gregcouch]

An issue with "pip install localdir" is how to pass arguments to setup.py (e.g., --no-user-cfg, --debug). It is much simpler to keep the building and the installing of the wheels separate.

[brainwane]

@uranusjr said:

I am torn on 8711; I want to do something about it before the resolver becomes the default, but that is a very involved issue that may delay the schedule a lot.

But while speaking with @pradyunsg yesterday I learned that, according to Pradyun, this issue is resolved. Pradyun, could you clarify?

[uranusjr]

Oops, I messed up the issue number, sorry. The complicated issue that may delay the rollout I had in mind is #8785, not this one.

[uranusjr]

I don’t think this one is resolved yet though? As mentioned above, this is related to #5780, and we need to make a decision on that one. Although we probably can close this one and only track #5780 (which contains a more concrete plan).

[pradyunsg]

❯ pip install pip==20.3b1                                               
Requirement already satisfied: pip==20.3b1 in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (20.3b1)
❯ pip install ../furo/dist/furo-2020.11.19b18-py3-none-any.whl --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
Installing collected packages: furo
Successfully installed furo-2020.11.19b18
❯ pip install ../furo/dist/furo-2020.11.19b18-py3-none-any.whl --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
❯ # that's not right.
❯ pip install ../sphinx --no-deps                                       
Processing /Users/pradyunsg/Projects/sphinx
Building wheels for collected packages: Sphinx
  Building wheel for Sphinx (setup.py) ... done
  Created wheel for Sphinx: filename=Sphinx-4.0.0.dev20201119-py3-none-any.whl size=2849507 sha256=e302a9ab8508612bd9d36bb5e1ea8e80437688348988a42047bf6f103e903d24
  Stored in directory: /private/var/folders/4d/bt0_xfx56bjfmmt2bv3r5_qh0000gn/T/pip-ephem-wheel-cache-g537cs6r/wheels/4c/be/2f/32e2f55209bb9cdf33815ad4c6bdffb5d6a14c950f1f4c3127
Successfully built Sphinx
Installing collected packages: Sphinx
Successfully installed Sphinx-4.0.0.dev20201119
❯ pip install ../sphinx --no-deps
Processing /Users/pradyunsg/Projects/sphinx
❯ # that's not right either

I think this issue does indeed boil down to #536. I think I have a straightforward patch for this -- always reinstall file:// URLs. :)

[pfmoore]

❯ pip install ../furo/dist/furo-2020.11.19b18-py3-none-any.whl --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
Installing collected packages: furo
Successfully installed furo-2020.11.19b18
❯ pip install ../furo/dist/furo-2020.11.19b18-py3-none-any.whl --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
❯ # that's not right.

I really don't see why you consider this as not right. You're installing the version that's already present. This issue is specifically talking about source distributions, which are more difficult because the practicalities of how people develop means we can't assume same version => same code in that case.

At best, I'd see reinstalling wheels as a bug that's "not worth fixing" if we are force-reinstalling source distributions.

[pradyunsg]

It's the same behavior with sdists as well. Overall, I think we're basically regressing on #536, which would be... not-great. :)

❯ pip uninstall furo                                          
Found existing installation: furo 2020.11.19b18
Uninstalling furo-2020.11.19b18:
  Would remove:
    /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages/furo-2020.11.19b18.dist-info/*
    /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages/furo/*
Proceed (y/n)? y
  Successfully uninstalled furo-2020.11.19b18
❯ pip install ../furo/dist/furo-2020.11.19b18.tar.gz --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: furo
  Building wheel for furo (PEP 517) ... done
  Created wheel for furo: filename=furo-2020.11.19b18-py3-none-any.whl size=84285 sha256=e7a92a5dd085b0aa80203ff34317548aeee7ba764c3f0237a3b8fb22b6757f45
  Stored in directory: /Users/pradyunsg/Library/Caches/pip/wheels/15/8f/3b/17b8285467d65b36fe3792aa5acf5036498d031a9ac3ce7237
Successfully built furo
Installing collected packages: furo
Successfully installed furo-2020.11.19b18
❯ pip install ../furo/dist/furo-2020.11.19b18.tar.gz --no-deps
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
❯

[pradyunsg]

(all these runs are with master _and_ 20.3.b1 -- same output)

[pradyunsg]

To reiterate what I think our behaviour here should be (and is with the legacy resolver):

I think local directories/files passed directly via the CLI (like pip install . or "pip install file) are a special case where we should unconditionally reinstall those.

[pfmoore]

Yeah, I get what you're saying, I just (still) disagree. I think that the special case should be local source code, and we shouldn't be special-casing wheels.

I've seen no-one claim that they need reinstall behaviour for wheels.

Longer-term, I think we should have a proper look at what constitutes pip's idea of the "same thing" when it comes to name/version etc. At the moment, we're making somewhat adhoc decisions based on individual use cases. Doing what people want is good! But at the same time, we need a proper model of the objects pip is working with, otherwise we'll end up with inconsistencies, and things get messy.

(As an example, pip freeze can't see the distinction between installing foo-1.0 from an index, and forcing an explicit install from a local wheel. So by making pip install ./foo-1.0-xxx.whl do something different, we may be breaking pip freeze. I don't think this is a significant problem, nor do I think it's something we should be worrying about, but it is an example of how not having a consistent model affects unrelated parts of the code).

[pfmoore]

I should say, I'm not going to make a fight of this. I've made my opinion known, and I'm fine with being overruled.

[pradyunsg]

ACK. I hear what you're saying and, I'm not in a "let's overrule without hearing/understanding" mode. :)

Accommodating for what you're asking is a matter of adding and not candidate.link.is_wheel, so I'm not concerned about implementation complexity either.

I'm only wondering whether it's more important to be able to say "any local file/path gets reinstalled when specified directly" or to be able to say "all wheels with the same version are equivalent".

The former is basically status quo with the old resolver, so I'm not too concerned about the nuances (it's as broken/feature full as before). The latter makes me want to add special cases for prerelease versions and, it just feels easier to stick with "reinstall local stuff" instead of local wheels are handled in a different way from local sdists/paths. Notably, if we do reinstall equivalent wheels, we're only making a self-nullifying change, which I'm A-OK with. :)

[pfmoore]

It's not a problem - I think that in the end it's very much a matter of opinion, and so there isn't much to understand here.

I didn't realise that the old resolver would reinstall wheels, and I've never seen anyone asking for that behaviour to be reinstated. We spend a lot of time pointing out to people "stuff with the same name and version should be equivalent", so I'm reluctant to confuse that message by reinstalling wheels. I'm fine with making an exception for source trees, because they are intended to be under development, and I'm OK with making the exception include sdists because the difference between sdists and source trees has never been that clear to people.

I would say that if we do reinstall wheels, we shouldn't add a test for that, as I'd like to leave the option open to stop doing so in the future (on the basis that it's an optimisation, if nothing else). I definitely don't think we should make "we'll reinstall equivalent wheels" something that we should guarantee (even under restricted circumstances).

As to your point, I don't think it's much more difficult to say "any local source file/path gets reinstalled when specified directly", so I'm not too bothered about explaining the behaviour.

[uranusjr]

I really don’t think this is a good idea. IMO URL/path to the same wheel (and sdist to some extent) should be treated the same no matter where they came from. If they are in fact not (for whatever reason, e.g. the version is not trustworthy during development, as described in the top post), the user should tell the installer to do otherwise. The old resolver had a bug (always re-installs bare-URLs), that allows users to work around this responsibility that is not present in the new resolver, and the proposed solution is basically saying we should introduce that bug back. Yes, if a bug exists long enough it becomes a feature, but that does not mean it should be supported until eternity.

At the very least, can we put up a deprecation warning telling people relying this they should fix their usage (e.g. pip install --force-reinstall --no-deps path/to/file) since the behaviour will eventually change?

[pfmoore]

Thinking about it, I agree with @uranusjr. I have been assuming that there is a good justification for the claim being made here that people need the old behaviour, but I'm not sure I've seen such justification clearly stated with the trade-offs explained. The original issue here includes the comment

PS. tox/users could pass --force-reinstall to force the operation I guess, but this would be a breaking feature from POV of pip users.

I'm perfectly fine with this being a breaking change - and specifically, I think it's entirely fine if it's a breaking change introduced with the new resolver (it's certainly not the only one). I don't think it's unreasonable for tox to add --force-reinstall, for example, so this change won't impact tox users (other than requiring them to be on the latest version, but again, I don't see a problem with that).

The comment here makes a stronger argument that this will impact people's workflows, but as usual it's extremely hard to gauge how widespread the impact will be, and IMO breaking a few more¹ workflows now so that we have a clearer, more understandable model for the future is a worthwhile trade-off.

¹ We're already breaking a bunch (e.g., some users of constraints) with the new resolver - we know that and have accepted it and done what we can to mitigate it.

[uranusjr]

I want to also note that I would not have an issue if a path/URL to something other than an sdist/wheel is always re-installed, since contents in e.g. local directory or VCS repositories are more obviously unreliable.

The impacted build wheel + install workflow described above can achieve the desired behaviour by attaching local version segments to the wheel (using e.g. the current timestamp), which I feel would also likely benefit the setup in other ways.

[pradyunsg]

As it stands, we're processing such packages and exiting without any output on why no packages were modified. That looks like "clearly something is wrong" to me.

The old resolver had a bug (always re-installs bare-URLs)

Huh. I was assuming that folks had context of #536 by now but this makes me think otherwise.

This exact behaviour was a long standing user request, and was intentionally added. It's not a bug, but a feature that was implemented and had a significant chunk of code written to make it possible.

---

I'm strongly opposed to completely regressing our behaviour on this front (i.e. deviating from #536 wrt directories) without someone making a strong case for why we shouldn't be reinstalling (rather than why we should, which was extensively discussed and iterated on in #536 and the various PRs).

The behaviour exists because it's useful in local development whereas the alternative is a lot of work. There are users are directly depending on this. It was specifically added upon request. Having a breakage here for a potential future cleanup where it might cause a bit more work - isn't a strong reason IMO.

[pradyunsg]

To be clear - my mental model is that if I'm directly specifying a wheel/sdist (rather that coming through a local index), it's because I want the contents of exactly that file in my environment (possibly because I've made changes to it somehow - development, local patching whatnot). What I would find surprising is that it doesn't work because the already installed thing matches the version number from the metadata of the package.

This is a takeaway from both having hit this myself many times to the point where I worked toward ensure we resolve #536.

[pfmoore]

I think we're circling round the same confusion again. I'll say "need" here to avoid implying we "must" do certain things. The discussion we're having is over whether we should reinstall when we don't "need" to.

• Wheels - definitely shouldn't need reinstalling
• Source distributions - shouldn't need reinstalling (but see below)
• Source trees - reinstalling is reasonable, and matches people's workflows.

I say that source distributions shouldn't need reinstalling, specifically because they are build artifacts, and so should be uniquely identified by name/version. That's the case when they come from an index such as PyPI, and we already have a strong principle that "where a distribution file comes from" shouldn't affect the result of an install.

The big problem here is that we're conflating pip's behaviour for source distributions and source trees, and this is where I think we're getting in a mess. I think we should be careful to distinguish the two cases, and at the moment we're not (at least in part because we can't tell if an arbitrary .tar.gz file is a sdist, or "just" a bundled up source tree - but also because the "source tree" concept itself never really caught on as an explicit idea).

To be clear - my mental model is that if I'm directly specifying a wheel/sdist (rather that coming through a local index), it's because I want the contents of exactly that file in my environment (possibly because I've made changes to it somehow - development, local patching whatnot). What I would find surprising is that it doesn't work because the already installed thing matches the version number from the metadata of the package.

Again, that's a model issue. We currently don't clearly distinguish (in our documentation or design) between "stuff that comes from an index (handwave about --find-links here)" and "stuff that comes from a file/directory". Maybe it's time that we do make that distinction explicit.

So ultimately, I guess what I'm saying is that this question has exposed some fairly fundamental design issues that we need to resolve. And of course, that will take time (potentially a lot of time, thanks to the usual resource issues).

My feeling is that a stopgap here would be:

1. Cautiously reinstate some of the old resolver's behaviour, with strong caveats that it's subject to revision.
2. Don't reinstall wheels - they are a clear case where we know they shouldn't be changing without a version change, and --force-reinstall should be a sufficient workaround.
3. Reinstall local directories, because we know they are "development areas".
4. Dither a bit more over explicit source files 🙂 I'd rather not, but I think we should probably go back to the old resolver behaviour for now at least, but I'd be sympathetic to warning if people don't explicitly use --force-reinstall.

But I very much don't want to lock ourselves into guaranteeing a particular approach here. (Which is why I objected to having a test that ensured we always reinstalled explicitly specified wheels).

[uranusjr]

The original request in #536 wants pip to always reinstall a directory, which I do not have problems with, as clarified above. But what eventually got implemented in the old resolver (and the PR proposed to the new resolver) also reinstalls wheels and sdists, which is the inconsisteny part.

BTW another possibility to consistency to this would be to always reinstall all direct URLs, which I also have no problems with, but that doesn’t seem to be favoured by most people.

[pradyunsg]

A different thought: let's reinstate the old behaviour here (for source trees, sdist and wheels) but add a message saying "hey, tell us here if you see this and talk to us about it" for the sdist and wheel cases.

In other words, treat this as a "we're making a change" as we usually do, rather than as a "we're changing things because the resolver changed and this affects dependency resolution and we have a bunch of communication set up about the change". Because, well, we don't have a bunch of communication written down about the change and this doesn't affect resolution but the post-resolution steps. :)

If we agree on this, I'll file a new issue as the feedback channel with the targeted goal of "we want to know if folks depend on the always-reinstall-local-sdist-and-wheels".

[pradyunsg]

that doesn’t seem to be favoured by most people.

Mostly because it has a significant network and build cost, and can be guaranteed to be stable in some sense by the hashes and whatnot.

[uranusjr]

add a message saying "hey, tell us here if you see this and talk to us about it" for the sdist and wheel cases.

I’m quite certain almost all respondents to that would encourage the current behaviour, because everyone wanting pip to not reinstall would be doing something else to avoid the command altogether right now…

[pfmoore]

A different thought: let's reinstate the old behaviour here (for source trees, sdist and wheels) but add a message saying "hey, tell us here if you see this and talk to us about it" for the sdist and wheel cases.

But people who would be fine with not reinstalling probably don't care about the whole discussion (reinstalling is basically a minor performance hit to them). So they'll (a) be annoyed by the message and (b) won't respond because they feel they don't have anything to say.

What we'd end up with is therefore likely to be a self-selected group of responders who care a lot about retaining the reinstall behaviour, which will skew our information even further towards having to maintain it forever.

We shouldn't be trying to please everyone here. Rather, we should be delivering a consistent, understandable, and flexible tool, and having confidence that it does its job well. Limiting scope and deciding not to support certain use cases is a hard thing to do, but it's fundamental to our job as maintainers IMO.

[pfmoore]

I should say, I'm not going to make a fight of this. I've made my opinion known, and I'm fine with being overruled.

It turns out that the more we discuss this, the more convinced I'm becoming that we shouldn't just blindly reinstate the old resolver behaviour 🙂

[pfmoore]

everyone wanting pip to not reinstall

In practice, I suspect a significant majority simply don't care either way. And that's the problem. Whichever way we go, a small minority will be annoyed. Which is why I prefer to look at clean design and long-term sustainability over just appeasing "whoever shouts loudest".

/me ducks as the UX people all throw things at me 🙂

[pradyunsg]

Okay, so what behaviors do we want to do here?

• Local directories: reinstall.
• Local sdists: ???
• Local wheels: ???

The old resolver (AKA status quo) behaviour is:

• Local directories: reinstall.
• Local sdists: reinstall.
• Local wheels: reinstall.

My proposal above would basically looked like:

• Local directories: reinstall.
• Local sdists: reinstall, but warn that we'll change it + ask for feedback.
• Local wheels: reinstall, but warn that we'll change it + ask for feedback.

Could someone phrase @pfmoore's earlier comment (https://github.com/pypa/pip/issues/8711#issuecomment-731123513) in these terms, as how to behave for each of these cases?

---

As an aside: Some part of me prefers the "reinstall all local things" because it's easier to implement and explain (at least IMO). All the other alternatives proposed (till now) are (1) more complex (2) more work 😛 and (3) make breaking changes.

It's bothering me (perhaps more that it should) that something more complex to implement (that is also a breaking change) is being termed as "clean design" / "more consistent" compared to maintaining status quo and avoiding breakages. That might just be me being inflexible in how I'm looking at this though. :)

[pradyunsg]

What we'd end up with is therefore likely to be a self-selected group of responders who care a _lot_ about retaining the reinstall behaviour, which will skew our information even further towards having to maintain it forever.

This is true, but I can't think of anything else so I'mma duck out and flip to "hey, what do you think we should do then?". :P

[sbidoul]

IMO, this closely relates to #5780, as @uranusjr mentioned before. I've long wanted to finish the discussion there, and unfortunately could not find the time to do the necessary research.

This definitely does not fit in my brain without external memory. What I want to do to help form my opinion is populate a table like this (which I post here in a very rough state in case anyone finds it useful).

The only thing I can say so far, is that I have the intuition that file:// and https:// URLs to archives and wheels should behave the same.

[uranusjr]

The old resolver (AKA status quo) behaviour is:

• Local directories: reinstall.
• Local sdists: reinstall.
• Local wheels: reinstall.

The old resolver behaves differently depending on whether you use the PEP 508 form or pass a path/URL directly. Which is obviously a bug (and the PEP 508 behaviour is very obviously wrong), but who knows if somebody is depending on that…

[pradyunsg]

@sbidoul That's really useful! Thanks for sharing!

5780

• same project.
• same version.
• remote URL (possibly VCS).
• URL changed.

8711

• same project.
• same version.
• _any_ local file/path.

To me... these are fairly different issues. :)

(editted out misformatted comment)

[pradyunsg]

The old resolver behaves differently depending on whether you use the PEP 508 form or pass a path/URL directly. Which is obviously a bug

That's true.

❯ sdist=/Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
❯ wheel=/Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
❯ pip install furo==2020.11.19b18
❯ pip install --no-deps --use-deprecated=legacy-resolver $wheel
Requirement already satisfied: furo==2020.11.19b18 from file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (2020.11.19b18)
❯ pip install --no-deps --use-deprecated=legacy-resolver "furo @ file://$wheel"
Requirement already satisfied: furo@ file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl from file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (2020.11.19b18)
❯ pip install --no-deps --use-deprecated=legacy-resolver $sdist
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: furo
  Building wheel for furo (PEP 517) ... done
  Created wheel for furo: filename=furo-2020.11.19b18-py3-none-any.whl size=84285 sha256=e7a92a5dd085b0aa80203ff34317548aeee7ba764c3f0237a3b8fb22b6757f45
  Stored in directory: /Users/pradyunsg/Library/Caches/pip/wheels/15/8f/3b/17b8285467d65b36fe3792aa5acf5036498d031a9ac3ce7237
Successfully built furo
Installing collected packages: furo
  Attempting uninstall: furo
    Found existing installation: furo 2020.11.19b18
    Uninstalling furo-2020.11.19b18:
      Successfully uninstalled furo-2020.11.19b18
Successfully installed furo-2020.11.19b18
❯ pip install --no-deps --use-deprecated=legacy-resolver "furo @ file://$sdist"
Requirement already satisfied: furo@ file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz from file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (2020.11.19b18)```

The only place the behaviour differs above is with PEP 508 + local sdist URLs. As I mentioned above, I think we should treat the discussion on remote URLs separately, because, well, the code treats them separately today and unifying that is exactly what the discussion in #5780 is about. :)

(and the PEP 508 behaviour is very obviously wrong), but who knows if somebody is depending on _that_…

Well... the new resolver's behaviour is also obviously very wrong in all these cases if you ask me. :)

❯ pip install --no-deps "furo @ file://$sdist"
Collecting furo@ file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
  Using cached furo-2020.11.19b18-py3-none-any.whl
❯ pip install --no-deps $sdist                 
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
❯ pip install --no-deps "furo @ file://$wheel"
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
❯ pip install --no-deps $wheel
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
❯

[pradyunsg]

Here's one more proposal (that I'm updating #9147 for):

• Local dirs: reinstall.
• Local sdists: reinstall.
• Local wheels: don't do anything, print a message nudging toward --force-reinstall.

(btw, yes, I noticed that my original "wheels get reinstalled" statement was not True)

[pfmoore]

Could someone phrase @pfmoore's earlier comment (#8711 (comment)) in these terms, as how to behave for each of these cases?

• Local directories: reinstall.
• Local sdists: reinstall with a deprecation warning, later changing to do not install if same name/version.
• Local wheels: do not install if same name/version.

As I said in my longer discussion, I view "local sdists" as open for some debate, mostly in the sense that I'm open to not bothering with a deprecation but going straight to the "do not install" approach.

Beyond this, I agree that @sbidoul's table is really useful, and exposes a whole load of additional interactions and cases that need considering. But I don't have the mental bandwidth to go through them all right now. This is one of the problems I'm trying to flag - discussions like this always end up considering a multitude of cases, because we've never established any "universal principles" to guide us.

My preferred universal principles are:

1. Whenever 2 sources for a project have the same name and version, pip can choose either, and depending on a particular one being preferred is an error.
2. Closely related to the above, we never prefer any source of packages over any other.

These principles do not allow for "development" setups, where the user is actively editing and installing the source code. But we should address that by first modifying the principles, and implementing the resulting consequences, not by special-casing situations.

It might be worth noting here that the recent UX survey results about "what's your mental model of pip" didn't include a single response, as far as I could see, that described pip as "managing my development sources" or anything using the term "development" or "sources". I don't think we should necessarily treat those results as definitive, but nor should we ignore them 🤷

[xavfernandez]

Local wheels: don't do anything, print a message nudging toward --force-reinstall.

--force-reinstall will impact all dependencies which means that the suggested workaround might have a strong performance penalty.

[pradyunsg]

The alternative is not suggesting anything. :)

[sbidoul]

To me... these are fairly different issues. :)

The main difference is local vs. remote URL it seems ? A file:// URL is not necessarily "local": for instance it could be file server that people use for the same reason as a web server if they find it convenient to serve their URLs. And an https URL could serve files from a local web server. That's why my instinct tells me https:// vs file:// should not trigger different behaviors.

[pfmoore]

@sbidoul While I agree, we don't have any actual way of distinguishing "local" from "remote" (much like we can't distinguish "sdist" from "archived source tree") so we have to make best guesses.

I'd support making the distinction explicit (somehow) but that's a bigger change.

[sbidoul]

we don't have any actual way of distinguishing "local" from "remote"

My point is that I don't see a good reason to do such guesses about local vs remote archives or wheels. Why would we ? Because we expect different performance characteristics ? Or because file:// would imply the user is somehow iterating rapidly during development ? That sounds like assuming a lot based on very little information.

A source tree (i.e. a directory) is different of course.

[pfmoore]

I'm getting confused over the different suggestions, so apologies, but are you saying that we shouldn't reinstall on any sort of archive file, and only reinstall unpacked source trees (local directories)? That's not going to address the tox case that was the original subject of this issue, but I'm not against it...

[sbidoul]

@pfmoore ...or reinstall all sorts, I honestly don't know. All I'm saying so far is that I _think_ the behavior should be same for file:// and https://, probably. To say anything more conclusive I'm afraid I'd need to do the full research on the table I posted... and just like you I don't have the bandwidth to do it correctly these days. Maybe someone has a clearer picture of the whole situation than me, I don't know. In an ideal world we'd need to do that research first, to know the as-is and wanted to-be situation and discuss a proper transition. But I guess there is the pressure to release the new resolver so :shrug: ...

Now, if anyone wants more of my instinct, I'd say @uranusjr's proposal at https://github.com/pypa/pip/issues/5780#issuecomment-678676463 looks promising (basically reinstall only if url changed compared to direct_url.json), but may require a new --force-reinstall-but-not-dependencies option to resolve this issue that tox raised without performance impact.

[gregcouch]

My group needs the reinstall behavior for wheels.

On Thu, Nov 19, 2020, 08:35 Paul Moore [e-mail] wrote:

Yeah, I get what you're saying, I just (still) disagree. I think that the
special case should be local source code, and we shouldn't be
special-casing wheels.

I've seen no-one claim that they need reinstall behaviour for wheels.

Longer-term, I think we should have a proper look at what constitutes
pip's idea of the "same thing" when it comes to name/version etc. At the
moment, we're making somewhat adhoc decisions based on individual use
cases. Doing what people want is good! But at the same time, we need a
proper model of the objects pip is working with, otherwise we'll end up
with inconsistencies, and things get messy.

(As an example, pip freeze can't see the distinction between installing
foo-1.0 from an index, and forcing an explicit install from a local wheel.
So by making pip install ./foo-1.0-xxx.whl do something different, we may
be breaking pip freeze. I don't think this is a significant problem, nor
do I think it's something we should be worrying about, but it is an
example of how not having a consistent model affects unrelated parts of the
code).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/8711#issuecomment-730492613, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVICTP4ZJMM45P2STFWYOLSQVCL3ANCNFSM4PVH2TUQ
.

[pfmoore]

Thanks for the data point @gregcouch - why is --force-reinstall (possibly with --no-deps) not sufficient for you?

[pradyunsg]

A few notes, after good night's sleep and re-reading this issue:

That's why my instinct tells me https:// vs file:// should not trigger different behaviors.

That matches my instinct as well. :)

The distinction to me is that #5780 is asking for a behavior change in how we handle URLs in general, toward making it more consistent. This issue is about making the new resolver behave like the old resolver w.r.t. local directories and files.

In an ideal world we'd need to do that research first, to know the as-is and wanted to-be situation and discuss a proper transition.

YES! Exactly. We could've used this resolver change to also make progress on #5780, but we just don't have the time/dev resource available right now to properly consider all-the-things and make those changes as we probably want to.

I think #9147 gets us mostly where we might want to be, in terms of this issue. The behaviors for local files and directories would be unchanged between the old vs new resolver (as far as I can tell), including the behaviors for wheels, albeit with more informative messages.

I honestly don't know.

I don't either! That's the whole reason behind me falling back to "let's just do what the old resolver does and revisit this when we have more time to think through this, instead of unintentionally breaking certain workflows". The only thing different is that there's a new deprecation, toward changing the sdist reinstall behavior. :)

I guess there is the pressure to release the new resolver

There is, definitely. I start a non-pip full-time role on Monday, which will affect my availability. Plus, we've already had fairly substantial delays for various other reasons, so I'm not too keen on waiting until after we have an extended discussion on this to move forward with the release. :)

Now, if anyone wants more of my instinct, I'd say @uranusjr's proposal at #5780 (comment)

And, I agree with this. I tried implementing this, but it quickly got a little more messy and I realized it was easier to replicate the old resolver's behavior than to deviate from it.

---

My group needs the reinstall behavior for wheels.

Could you elaborate a little further on this? Based on my testing here, even the old resolver did not reinstall wheels if the versions match. Either you are asking for behavior that wasn't in the old resolver or I am missing something. :)

❯ pip install --no-deps --use-deprecated=legacy-resolver $wheel
Requirement already satisfied: furo==2020.11.19b18 from file:///Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl in /Users/pradyunsg/.virtualenvs/pip/lib/python3.8/site-packages (2020.11.19b18)
❯

vs

❯ pip install --no-deps ../furo/dist/furo-2020.11.19b18-py3-none-any.whl
Processing /Users/pradyunsg/Projects/furo/dist/furo-2020.11.19b18-py3-none-any.whl
furo is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

[pradyunsg]

puts on release manager hat

I'd like to get #9147 merged to unblock us on one of the two major blockers for the main 20.3 release. If there are any concerns around how the new resolver's behavior post #9147 deviates from the legacy resolver, please flag them there.

[sbidoul]

I just did a quick test with 20.2.4:

pip install ./packaging-20.4-py2.py3-none-any.whl does not reinstall
pip install "packaging @ file://$PWD/packaging-20.4.tar.gz" does not reinstall
pip install ./packaging-20.4.tar.gz reinstalls

9147 does this:

pip install ./packaging-20.4-py2.py3-none-any.whl does not reinstall
pip install "packaging @ file://$PWD/packaging-20.4.tar.gz" __reinstalls__
pip install ./packaging-20.4.tar.gz reinstalls

So that does not _quite_ reproduce what the old resolver did.

It looks like the behaviour of 20.2.4 is kind of a side effect of pip needing to build to find the package name.

[pradyunsg]

We don't need bug for bug compatibility IMO - just something that's not _obviously borked_. 😅

I'm inclined to say that the behaviour of PEP 508 vs non-PEP 508 situations to be a bit of a bug-that-is-difficult-to-recreate, and making it consistent on that front right now is A-OK.

[sbidoul]

bug for bug for compatibility

Sure, but _what is the bug_ in the first place ? That's not obvious to me. Perhaps the bug is that it did reinstall at all.

[gregcouch]

As you know the problem with --force-install is that it reinstalls all of
the dependencies. --no-deps is usually okay, but fails when the
dependencies are changed. I want to be able to write the build script and
forget the details. And --force-reinstall --no-deps doesn't do that.

On Sat, Nov 21, 2020, 01:31 Paul Moore [e-mail] wrote:

Thanks for the data point @gregcouch https://github.com/gregcouch - why
is --force-reinstall (possibly with --no-deps) not sufficient for you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pip/issues/8711#issuecomment-731554190, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVICTKANZACJY6IFADFWHTSQ6CIZANCNFSM4PVH2TUQ
.

[bluenote10]

My group needs the reinstall behavior for wheels.

Same in my team, and why I have raised #9116.

Thanks for the data point @gregcouch - why is --force-reinstall (possibly with --no-deps) not sufficient for you?

For our developers the workflow is to re-run pip install -e whenever they have made "install related" changes to a local package, which in our case typically include:

• modifications to entry points
• changes to Cython modules which require a re-compile

Think of it: The only reason to re-run pip install -e on a local package is to pick up the change. I'm currently teaching our team to always use pip install --force -e instead. If a command line option like --force is always required, the default behavior clearly doesn't seem very useful.

EDIT: According to @pfmoore's comment I may have misunderstood the issue description.