pip does not reinstall an already installed package from a different direct url requirement

[sbidoul]

What's the problem this feature will solve?

Assume packaging has been installed with pip install "packaging @ git+https://github.com/pypa/[email protected]".

Then try to install from the 20.1 tag: pip install "packaging @ git+https://github.com/pypa/[email protected]". The second install does nothing, saying the requirement is already satisfied.

Describe the solution you'd like

The second install should reinstall with the requested requirement.

Alternative Solutions

• Document this as a limitation of direct URL references support in PIP.

Additional context

This requires PEP 610 to implement. This may or may not influence the ongoing reflection about the resolver.

[uranusjr]

Does the old #egg style URL correctly upgrade? I’d think the two should behave the same (i.e. if the old style upgrades, this should as well; if it doesn’t, we can treat this as a limitation).

[sbidoul]

Does the old #egg style URL correctly upgrade?

No, it does not.

Note a direct reference without package name does upgrade (unconditionally reinstalls? - I've not checked yet). I think in this case it should first get metadata to find out the name and then behave the same as when the name is provided upfront).

I think all 3 cases (without name, #egg and PEP 440 name-in-front) should behave the same wrt reinstalling or not.

In editable mode, it always upgrades (which makes sense).

[uranusjr]

Completely agree that the three cases should behave the same. I wonder why nobody really noticed the nameless and #egg cases have different behaviour (or maybe there is, just no maintainers care).

[uranusjr]

3212 looks similar (but instead of deliberately wrong name this provides no name at all). #6457 also observe the non-upgrades when #egg= is specified.

[pfmoore]

Agreed all 3 should work the same. But given that pip install packaging would need -U to upgrade if packaging 20.0 was installed, but 20.1 was available on the index, I could argue that -U should be needed here. Does the upgrade happen if -U is specified?

Note: I understand that the user's intention is clearly different in this case - the explicit specification of name + version strongly suggests that the user wants 20.1 whether or not 20.0 is present. But I'd like to better understand how we would cleanly describe the semantics here - it feels like we're getting dangerously close to wanting pip to just "do what I mean" which is notoriously hard to get right, and even harder to document (people don't always mean the same thing, even when the same command is specified!).

[pfmoore]

https://github.com/pypa/pip/issues/536 has an extensive discussion over whether we should force reinstall explicitly specified locations. Apparently I was ambivalent when it was discussed then, as well 🙂

[sbidoul]

Interesting discussion in #536. I'm not sure why it is closed, though.

Was it the intent to reinstall always (which is the case today for nameless direct requirements)?

[pfmoore]

Yeah, I wasn't sure either. The discussion looks inconclusive to me, and the linked PR that supposedly closed it doesn't seem to cover the same topic.

I think the problem is that there wasn't consensus on an agreed intent.

Personally, I prefer simpler mental models, so my preference is to say that if you want to upgrade, you should just add --upgrade or --force-reinstall. But I can see the arguments that the user "obviously" means to do an upgrade, so we should do that.

[sbidoul]

With PEP 610 direct_url.json we can reinstall only if the URL has changed. That covers common cases of installing from a VCS tag or commit id, or an archive with a version number in its name. When installing from a VCS branch or local directory, the user can use --force-reinstall if needed.

[sbidoul]

I seems using --upgrade makes pip always reinstall direct URLs, which is a reasonable workaround.

The default behaviour of not upgrading is unintuitive in most cases, however.

So maybe we can approach this in two steps:

1. Make it work identically for nameless and named URLs. We need to decide which is the best:

   • reinstall always (as pip does for nameless direct URL requirements now)
   • reinstall when --upgrade is requested (as pip does for named direct URL requirements now)

2. In a second step, consider a smart approach of reinstalling

   • when --upgrade is provided
   • or when the direct URL changes compared to the installed one (that we can discover in direct_url.json following #7612)

[sbidoul]

PR #6402 trying to address #5780 also touches this subject.

[pfmoore]

Note that the new resolver will affect behaviour here. Currently, requesting an install of a URL or file (named or nameless) will basically do that install (subject to the question here about upgrades). With a real resolver, there's the possibility that installing will cause a conflict, and so will get rejected for that reason.

So the user expectation will need to be a little more nuanced that "reinstall always" in any case under the new resolver.

I'm not sure if this affects the discussion here, but I wanted to point it out because it's a subtle but important UX difference with the new resolver that will likely need careful managing as part of the resolver rollout.

[sbidoul]

@pfmoore my impression (trying to factor out my own use cases) is that the common use case for direct URLs is very similar to pinning a version. For instance when pointing to a VCS tag, a VCS commit id, an archive with a version number in it, or a local directory, the user says "I want this specific version and not any other one".

So this could mean direct URL requirements could be handled the same as == version specifiers wrt conflict handling.

[pfmoore]

Yes, that's how I'd expect the new resolver to handle it.

[pradyunsg]

Yes, that's how I'd expect the new resolver to handle it.

We'd want to check this out. :)

[pradyunsg]

Hey, is this basically a part of #5780 now?

[uranusjr]

I’ll go ahead and close this in favour of #5780. Kindly raise your voice if you feel it is a mistake.