Pip-tools: Deprecation policy of supported pip versions

[atugushev]

What's the problem this feature will solve?

Since EOL for Python 2 is coming it's time to discuss about deprecation policy of supported pip versions. Currently, pip-tools supports the following pip versions:

• 8.1.1 (2016-03-17)
• 9.0.1 (2016-11-06)
• 9.0.3 (2018-03-21)
• 10.0.1 (2018-04-19)
• 18.0 (2018-07-22)
• 19.0.3 (2019-02-20)
• 19.1 (2019-04-23)
• 19.2 (2019-07-23)
• 19.3 (2019-10-14)

Fortunately, pip is under active development and refactoring. The problem is each supported pip version increases CI dimensions and version branches in code, which makes hard to maintain pip-tools.

Describe the solution you'd like

Possible solutions are to prepare a deprecation policy based on:

1. calendar, for example:
   
   
   
   • v1 deprecated on date1, will be removed on date 1.1
   
   
   • v2 deprecated on date2, will be removed on date 2.1
   
   
   • etc
   
   
2. support of N latest versions (which is, I think, the current policy except N is undefined)
3. support one minor version at a time (e.g., only pip 19.3.x), suggested by @cjerdonek in https://github.com/jazzband/pip-tools/issues/853#issuecomment-517948771
4. any other suggestions?

Alternative Solutions

Vendor pip.

Additional context

This issue follows up on the discussion started in #853.

[atugushev]

@vphilippon @davidovich I've pinned this and pinged you guys to bring more attention to the important issue.

[AndydeCleyre]

For extra context, I'm linking here pip's changelog, and noting from it that:

• pip 19.0 is the first release that warns about Python 2.7 EOL, deprecates Python 3.4, and supports PEP 517. IMO anything older should be deprecated ASAP.
• pip 19.2 drops support for Python 3.4, which is past EOL now.

Notes on other projects:

Project | Release | pip | EOL
--- | --- | --- | ---
CentOS | 6 | no Python3! | 2020.11.30
Debian | 9 | 9.0.1 | ~2020
CentOS | 7 | 9.0.3 | 2024.06.30
CentOS | 8 | 9.0.3 | 2029.05.31
Debian | 10 | 18.1 | ~2022
Alpine Linux | 3.7 | 18.1 | 2019.11.01
Alpine Linux | 3.8 | 18.1 | 2020.05.01
Alpine Linux | 3.9 | 18.1 | 2020.11.01
Alpine Linux | 3.10 | 19.2.3 | 2021.05.01

Looking at these, I'd support "casually" deprecating <19.0.3 and "aggressively" deprecating <9.0.3.

But I'm an update junkie. I personally wouldn't mind deprecating anything older than Alpine's stable offering, or option 3 in OP.

I don't think option 2 would be sensitive enough to specific pip release fixes or features that this project may find important.

It's disheartening to see these very old packages in the table, and while I don't normally like vendoring, this data along with the nature of pip-tools does make the option more attractive than usual.

[vphilippon]

Thanks @atugushev for opening up the discussion, and thanks @AndydeCleyre for the breakdown of pip version available per linux distro & version.

(Grab onto something, wall-of-text incoming!)

First off, I'd strongly suggest against vendoring pip once again. By doing that, pip-tools takes the ownership and responsibility of which version of pip is used with pip-compile. Right now, it feels as if it was already the case due to the major refactoring happening in the pip project, but that will likely not last forever. At some point, pip version should generally remain compatible pip-tools without intervention on our part (that's what I'm betting on in the long run). Vendoring pip is a long term guaranteed maintainership cost, and while there's currently an active community contributing to pip-tools (once again, thank you all!) there's no guarantee this will be like it forever either (if you want to prove me wrong over time, I'll be happy to admit my misjudgment :D ).

---

Now about the range of supported pip version.
One thing to keep in mind, IMO, is how pip-tools is installed by the end users[1]. Unlike pip, pip-tools is not distributed as a package in the various linux package repo (AFAIK). That means that pip-tools would generally be installed by a simple pip install pip-tools command, or added unceremoniously to some variant of requirements.txt somewhere. What that in turn means is that while the pip version will eventually "freeze" as distro versions age (e.g CentOS 7 using pip 9.0.3), the latest pip-tools will usually be grabbed. Now extend what I just described to having automated scripts that setup build machines and other stuff, on cloud-instantiated servers that come and go on a somewhat regular basis, and those re-grab the latest pip-tools available each time. That leads me to my opinion on the matter: There's a legitimate reason for keeping compatibility with older pip for a longer time than you would in most projects dependencies.

As I was done writing this bit, I was about to suggest keeping support for pip versions used by most common linux distros that haven't reached EOL, so pip 9.0.3 at the moment. I'll keep this paragraph above as thought sharing.

And then I read the table again, just to see how that suggestion held up over time.

Project | Release | pip | EOL
-------- | -------- | ---- | ----
CentOS | 8 | 9.0.3 | 2029.05.31

Oh, hohohoho heck no.
I've been out of the Linux world for a while now. Is it likely for CentOS 8 to have a newer pip version available at some point or not? If yes, then maybe we can keep the idea I mentioned above. If not, I suggest we find another baseline, because this is not reasonable to maintain this long with the changes occurring in pip.

For that other baseline, maybe keeping 1 year worth of pip versions, maybe 6 months, I'm not sure.

[atugushev]

For that other baseline, maybe keeping 1 year worth of pip versions, maybe 6 months, I'm not sure.

I like the idea of supporting only the last minor version. Add pip>=19.3 to the setup.py to dependencies and force users to upgrade pip on install pip-tools. If a user can't afford to upgrade pip, he/she can use old pip-tools versions or use virtualenv (which is the best option, don't even try to upgrade the system's pip by sudo python -mpip --upgrade pip!).

^{On the other hand, it makes a developer's life easier, but not a user's one, which is a poor strategy in most cases of software development.}

[codingjoe]

I second @atugushev comment. I believe should someone prefer to use an old version of pip, they might as well use an old version of pip-tools too. We have barely enough maintainers as it is, we simply can't effort to support all old versions.

I personally don't even consider dropping an old version a breaking change every package explicitly defines its dependencies. Possible dependency conflicts will be displayed by pip.

In any event, I don't see this as a problem for users. They will see the dependency update and update it or not 🤷‍♂

Still, it doesn't change the fact, that we should definitely drop support for all old version. It will significantly reduce overhead for us and propel our other efforts. New features that users will benefit from.

[vphilippon]

All things considered, I think its reasonable to drop support of older pip versions.

As suggested by others before, it would be best to make sure we have at least one version of pip-tools available that support each version of pip.

I would still treat this pip support drop as major/breaking change, at least for this one. We did not specify the pip version so far in the dependencies, and this change in behavior is bound to cause surprise or breakage on the users' side. Let's be as explicit as possible, and it doesn't cost us much to do so. I agree we may want to revisit this approach in the future though, once it becomes an expected thing.

About adding pip directly into pip-tools dependencies (install_requires), I'm not a fan of meddling with the user's pip installation. My main reason is that I still hit issues (EnvironmentError: [Error 5] Access is denied) on Windows if pip ends up trying to install/upgrade itself. Even if a newer pip version was to somehow fix this, remember that people who need to upgrade pip are on an older pip version :).
We can document the behavior in our README and tell windows user to use python -m pip install pip-tools instead, although I do find that to be a pain as a Windows user.
I'm not sure if I would prefer a potentially cryptic failure on updating pip as a dependency, or having pip-tools report a clearer error at runtime. Both options have pros/cons.
Depending on pip always has been an odd case, with no clear good way to deal with it IMHO.
There's a reason pip-tools has a set of UNSAFE_PACKAGES it doesn't pin in pip-compile output by default.
Sorry I can't give a clear opinion on this, feel free to give your own opinion.

[atugushev]

@vphilippon

We can document the behavior in our README and tell windows user to use python -m pip install pip-tools instead, although I do find that to be a pain as a Windows user.

I'm okay with python -m pip install pip-tools. Even official doc suggests installing packages in that way to avoid ambiguity (see https://bugs.python.org/issue22295) and there is also a good article written by %brettsky wich explains clearly why we should use python -m pip ....

All things considered, I think its reasonable to drop support of older pip versions.

So we can release pip-tools==5.0.0 which will support only 20.0.* and 19.3.* pip versions and drop the older ones?

[codingjoe]

Yes please, we should also drop Python to in the same release. A clean cut.

[atugushev]

Do we need a depreciation period? This would help to bring the users' attention and we could get feedback.

[codingjoe]

Do we need a depreciation period? This would help to bring the users' attention and we could get feedback.

Strictly speaking we don't need to do anything. But it's rather uncommon to deprecate dependencies. Deprecation is usually done for Python APIs. What we should do however is to add pip with the correct version specifier to the install_requires section in our setup config. I am baffled that it's actually not there.

[atugushev]

@codingjoe

Strictly speaking we don't need to do anything. But it's rather uncommon to deprecate dependencies.

I'm inclined to agree with you 👍

I am baffled that it's actually not there.

FYI #889.

[vphilippon]

Regarding using python -m ..., you have me convinced. All that is left is my darn resistance to change, and that's a "me" problem 😄 .

I think we can have pip-tools==5.0.0 made, keeping support only to recent pip versions.
For which versions should exactly be supported, I leave that decision to you @atugushev. You know what's best at this point.

---

Regarding Python 2.7 support drop, suggested by @codingjoe, I would rather suggest to follow what pip has set to do: Keep supporting Python 2.7 as long as it remains reasonably doable (https://pip.pypa.io/en/latest/development/release-process/#python-2-support).

My reasons are based on the discussion that lead pip maintainers to take that decision, particularly these post from Donald Stufft:
https://discuss.python.org/t/packaging-and-python-2/662/7
https://discuss.python.org/t/pip-19-0-is-now-available/757/3
Short version: At the moment, they chose to keep new pip releases available on Python 2.7 so other projects that keep supporting Python 2.7 can still use future new packaging mechanics/standards that would require a pip update to use. The idea is to avoid hindering the adoption of new packaging mechanics/standards.

If there's a need for new pip releases to be available on Python 2.7, then I think there's one for pip-tools too, especially since pip-tools updates are needed to keep pip compatibility with the current ongoing changes in pip. Python 2.7 users cannot simply use an older pip-tools release if they want to update pip.
Also, as I recently learned, PyPy 2.7 isn't reaching EOL, so there's that to consider too.
https://twitter.com/pypyproject/status/1160209907079176192
https://pypy.readthedocs.io/en/latest/faq.html#how-long-will-pypy-support-python2
"Python 2.7 EOL" isn't as all-encompassing as it may seem.

[codingjoe]

@vphilippon I see your point regarding Python 2 and I'd love to agree if it wasn't for previous experiences. Pip's policy is very vague, which is likely to become a problem. It does not clearly define responsibilities or documents how bugs reported on Python 2 will be treated. It leaves me with more questions than answers and might promote a false sense of security to Python 2 users.

I am also not super unhappy with the discussion. It's mostly driven by corporations running won't update their legacy systems at the cost of people who dedicate their free time to open source. I for one will not write a single line of Python 2 compliant code anymore.

Finally, if Django with 100 times more users than this package was able to drop Python 2 support years ago, I don't see why we can't.

I personally see a lot more risks and issues keeping Python 2 support, than simply with dropping it.

[codingjoe]

It's also terrible for the planet to keep Python 2 tests running. What would Greta do 🙄

[atugushev]

I'd prefer to start with dropping the old pip version. Later, we can drop the python2 support as soon as pip does.

[codingjoe]

I'd prefer to start with dropping the old pip version. Later, we can drop the python2 support as soon as pip does.

Certainly, I'd also prefer to do those changes separately.

[atugushev]

I think we can have pip-tools==5.0.0 made, keeping support only to recent pip versions.
For which versions should exactly be supported, I leave that decision to you @atugushev. You know what's best at this point.

It's settled then, let's drop the old versions. It's a huge amount of work to drop pip<19.3 support (mostly for reviewers, for it would touch lots of LoC). Next version pip==20.0 is going to be released very soon, approximately 2020-01-15. Owing to a lack of time I suggest at first to release pip-tools compatible with pip==20.0 in the old way (WIP #1024) and after that start to work on dropping the old versions.

[atugushev]

Regarding using python -m ..., you have me convinced. All that is left is my darn resistance to change, and that's a "me" problem 😄 .

@vphilippon FYI, pip install ... is going to be deprecated later in favor of python -m pip install .... See https://github.com/pypa/pip/issues/3164.

[atugushev]

@fumblehool was the unpin unintentional? I believe we should keep the issue pinned, until we close it.

[fumblehool]

Apologies @atugushev , the unpin was unintentional.

[atugushev]

@fumblehool no worries! 😉

[atugushev]

FYI, #1055 is WIP for this issue.

[atugushev]

Hi all, #1055 is ready to review. I'd appreciate any feedback 🙏

[sfdye]

Now that #1055 is merged, anything left for this issue?

[atugushev]

It might be considered to be resolved. Recently though, we discussed privately with @vphilippon that it would be nice to have a section in README which would help users find the version of pip-tools with support for a specific version of pip. It could be a table, for the instance:

• pip-tools==4.5.1 compatible with pip>=8.1.1,<=20.0
• pip-tools==5.0.0 compatible with pip>=20.1,<=20.2.*
• pip-tools==5.1.0 compatible with pip>=20.2,<=20.3.*
• pip-tools==5.2.0 compatible with pip>=20.3
• etc ...

[atugushev]

it would be nice to have a section in README which would help users find the version of pip-tools with support for a specific version of pip.

Tracking issue #1089.

[atugushev]

pip-tools v5.0.0 is released! 🚀

[atugushev]

I've unpinned the issue since 5.0.0 is went smoothly.