Pip: update pip install docs for ubuntu:focal

[mirekphd]

What's the problem this feature will solve?
Lack of documented method of installing pip under python 3.8 (pre-installed default) in the latest LTS version of Ubuntu - Focal Fossa , where there is no python 2 anymore. Lack of pre-installed python 2 in python3-only Ubuntu LTS systems leads to failure to to install pip with pip (as opposed to with pip3), at least in docker containers, where we start from fresh operating system at every build (see docker pull ubuntu:focal).

Describe the solution you'd like
Documented fully reproducible steps to install pip on Ubuntu Focal Fossa (tested on the official Docker image from Docker Hub):

$ sudo apt update && sudo apt install -y python3-pip
$ pip3 install --upgrade pip
$ pip install <package1> <package2> <package3>

Additional context
This problem affects building any Docker container with ubuntu:focal as base image.
This is intended for Github docs, not for PyPI installation instructions, where pip3 install pip would not be any better, as this is needed mostly for docker builds - the only scenario where we encounter this problem every time (as opposed to only first time we install pip, until pip is symlinked to pip3).
Notice that after the above installation instructions, pip is a python 3.8 package, not python 2.7:

$ pip --version
pip 20.1.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)

[uranusjr]

Ubuntu (and various other Linux distributions) supply their own modified version of pip, with their own policies and installation methods. Personally I’d disfavour documenting such topics since it adds responsibilities to pip maintainers without any merits whatsoever.

[pfmoore]

As this is a request for specific documentation on "how to install pip on Ubuntu", this would be better published as an Ubuntu document. The pip documentation is intended to be (and should remain) generic.

That's true even if we ignore any customisations Ubuntu did to pip.

[mirekphd]

this would be better published as an Ubuntu document. The pip documentation is intended to be (and should remain) generic.

That's true even if we ignore any customisations Ubuntu did to pip.

This issue was formulated as Ubuntu-only issue for reproducibility (did not know about their customisations though), but it's likely someone runs into its equivalent on Centos and other systems. A more generic problem that causes this issue is the dual naming of pip that seems to have lost its rationale, once py2 is gone. Again, you might argue the issue is an Ubuntu-specific one, as it was their system library that installed pip3 above:)

[McSinyx]

Hi @mirekphd, I've reread this thread a couple of times and couldn't figure out clearly about the problem you are facing. IIUC, you're referring to the packaging guide on python.org where the instruction for Debian-based distros is documented as

• Python 2:

  sudo apt install python-pip

• Python 3:

  sudo apt install python3-venv python3-pip

Warning: Recent Debian/Ubuntu versions have modified pip to use the “User Scheme” by default, which is a significant behavior change that can be surprising to some users.

(I included the warning here to low-key ask if we want to remove it since recent Debian-based distros would have pip 20+ where the distros' patches are dropped in favor of pip handling such thing.)

I took a look at Focal's downstream content I wonder why python3-* packages and binary are not provided dummy python-* like how transition is handled in Debian packaging ecosystem (Debian have yet to drop Python 2, I'm curious how it will handle the situation). Usually, such problem is handled via alternatives.

That being said, I don't think Focal introduced any behavioral change to python3-pip. What we might want to do here is to let new users know that the package provides solely pip3 instead of also pip and pip3.8 like upstream, although I don't know if we want to do that since we can nudge distros maintainers to instead provide the binary as there's no ambiguity after Python 2 sunset.

IIRC, the recommend method to invoke pip is via python -m pip. Again, I am surprised that Focal doesn't seem to provide python either.

[mirekphd]

Hi @McSinyx. Apologies for lack of clarification of what I meant by 'docs'. The document you are referring to is a bit dated now (Last Reviewed: 2015-09-17, so nearly 5 years old) and thus arguably would welcome some refreshing (e.g. to reflect such changes as python 3 "LTS" (pre-packaged) versions upgrading from 3.4 to 3.8. But the problem I'm trying to point out here is that it does not offer installation instructions for pip, but for pip3 (as we saw in my code snippet above only pip installs pip, while python3-pip installs... pip3. Why do we need this distinction where pip2 is no more a) pre-installed, b) supported? I think it's high time pip3 became just pip, and if someone still wants to install python2 (which is possible even in Ubuntu Fossa), then let them have the fun that we have been having for the past decade where the default was obsolete python2.

The changes in ubuntu:focal seem to be mostly beneficial and backward compatible and need to be celebrated and embraced to the fullest in the docs, which have gotten a bit obsolete now that pip2 is no longer pre-installed with popular linux distributions.

I agree that getting rid of the python version suffix and having APT install just pip rather than pip3 is not absolutely future-proof, as it is not prepared for the situation where python4 ever arrives (and becomes immediately the version of choice, so that we are back to the last decade's situation with pip2==pip vs pip3).

_Boostrap script (a side note)_
And then there is the "bootstrap" remote loader script, that used to be the only installation route (prior to bionic and if you did not use Miniconda). It had to be side-loaded with curl, renamed to .sh and run. In corporate settings its non-standard URL had to be first white-listed and then you would need to convince bearded guys from the security dept that it's the recommended pip installation method (not at all suspect) to download executable code pasted into a text file only to install everything else with it. May I take this opportunity to kindly ask you to deprecate this method that caused so much unnecessary discussions with security ppl (just like the competing bootstrapper, Miniconda)? Or at least please move it out of its prominent place in the docs (I could not believe today it's still at the top of the page). I'd argue the existence of this bootstrapping script has been made obsolete by apt-installed pip as far as in the _previous_ Ubuntu LTS, bionic (and the docs indeed precede bionic 2018 release by some 2-3 years).

[McSinyx]

@mirekphd, before we jump back into the discussion on what downstream should do and what PyPA should document to help users realizing such thing, let me throw in a few relevant information. First, let's acknowledge that upstream pip provide 3 different entry points, pip[major[.minor]]: https://github.com/pypa/pip/blob/445711c533b707f056a8c9512d55aa2986dee157/setup.py#L77-L85

I think (if any maintainer see this, please voice your opinion) it is generally favored to use python[major[.minor]] -m pip instead to distinguish between different Python version of pip (on Debian testing ATM I'm having py27, py37 and py38 installed from the distro's repo--what a time to be alive!) instead of using the entry points directly. I'm not experienced enough, but I believe entry points are to provide Python-version-independent utilities, it's not designed to cleanly handle situations like this, e.g. if one have pip for all py27, py37 and py38 (like I have on Debian testing ATM--what a time to be alive!), which pip in $PATH would be installation-time-dependent (ugh!).

GNU/Linux distributions, however, are created to solve such kind of problem by pulling off a whole bunch of hacks. And hacks are hacks, people never agree on which kind of hack is least ugly (proof: look up the number of distributions :smile:). On Debian (and assumably on Ubuntu as well), the established way to solve this is through alternatives, to have a consistent way to determine which pip is pip. As for backward compatibility, there's LTS for a reason: one can always choose not to upgrade. During the upgrade process there's system mail BTW.

I strongly believe not providing (configuring it by default I mean) the alternatives for pip on Focal is a packaging bug. I'll file this to Debian when Bullseye enter the freeze phase, but Ubuntu is out of my personal interest.

Additionally, it is strongly recommended by most distros to only obtain a piece of software from their repos, unless one's developing it. so the bootstrap should be ignored by any user that care to RTFM (hint: no one, including me, does). get-pip.py is for obsolete systems that somehow don't have a better way to install pip.

So what can we (maintainers and contributors like you and me) can do? Off my mind at the moment there are quite a few things:

• Nudge downstreams to provide a convenient way to specify which pip is pip
• Move the installation from distro's repo above the boostrap method, since the section is literally a one-liner.

I'll try my best to not get driven away by this conversation, since as stated above, arguments on what downstreams could/should do and upstream-downstream relationship is something people in general can never agree on. If your goal is simply having a deterministic way to invoke pip, use python3.8 -m pip:

Explicit is better than implicit.

[mirekphd]

python[major[.minor]] -m pip

I completely agree that this is the unambiguous (and recommended) way to invoke pip. I suppose it's just too time consuming to re-write the legacy code which invokes pip directly a few hundred times too many... like this beauty over there (which could be stripped of a few hundred pip install's as well...:): Kaggle/docker-python/Dockerfile.

I strongly believe not providing (configuring it by default I mean) the alternatives for pip on Focal is a packaging bug. I'll file this to Debian when Bullseye enter the freeze phase, but Ubuntu is out of my personal interest.

I'm glad I could bring your attention to this issue. Let's move the discussion over there, so please make sure to mention #8429 when you are at it, so that we can close this issue here (rather than forgetting it completely).

[McSinyx]

it's just too time consuming to re-write the legacy code which invokes pip directly a few hundred times

In this case something like sed -i s/ pip/ python3.8 -m pip/g Dockerfile plus manual final correction might accelerate the process.

Let's move the discussion over [Debian bug tracker]

I think you might want to file a bug report to Ubuntu (or there might be one already) because at this point I've yet to receive any news on how Debian will deprecate Python 2. Bullseye will freeze early next year so I don't think the deprecation will happen anytime soon, but if you find some interesting discussion on the matter, I'd love to be tagged in (see my email on my profile). Cheers!

[mirekphd]

it's just too time consuming to re-write the legacy code which invokes pip directly a few hundred times

In this case something like sed -i s/ pip/ python3.8 -m pip/g Dockerfile plus manual final correction might accelerate the process.

Or maybe it's not so much the lack of bash expertise (they are Google subsidiary after all:), but _brevity_ that makes "pip" preferred over "python3.x -m pip" by such practitioners as Kaggle there... Notice also Kaggle's appalling disregard for another of pypa recommendations (install python3-venv) - python / conda virtual environments are perfectly fine... except when your data scientists want access to all packages at any time (and this is why large data science platforms like theirs will install all packages (from pypi, github, and conda) in a single complex root env - after all, it's usually a docker container, so a very predictable environment).

I'm surprised to learn that Debian still comes shipped with Python 2 - who will patch critical vulnerabilities if and when they are discovered (for how long is it, whole year, really?:)