Python: Add pipenv to python images

[riconnon]

Since pipenv is now the "recommended" packaging tool for python dependencies it would be nice to have it included in the official python images (or have additional tags available which include it)

[riconnon]

The simplest way to do this would be to simply replace the sourcing and running of get-pip.py with get-pipenv.py which bootstraps pip and pipenv together.

[yosifkit]

We install pip because it comes bundled with python source itself. We only use get-pip.py to get a more up-to-date version. I cannot find a recommendation on python.org that it should be installed by default with python. Even the dependency tutorial for managing dependencies assumes that it needs to be installed. Is there a PEP or document from the Python Software Foundation that recommends it?

The closest I can find is in the meeting notes from September:

Working vigorously on Pipenv, which is now the (soon–to–be) officially recommended packaging tool

- https://www.python.org/psf/records/board/minutes/2017-09-26/#kenneth-reitz

[dchevell]

@yosifkit the pipenv project now lives under the Python Package Authority's official projects on github, and describes itself as "the officially recommended Python packaging tool from Python.org, free (as in freedom)."

The link above points to a guide on packaging.python.org that notes: "Pipenv is recommended for collaborative projects as it’s a higher-level tool that simplifies dependency management for common use cases."

[techalchemy]

Hi @yosifkit, I'm a pipenv maintainer -- thanks for whatever docker libraries you are keeping up to date, it is much appreciated and quite helpful!

As far as deploying pipenv by default, I can't really say what makes sense for you to do by default since I'm not too sure what the average docker user wants or expects. I can tell you a bit about pipenv however:

Pipenv is designed for managing environments & dependencies, ensuring you don't install conflicting dependencies in your environment, and making sure you can reproduce your exact environment when you deploy. That last case is probably the best use case for a docker install of pipenv.

I'm not too sure how people are using docker these days, but pipenv install has a bit of overhead in terms of building a local cache for all the packages it finds during resolution, compared to pip which just installs the first matching version of any package it finds. It also by default puts your packages into a virtualenv in $WORKON_HOME or $HOME/.virtualenvs unless you have set PIPENV_VENV_IN_PROJECT=1. Then you can call it via pipenv run <command> instead of activating your virtualenv like normal.

I can see a good argument for either a tag or a default option, but there is no universe in which pip shouldn't also be kept installed and up to date. We are part of the packaging authority and we both leverage and extend pip, rather than replacing it.

That's probably good enough, but i'll /cc @ncoghlan just in case

[ncoghlan]

pipenv is mainly a development tool, so I don't think it really belongs in the base runtime docker images for Python - there are lots of reasons to keep those as small as is practical, and pipenv is mainly intended for use on developer workstations and in CI pipelines, not as part of the final deployed environment.

Where I think including pipenv by default may make sense is if Docker were to publish a python-build image for use in multi-stage pipelines: https://docs.docker.com/develop/develop-images/multistage-build/#before-multi-stage-builds

That way, pipenv could be used to help construct a runtime image, but it wouldn't need to be present in the runtime image itself. (This kind of approach is also likely to interoperate better with CI build pipelines that allow build image caches to be preserved from run to run)

[dbazile]

While I agree it’s probably premature to include pipenv at this point, I think the core of what most are looking for is to not have to use both requirements.txt and Pipfile.

Admittedly, I’m not sure if dev effort is being diverted from implementing pipfile in vanilla pip because pipenv exists. But if the community will converge on pipenv, it’s inevitable that it will cease to just be a “devtool”.

Just my view from the peanut gallery anyway. :)

[ncoghlan]

I expect pipwill eventually gain the ability to read Pipfile.lock files (they're just JSON metadata after all), but I also expect it will be a while before the pipenv devs declare that format stable enough for other tools to start consuming it. In the meantime, exporting to the flat requirements.txt format doesn't actually lose any information that matters to pip.

More generally, even pip would ideally be missing from the base Python docker images, as an immutable deployment artifact doesn't need the ability to dynamically update or install packages.

While backwards compatibility makes it hard to remove at this late stage, the main reason it is there at all is the fact that Docker historically made it painful to have separate build and deployment images, but that's no longer the case given the introduction of multi-stage builds.

[dbazile]

In the meantime, exporting to the flat requirements.txt format doesn't actually lose any information that matters to pip.

Though it is an extra step, that's an entirely reasonable workaround.

More generally, even pip would ideally be missing from the base Python docker images, as an immutable deployment artifact doesn't need the ability to dynamically update or install packages.

@ncoghlan Off topic but out of curiosity, if pip was removed from the base Python Docker image, how would one get dependencies into the image? Would someone have to install pip on their own or try to use apt to install it and/or other packages? Or did you have in mind something similar to what the the official maven image does by layering on top of the official openjdk image, having an official pip/pipenv and a packager-less official python?

Edit: never mind, I saw you answered this in the first message.

[tianon]

Pip is included because the standard distribution of Python includes it --
we simply update it explicitly to ensure it's a fresh and consistent
version.

[ncoghlan]

@dbazile Longer term, I'd recommend pursuing the maven approach for Python as well (perhaps by offering a python-minimal image, rather than by changing the regular python image - then the full python image could be used as the builder image, without breaking backwards compatibility).

We default to providing pip by default in the upstream Python installers, because we're optimising for the "new Python developer working through an online tutorial without a local teacher" case. Folks in that situation are far more dependent on default behaviours to get going than more experienced developers optimising an immutable Docker image for their application.

[tianon]

Ah, that makes sense. So without pip in the image, how would you imagine the workflow going for someone to install a package from PyPI in their image? Downloading get-pip.py themselves? Downloading the package explicitly and invoking setup.py directly?

[ncoghlan]

They wouldn't - they'd create the runtime virtual environment using a builder image (which would provide pip as usual), and then emit an immutable pip-free runtime image. It's the same general principle as omitting maven from Java runtime images, or gcc (et al) from C/C++ runtime images.

[thernstig]

I'm nowhere near the expertise of you guys in regards to pip/pipenv, but I think it would be great if pipenv was included by default in the official docker images. I've worked on more than one project moving from pip -> pipenv (e.g. requirements.txt to Pipefile+Pipefile.lock).

All these projects build Docker files that require us to start with something like:

```Dockerfile
RUN pip install --no-cache-dir --trusted-host pypi.python.org pipenv
COPY Pipfile ./
COPY Pipfile.lock ./
RUN pipenv install --system --deploy
````

Also GitLab, which is used by more and more, often use Docker images to run CI jobs like this:

```yaml
pylint:
image: python:3.6
stage: lint
script:
- pip install --no-cache-dir --trusted-host pypi.python.org pipenv
- pipenv install --system --deploy --dev
- pylint somefile.py
````

The more that move towards pipenv the more will require the extra step of installing pipenv. It just becomes a copy&paste to use pipenv with Docker/containers. A concept which is also growing.

[tianon]

Ok, I've gone and done a little digging to put some hard numbers to the two proposals here.

I'm performing my comparisons using the latest python:3.6-alpine3.8, which at the time of this writing is ~74.2MB (sha256:9504ba5b60fcb1658c9ac0bdc9e26bd128c51d4dbfe73cedff4cba570e602fd1 for anyone who wants an explicit specific digest to the exact amd64 image I'm currently looking at).

Looking at docker history, the layer which installs pip is ~5.93MB.

Doing pip install pipenv, we get a layer which is ~32.7MB (:exclamation:) by itself (that only goes down to ~25.5MB if I add --no-cache-dir).

So, for the idea of installing pipenv by default, I'm not comfortable with a ~1.5x increase in total image size when this isn't a tool officially provided as part of a standard Python install from https://www.python.org/ (unlike pip).

Regarding the idea of having _another_ variant which does not include pip, the savings do not justify the additional maintenance burden. At only ~6MB, the extracted on-disk size is already a very negligible part of a ~74MB image, which gets _even smaller_ when you consider that Docker applies compression in-transit. If we take a look at https://github.com/docker-library/repo-info/blob/88fb4c9740c15b399a20c0f9a25d362680605b49/repos/python/remote/3.6-alpine3.8.md#python36-alpine38---linux-amd64, we can see that this image is ~27.4MB compressed, and I cannot possibly imagine pip being any significant portion of that (in fact, I did some testing, and removing the pip layer results in a ~1MB decrease in compressed size).

[max-wittig]

The main pain point with pip itself is hashes, which it cannot generate.

The python packaging ecosystem still feel broken.