pipenv 2020.8.13 failing to install some package dependencies

Created on 19 Aug 2020  Â·  7Comments  Â·  Source: pypa/pipenv

Issue description

installing packages with pipenv 2020.8.13 is failing to install all package dependencies, or installing them in such a way that I can't _tell_ they're installed. ¯\_(ツ)_/¯

specifically I'm seeing it having issues installing six for some strange reason, which may very well have something to do with my invocation of pipenv, so please refer to the "Steps to replicate" section for a detailed reproduction example.

Expected result

I expect all packages and dependencies to be installed.

example output after installing with pipenv 2020.6.2, which works as expected:

$ python -m pip list
Package        Version
-------------- -------
attrs          19.3.0
iniconfig      1.0.1
more-itertools 8.4.0
packaging      20.4
pip            20.2.2
pluggy         0.13.1
py             1.9.0
pyparsing      2.4.7
pytest         6.0.1
setuptools     49.3.1
six            1.15.0
toml           0.10.1
wheel          0.34.2

Actual result

six is clearly missing and understandably fails when I attempt to import it.

example output after installing with pipenv 2020.8.13, which show six doesn't seem to be installed:

$ python -m pip list
Package        Version
-------------- -------
attrs          19.3.0
iniconfig      1.0.1
more-itertools 8.4.0
packaging      20.4
pip            20.2.2
pluggy         0.13.1
py             1.9.0
pyparsing      2.4.7
pytest         6.0.1
setuptools     49.3.1
toml           0.10.1
wheel          0.34.2

Steps to replicate

given the Pipfile and Pipfile.lock from the pipenv --support details below and the following Dockerfile:

FROM python:3.8-alpine as build-old

ENV PYTHONUSERBASE /pyvenv_old
WORKDIR /build

COPY Pipfile Pipfile.lock ./

RUN set -x && \
    python -m pip install pipenv==2020.6.2 && \
    PIP_USER=1 \
    PIP_IGNORE_INSTALLED=1 \
    pipenv install --system --deploy


FROM python:3.8-alpine as build-new

ENV PYTHONUSERBASE /pyvenv_new
WORKDIR /build

COPY Pipfile Pipfile.lock ./

RUN set -x && \
    python -m pip install pipenv==2020.8.13 && \
    PIP_USER=1 \
    PIP_IGNORE_INSTALLED=1 \
    pipenv install --system --deploy


FROM python:3.8-alpine

COPY --from=build-old /pyvenv_old /pyvenv_old
COPY --from=build-new /pyvenv_new /pyvenv_new

RUN echo "# 2020.6.2 packages" > pipenv_installed_packages && \
    PYTHONUSERBASE=/pyvenv_old python -m pip list >> pipenv_installed_packages && \
    echo -e "\n# 2020.8.13 packages" >> pipenv_installed_packages && \
    PYTHONUSERBASE=/pyvenv_new python -m pip list >> pipenv_installed_packages

CMD ["cat", "pipenv_installed_packages"]

if you place all three files in the same dir you can verify what I'm seeing with these commands:

$ docker build -t pipenv_six_bug .
<snip build output>

$ docker run --rm pipenv_six_bug
# 2020.6.2 packages
Package        Version
-------------- -------
attrs          19.3.0
iniconfig      1.0.1
more-itertools 8.4.0
packaging      20.4
pip            20.2.2
pluggy         0.13.1
py             1.9.0
pyparsing      2.4.7
pytest         6.0.1
setuptools     49.3.1
six            1.15.0
toml           0.10.1
wheel          0.34.2

# 2020.8.13 packages
Package        Version
-------------- -------
attrs          19.3.0
iniconfig      1.0.1
more-itertools 8.4.0
packaging      20.4
pip            20.2.2
pluggy         0.13.1
py             1.9.0
pyparsing      2.4.7
pytest         6.0.1
setuptools     49.3.1
toml           0.10.1
wheel          0.34.2

obviously simplified for repro, this is a common pattern we use at my company to pare down the final image
and only use pipenv for package installation.

$ pipenv --support for 2020.6.2

Pipenv version: '2020.6.2'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

  • 3.8.5: /usr/local/bin/python3
  • 3.8.5: /usr/local/bin/python3.8

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-42-generic',
 'platform_system': 'Linux',
 'platform_version': '#46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

  • HOSTNAME
  • PYTHON_PIP_VERSION
  • SHLVL
  • HOME
  • GPG_KEY
  • PYTHON_GET_PIP_URL
  • PATH
  • LANG
  • PYTHONUSERBASE
  • PYTHON_VERSION
  • PWD
  • PYTHON_GET_PIP_SHA256
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • LANG: C.UTF-8
  • PWD: /build

Contents of Pipfile ('/build/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
pytest = "==6.0.1"

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/build/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "224c1f1c10cc0c59f97d80c3a38b2d19dffd6763a9da042cec329a0abda08b99"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==19.3.0"
        },
        "iniconfig": {
            "hashes": [
                "sha256:80cf40c597eb564e86346103f609d74efce0f6b4d4f30ec8ce9e2c26411ba437",
                "sha256:e5f92f89355a67de0595932a6c6c02ab4afddc6fcdc0bfc5becd0d60884d3f69"
            ],
            "version": "==1.0.1"
        },
        "more-itertools": {
            "hashes": [
                "sha256:68c70cc7167bdf5c7c9d8f6954a7837089c6a36bf565383919bb595efb8a17e5",
                "sha256:b78134b2063dd214000685165d81c154522c3ee0a1c0d4d113c80361c234c5a2"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==8.4.0"
        },
        "packaging": {
            "hashes": [
                "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8",
                "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==20.4"
        },
        "pluggy": {
            "hashes": [
                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==0.13.1"
        },
        "py": {
            "hashes": [
                "sha256:366389d1db726cd2fcfc79732e75410e5fe4d31db13692115529d34069a043c2",
                "sha256:9ca6883ce56b4e8da7e79ac18787889fa5206c79dcc67fb065376cd2fe03f342"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.9.0"
        },
        "pyparsing": {
            "hashes": [
                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
            ],
            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==2.4.7"
        },
        "pytest": {
            "hashes": [
                "sha256:85228d75db9f45e06e57ef9bf4429267f81ac7c0d742cc9ed63d09886a9fe6f4",
                "sha256:8b6007800c53fdacd5a5c192203f4e531eb2a1540ad9c752e052ec0f7143dbad"
            ],
            "index": "pypi",
            "version": "==6.0.1"
        },
        "six": {
            "hashes": [
                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.15.0"
        },
        "toml": {
            "hashes": [
                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
            ],
            "version": "==0.10.1"
        }
    },
    "develop": {}
}

$ pipenv --support for 2020.8.13

Pipenv version: '2020.8.13'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/bin/python'

Python installations found:

  • 3.8.5: /usr/local/bin/python3
  • 3.8.5: /usr/local/bin/python3.8

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-42-generic',
 'platform_system': 'Linux',
 'platform_version': '#46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020',
 'python_full_version': '3.8.5',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

  • HOSTNAME
  • PYTHON_PIP_VERSION
  • SHLVL
  • HOME
  • GPG_KEY
  • PYTHON_GET_PIP_URL
  • PATH
  • LANG
  • PYTHONUSERBASE
  • PYTHON_VERSION
  • PWD
  • PYTHON_GET_PIP_SHA256
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • LANG: C.UTF-8
  • PWD: /build

Contents of Pipfile ('/build/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
pytest = "==6.0.1"

[requires]
python_version = "3.8"

Contents of Pipfile.lock ('/build/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "224c1f1c10cc0c59f97d80c3a38b2d19dffd6763a9da042cec329a0abda08b99"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "attrs": {
            "hashes": [
                "sha256:08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c",
                "sha256:f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==19.3.0"
        },
        "iniconfig": {
            "hashes": [
                "sha256:80cf40c597eb564e86346103f609d74efce0f6b4d4f30ec8ce9e2c26411ba437",
                "sha256:e5f92f89355a67de0595932a6c6c02ab4afddc6fcdc0bfc5becd0d60884d3f69"
            ],
            "version": "==1.0.1"
        },
        "more-itertools": {
            "hashes": [
                "sha256:68c70cc7167bdf5c7c9d8f6954a7837089c6a36bf565383919bb595efb8a17e5",
                "sha256:b78134b2063dd214000685165d81c154522c3ee0a1c0d4d113c80361c234c5a2"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==8.4.0"
        },
        "packaging": {
            "hashes": [
                "sha256:4357f74f47b9c12db93624a82154e9b120fa8293699949152b22065d556079f8",
                "sha256:998416ba6962ae7fbd6596850b80e17859a5753ba17c32284f67bfff33784181"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==20.4"
        },
        "pluggy": {
            "hashes": [
                "sha256:15b2acde666561e1298d71b523007ed7364de07029219b604cf808bfa1c765b0",
                "sha256:966c145cd83c96502c3c3868f50408687b38434af77734af1e9ca461a4081d2d"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==0.13.1"
        },
        "py": {
            "hashes": [
                "sha256:366389d1db726cd2fcfc79732e75410e5fe4d31db13692115529d34069a043c2",
                "sha256:9ca6883ce56b4e8da7e79ac18787889fa5206c79dcc67fb065376cd2fe03f342"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.9.0"
        },
        "pyparsing": {
            "hashes": [
                "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1",
                "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
            ],
            "markers": "python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==2.4.7"
        },
        "pytest": {
            "hashes": [
                "sha256:85228d75db9f45e06e57ef9bf4429267f81ac7c0d742cc9ed63d09886a9fe6f4",
                "sha256:8b6007800c53fdacd5a5c192203f4e531eb2a1540ad9c752e052ec0f7143dbad"
            ],
            "index": "pypi",
            "version": "==6.0.1"
        },
        "six": {
            "hashes": [
                "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259",
                "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.15.0"
        },
        "toml": {
            "hashes": [
                "sha256:926b612be1e5ce0634a2ca03470f95169cf16f939018233a670519cb4ac58b0f",
                "sha256:bda89d5935c2eac546d648028b9901107a595863cb36bae0c73ac804a9b4ce88"
            ],
            "version": "==0.10.1"
        }
    },
    "develop": {}
}

Type
Source
picture joshsleeper

Most helpful comment

Here's my solution using the venv method for a multi-stage build:

FROM python:3.8-alpine as build

WORKDIR /build

COPY Pipfile Pipfile.lock ./

RUN set -x && \
    python -m pip install pipenv==2020.8.13 && \
    PIPENV_VENV_IN_PROJECT=1 \
    pipenv install --deploy


FROM python:3.8-alpine

COPY --from=build /build /build
ENV PATH /build/.venv/bin:$PATH

# ...

Using PIPENV_VENV_IN_PROJECT=1 puts the venv in a known/stable path that you can copy to the second stage, and adding .venv/bin to your PATH makes any later commands running python use the venv.

picture youngbob on 20 Aug 2020
❤2

All 7 comments

Changing the user base and relying on the pip behavior is undocumented so it isn't guaranteed to work well. If you install by the normal venv method every package is installed correctly.

frostming picture frostming on 19 Aug 2020
👍1

I'm encountering this issue as well, utilizing the USERBASE environment variable is very useful for multi-stage docker builds, is there a way that the old behavior can be supported?

Zerthick picture Zerthick on 19 Aug 2020

@frostming I can understand where you're coming from, although I guess that leaves me in a weird place for the moment.

how about this: if we wanted to use pipenv to handle the install with intent to copy that single folder into a second docker build stage, do you have any suggestions or thoughts for how that could be done?

since pipenv isn't used for the shipped application, it seems like there should be a reasonable way to use it only for the install portion of the build without including it in the final image.

joshsleeper picture joshsleeper on 19 Aug 2020
👍2

Here's my solution using the venv method for a multi-stage build:

FROM python:3.8-alpine as build

WORKDIR /build

COPY Pipfile Pipfile.lock ./

RUN set -x && \
    python -m pip install pipenv==2020.8.13 && \
    PIPENV_VENV_IN_PROJECT=1 \
    pipenv install --deploy


FROM python:3.8-alpine

COPY --from=build /build /build
ENV PATH /build/.venv/bin:$PATH

# ...

Using PIPENV_VENV_IN_PROJECT=1 puts the venv in a known/stable path that you can copy to the second stage, and adding .venv/bin to your PATH makes any later commands running python use the venv.

youngbob picture youngbob on 20 Aug 2020
❤2

This solution doesn't seem to work for me, the libraries don't seem to be found even though they are on the path, for example running which gunicorn in an attached shell returns the path to the gunicorn binary in the venv, but when the python script executes it returns FileNotFoundError: [Errno 2] No such file or directory: 'gunicorn': 'gunicorn'

Zerthick picture Zerthick on 20 Aug 2020

Ok I got it to work, if anyone is looking for a more complete example using a venv install you can look here: https://sourcery.ai/blog/python-docker/

Zerthick picture Zerthick on 21 Aug 2020

@youngbob's solution feels correct to me (thanks for that!), and I'm guessing that's what you were implying with your statement @frostming.

if you've got a better idea than that I've love to see an example, but otherwise I'm happy to consider this question answered.

joshsleeper picture joshsleeper on 23 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

fbender picture fbender  Â·  3Comments
AkiraSama picture AkiraSama  Â·  3Comments
jakul picture jakul  Â·  3Comments
Californian picture Californian  Â·  3Comments
bgjelstrup picture bgjelstrup  Â·  3Comments