Pipenv: Pipenv fails when a wheel of a dependency has been uploaded after the Pipfile was generated

Created on 16 Apr 2018 · 15Comments · Source: pypa/pipenv

Pipenv installations fail if an egg was uploaded for a dependency after Pipenv had already written the sdist hash to the Lockfile.

$ python3 -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '11.8.3'

Pipenv location: '/usr/lib/python3/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.14: /usr/bin/python
2.7.14: /usr/bin/python2
3.6.3: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-39-generic',
 'platform_system': 'Linux',
 'platform_version': '#44-Ubuntu SMP Thu Apr 5 14:25:01 UTC 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CLUTTER_IM_MODULE
LC_MEASUREMENT
LESSCLOSE
LC_PAPER
LC_MONETARY
ANDROID_HOME
XDG_MENU_PREFIX
LANG
LESS
MANAGERPID
DISPLAY
INVOCATION_ID
GNOME_SHELL_SESSION_MODE
COLORTERM
USERNAME
JAVA_HOME
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
LC_NAME
XDG_SESSION_ID
USER
DESKTOP_SESSION
QT4_IM_MODULE
TEXTDOMAINDIR
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
LESSHISTFILE
JOURNAL_STREAM
TEXTDOMAIN
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
PYTEST_ADDOPTS
XDG_SESSION_DESKTOP
TILIX_ID
LC_ADDRESS
DBUS_STARTER_ADDRESS
LC_NUMERIC
GTK_MODULES
WINDOWPATH
SELECTED_EDITOR
VTE_VERSION
SHELL
TERM
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
DBUS_STARTER_BUS_TYPE
XDG_CURRENT_DESKTOP
MOZ_USE_OMTC
PYTHONSTARTUP
SHLVL
XDG_SEAT
LC_TELEPHONE
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_CONFIG_DIRS
PATH
LC_IDENTIFICATION
SYSTEMD_NSS_BYPASS_BUS
SESSION_MANAGER
GCC_COLORS
LESSOPEN
GTK_IM_MODULE
LC_TIME
OLDPWD
_
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/remco/.local/share/android-sdk/tools/bin:/home/remco/.local/share/android-sdk/tools:/home/remco/.local/share/android-sdk/platform-tools:/home/remco/.local/share/gradle/bin:/home/remco/.local/bin:/home/remco/.local/share/android-sdk/build-tools/27.0.3:/home/remco/.gem/ruby/2.3.0/bin:/home/remco/.local/share/android-sdk/tools/bin:/home/remco/.local/share/android-sdk/tools:/home/remco/.local/share/android-sdk/platform-tools:/home/remco/.local/share/gradle/bin:/home/remco/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/remco

Steps to replicate

Given a project which has the following Pipfile:

[requires]
python_version = "3.6"

[packages]
pluggy = "*"

If this was installed before April 15th, 2018, this would have produced the following lockfile:

{
    "_meta": {
        "hash": {
            "sha256": "c9c0edfb60fe650018ada97d2fd71a66171e3d74def36de484f6a156802bcc5a"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pluggy": {
            "hashes": [
                "sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff"
            ],
            "index": "pypi",
            "version": "==0.6.0"
        }
    },
    "develop": {}
}

The pluggy hash matches the sha value from the sdist found on https://pypi.org/project/pluggy/#files.

Actual result

If pipenv sync is run from this project now (after pluggy wheels have been published), this outputs the following warning:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pluggy==0.6.0 from https://pypi.python.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl#md5=295745cab038ef139c75aa2cdb79a5b0 (from -r /tmp/pipenv-odt4x3d3-requirements/pipenv-5t_4hr2b-requirement.txt (line 1)):
        Expected sha256 7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff
             Got        e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5

It appears that pipenv downloads the newly uploaded wheel, but matches this against the known hash of the sdist.

Expected result

I guess pipenv should fall back to the sdist if the wheel doesn’t match any hashes.

Source

remcohaszing

👍5

Most helpful comment

@Overdrivr I’m not quite sure what your question is. pip does not know what hash belongs to which file; it simply receives a set of hashes, and make sure the file it downloads matches one of it. If you lock file includes only the tarball’s hash, but pip downloads a wheel file, the check would fail. Which is the problem.

What OP did was actually conceptually correct in terms of how this check is supposed to work:

Maintain a list of expected hashes.
Download a file.
Oops, the file does not match the hashes.
OP: “Hey so I downloaded this file and it does not match the previous one; did you change it?”
Maintainer: “Oh yeah that’s me, no worry.”
Okay cool so I update the list of expected hashes to include the new one.

Note that pipenv lock does not hash the downloaded file. The hashes it uses are from PyPI. It simply automates the process.

uranusjr on 20 Apr 2018

👍2

All 15 comments

Makes sense, but I don’t think we would want to do that silently. Will need to consider

techalchemy on 16 Apr 2018

I don't know enough about packaging conventions here, but would it make sense to continue downloading the sdist in this case, unless there's some way to guarantee that the the sdist and the wheel have the same contents? Otherwise the wheel could contain genuinely different code than the sdist, which would circumvent at least one of the points of using hashes in the first place.

alecbz on 16 Apr 2018

+1 on using sdist as fallback, but the implementation would probably be a bit messy because the error is not actually generated by Pipenv, but pip (Pipenv replaces the “requirement file” part with “Pipfile.lock”).

Maybe something like this (pseudo code):

def install(req_file_name):
    no_binary_packages = set()   # These are passed in the --no-binary option to pip.
    while True:
        c = pip_install(req_file_name, no_binary_packages)
        if c.return_code == 0:    # Success.
            return
        packages = parse_no_binary_packages(c.out)
        # Give up if the error does not result from a hash mismatch, or if a mismatch happens with --no-binary.
        if not packages or any(p in no_binary_packages for p in packages):
            raise PipInstallError(c.out)
        no_binary_packages.update(packages)

uranusjr on 17 Apr 2018

I'm a little slow following the issue here... is there a recommended work-around for the time being? Continue to pipenv lock when we encounter the error? Im just concerned that behavior will lead to me ignoring all future security warnings about mis-matched hashes. Not criticizing anyone - since I understand this is a free-software project :) just wondering if there any group recommendations in the mean-time.. thanks!

msheiny on 17 Apr 2018

I think this issue is rare in practice. Also the cause is fairly simple to find by simply looking at the pypi page of the mismatched hash. In this case: https://pypi.org/project/pluggy/#files

The workaround for now is to verify the mismatched hash against the newly uploaded dist and adding it to your lockfile manually.

remcohaszing on 17 Apr 2018

❤2

for others finding this thread, https://github.com/pytest-dev/pluggy/issues/134 clarified a lot for me :)

msheiny on 17 Apr 2018

👍1

Why did this happen in the first place ? Is it possible to replace the wheel of a package for a given version ?

Overdrivr on 18 Apr 2018

Yes it totally is, unfortunately.

uranusjr on 18 Apr 2018

If you do encounter this I would probably recommend just rerunning pipenv lock if you are opposed to adding things by hand

And yeah as a package maintainer you can upload the source distribution (.tar.gz) and then come back much later with wheels which pip will always prefer (thus the mismatch)

techalchemy on 18 Apr 2018

Just to say that I had the same issue and running a pip lock again did fix it.

andrecp on 20 Apr 2018

@techalchemy but in that case the hash should be for the .tar.gz not the wheel file right ? Pip would not use the hash for the .tar.gz file if it's downloading the wheel file, or am I wrong ?

I feel encouraging users to change hashes manually (or by doing a pip lock, with the same result), weakens this whole hash system. The goal is, after all, to make sure the package pip is downloading has not been tampered with.

Overdrivr on 20 Apr 2018

What OP did was actually conceptually correct in terms of how this check is supposed to work:

Maintain a list of expected hashes.
Download a file.
Oops, the file does not match the hashes.
OP: “Hey so I downloaded this file and it does not match the previous one; did you change it?”
Maintainer: “Oh yeah that’s me, no worry.”
Okay cool so I update the list of expected hashes to include the new one.

Note that pipenv lock does not hash the downloaded file. The hashes it uses are from PyPI. It simply automates the process.

uranusjr on 20 Apr 2018

👍2

I’m going to close this for now as it’s uncommon enough that we can just add a documentation note for it

techalchemy on 20 Apr 2018

👎1

Hey all! We just got bit by this with isort. It looks like they released a wheel of their package a couple of days after the source tar.gz was uploaded to pypi. I did some digging before finding this issue, so I thought it might useful to post some of the stuff I found.

On pypi (https://pypi.org/simple/isort/), I now see two isort 4.3.9 versions:

sha256=ee5fddfd792e6e1d664ee28f3fbe00dfc26d8d3c6f059ee78f4da4c19718007c isort-4.3.9-py2.py3-none-any.whl
sha256=f19b23b22fb5a919a081bc31aabcc0991614c244d9215267e11abf2ca7b684ce isort-4.3.9.tar.gz

Here's a simplified version of the pip command that pipenv was running when it failed:

$ pip install --upgrade --no-deps -r <(echo "isort==4.3.9 --hash=sha256:f19b23b22fb5a919a081bc31aabcc0991614c244d9215267e11abf2ca7b684ce") -i https://python-packages.honordev.com/build/dev/ --require-hashes
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://python-packages.honordev.com/build/dev/
Collecting isort==4.3.9 (from -r /proc/self/fd/11 (line 1))
  Using cached https://python-packages.honordev.com/root/pypi/+f/ee5/fddfd792e6e1d/isort-4.3.9-py2.py3-none-any.whl
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    isort==4.3.9 from https://python-packages.honordev.com/root/pypi/+f/ee5/fddfd792e6e1d/isort-4.3.9-py2.py3-none-any.whl#sha256=ee5fddfd792e6e1d664ee28f3fbe00dfc26d8d3c6f059ee78f4da4c19718007c (from -r /proc/self/fd/11 (line 1)):
        Expected sha256 f19b23b22fb5a919a081bc31aabcc0991614c244d9215267e11abf2ca7b684ce
             Got        ee5fddfd792e6e1d664ee28f3fbe00dfc26d8d3c6f059ee78f4da4c19718007c

Does this feel like a pip bug to anyone else? I get that pip normally prefers wheels, but if I specify a hash of a source dist, shouldn't pip just go ahead and use the source dist for installation?

Did any documentation of this ever get added? If not, I'd be happy to add some. This is uncommon, but quite confusing when it does happen.

Edit: Looks like this was discussed over on the pip issue tracker here: https://github.com/pypa/pip/issues/5874, and appears to have been fixed over in https://github.com/pypa/pip/pull/6699! According to the [pip release notes], the fix went out in pip 19.2. I confirmed that upgrading to pip >= 19.2 does address this problem without any changes to pipenv =)

jfly on 28 Feb 2019

this is happening to me today with botocore even though there's a wheel and a source tarball already present in PyPi. pipenv install on my Mac is pulling the source tarball when I create the Pipfile.lock, whereas the Linux CI machine is pulling the wheel and that SHA is not present in the lockfile.

Can we get pipenv to pull _all_ available SHAs when creating the lockfile, instead of just whichever one works for it? Anything that is available to be installed at the time of installing should be valid, right?