Pipenv: Outlaw packages with version =* from the lock file

Created on 21 Oct 2019  ·  9Comments  ·  Source: pypa/pipenv

I have come across a number of packages where one or more dependency has a version of * i.e. latest or any but it is present in the lock file. This means that I cannot install that package unless:

  1. The current latest happens to match or
  2. I can guess which version it is supposed to be

It also results in automated tests such as travis failing after a time even if there have been zero code changes, e.g. a rebuild because of a documentation only change since the dependencies have moved on since the last time that there was a push to the project. This is an extra burden for the maintainers and tends to be a barrier for new contributors.

Describe the solution you'd like

I would like a version specifier of * to either result in the lock file being ignored for that package.

Describe alternatives you've considered

If a version specifier of * with a SHA in lock was an error then this would tend to force the original authors to address the issue by specifying the specific version - but may be frustrating.

Additional context

An example of this happening:
https://github.com/psf/requests-html/pull/338

Worker information
0.19s0.01s0.00s0.01s
system_info
Build system information
0.02s0.01s0.35s0.28s0.06s0.00s0.05s0.00s0.01s0.01s0.01s0.01s0.01s0.00s0.00s0.03s0.00s0.01s0.41s0.00s0.00s0.00s0.01s0.00s0.13s0.01s0.95s0.00s0.00s0.07s0.00s2.90s0.00s2.41s
docker_mtu
resolvconf
git.checkout
0.97s$ git clone --depth=50 https://github.com/psf/requests-html.git psf/requests-html
0.01s0.01s$ source ~/virtualenv/python3.6/bin/activate
$ python --version
Python 3.6.7
$ pip --version
pip 19.0.3 from /home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pip (python 3.6)
install.1
0.61s$ pip install pipenv --upgrade-strategy=only-if-needed
37.24s$ pipenv install --dev
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing dependencies from Pipfile.lock (7312a6)…
An error occurred while installing pluggy==0.6.0 --hash=sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff! Will try again.
     ================================ 46/46 — 00:00:26
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pipenv/core.py", line 1874, in do_install
[pipenv.exceptions.InstallError]:       keep_outdated=keep_outdated
[pipenv.exceptions.InstallError]:   File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pipenv/core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pipenv/core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pipenv/core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/pipenv/core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Looking in indexes: https://pypi.python.org/simple', 'Collecting pluggy==0.6.0 (from -r /tmp/pipenv-03jpv10e-requirements/pipenv-3k4kv96o-requirement.txt (line 1))', '  Using cached https://files.pythonhosted.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl']
[pipenv.exceptions.InstallError]: ['THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.', '    pluggy==0.6.0 from https://files.pythonhosted.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl#sha256=e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5 (from -r /tmp/pipenv-03jpv10e-requirements/pipenv-3k4kv96o-requirement.txt (line 1)):', '        Expected sha256 7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff', '             Got        e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5']
ERROR: ERROR: Package installation failed...

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: 'c:\\python38_64\\lib\\site-packages\\pipenv'

Python location: 'c:\\python38_64\\python.exe'

Python installations found:

  • 3.8.0: C:\Python38_64\python.exe
  • 3.7.4: C:\Python37\python.exe
  • 3.7.4: C:\Python37-32\python.exe
  • 3.6.5: C:\Python36_64\python.exe
  • 2.7: C:\Python27\python.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.0',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.18362',
 'python_full_version': '3.8.0',
 'python_version': '3.8',
 'sys_platform': 'win32'}

System environment variables:

  • ALLUSERSPROFILE
  • ANSICON
  • ANSICON_DEF
  • APPDATA
  • CAMLIBS
  • CHOCOLATEYINSTALL
  • CHOCOLATEYLASTPATHUPDATE
  • COMMONPROGRAMFILES
  • COMMONPROGRAMFILES(X86)
  • COMMONPROGRAMW6432
  • COMPUTERNAME
  • COMSPEC
  • CONEMUANSI
  • CONEMUANSILOG
  • CONEMUARGS
  • CONEMUARGS2
  • CONEMUBACKHWND
  • CONEMUBASEDIR
  • CONEMUBUILD
  • CONEMUCONFIG
  • CONEMUDIR
  • CONEMUDRAWHWND
  • CONEMUDRIVE
  • CONEMUHOOKS
  • CONEMUHWND
  • CONEMUPALETTE
  • CONEMUPID
  • CONEMUPROMPT0
  • CONEMUPROMPT1
  • CONEMUPROMPT2
  • CONEMUPROMPT3
  • CONEMUSERVERPID
  • CONEMUTASK
  • CONEMUWORKDIR
  • CONEMUWORKDRIVE
  • DRIVERDATA
  • HOMEDRIVE
  • HOMEPATH
  • IOLIBS
  • LOCALAPPDATA
  • LOGONSERVER
  • MAGICK_HOME
  • MOZ_PLUGIN_PATH
  • NUMBER_OF_PROCESSORS
  • ONEDRIVE
  • ONEDRIVECONSUMER
  • OS
  • PATH
  • PATHEXT
  • PROCESSOR_ARCHITECTURE
  • PROCESSOR_IDENTIFIER
  • PROCESSOR_LEVEL
  • PROCESSOR_REVISION
  • PROGRAMDATA
  • PROGRAMFILES
  • PROGRAMFILES(X86)
  • PROGRAMW6432
  • PROMPT
  • PSMODULEPATH
  • PUBLIC
  • SESSIONNAME
  • SYSTEMDRIVE
  • SYSTEMROOT
  • TEMP
  • TMP
  • USERDOMAIN
  • USERDOMAIN_ROAMINGPROFILE
  • USERNAME
  • USERPROFILE
  • VBOX_MSI_INSTALL_PATH
  • WINDIR
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenvûspecific environment variables:

Debugûspecific environment variables:

  • PATH: "C:\Program Files\Tesseract-OCR"\;C:\Program Files\ConEmu\ConEmu\Scripts;C:\Program Files\ConEmu;C:\Program Files\ConEmu\ConEmu;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Docker\Docker\Resources\bin;C:\Program Files\ImageMagick-7.0.8-Q16;C:\Program Files (x86)\ImageMagick-7.0.8-Q16-HDRI;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\ImageMagick-7.0.7-Q16-HDRI;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseGit\bin;C:\Program Files\Git\cmd;C:\Program Files\Calibre2\;C:\Program Files\nodejs\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\TortoiseHg\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\CMake\bin;E:\toolbuild\sK1_Project\UniConvertor-1.1.5\;E:\toolbuild\sK1_Project\UniConvertor-1.1.5\DLLs;C:\Program Files\doxygen\bin;C:\ProgramData\chocolatey\bin;C:\Python38_64\Scripts\;C:\Python38_64\;C:\python36_64\Scripts;C:\python36_64;C:\Users\Gadget\AppData\Local\Microsoft\WindowsApps;C:\Users\Gadget\AppData\Roaming\npm;C:\Users\Gadget\AppData\Local\Programs\MiKTeX 2.9\miktex\bin\x64\;C:\Users\Gadget\AppData\Local\Pandoc\;C:\Program Files (x86)\Nmap;C:\Users\Gadget\AppData\Local\GitHubCLI\bin;C:\Users\Gadget\AppData\Local\Microsoft\WindowsApps;;c:\python38_64\lib\site-packages\pywin32_system32

triage
Source
picture GadgetSteve

Most helpful comment

Thanks @rmbrad , so I can conclude that this issue was not caused by Pipenv.

Any more thoughts @GadgetSteve ?

picture frostming on 13 Nov 2019
👍2

All 9 comments

Sorry, I can't get what the point of the OP is. It looks to me that the lock file should be updated with the new hashes. The cause is because the package author has re-updated the artifact and the hash changes.

frostming picture frostming on 22 Oct 2019

The point is that the package author has specified any version of the dependencies is suitable with * but at the same time that the package needs to have specific SHA-256 values - these are mutually contradictory. Worse there is no way to tell which package versions to install to get those values.

If the required version is * then the lock values should be ignored or auto-update!

GadgetSteve picture GadgetSteve on 22 Oct 2019

@GadgetSteve your link An example of this happening: psf/requests-html#338 points to a pull request where the changes don't include any dependencies.

Perhaps if you included a Pipfile and Pipfile.lock or snippets therefrom, your concern would be easier to see?

Here's what I get if I include "any version" of a dependency:

> grep -C 6 requests Pipfile*
Pipfile-[requires]
Pipfile-python_version = "3.7"
Pipfile:requests = "*"

Pipfile.lock-        "requires": {
Pipfile.lock-            "python_version": "3.7",
Pipfile.lock:            "requests": "*"
Pipfile.lock-        },

Pipfile.lock:        "requests": {
Pipfile.lock-            "hashes": [
Pipfile.lock-                "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",
Pipfile.lock-                "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"
Pipfile.lock-            ],
Pipfile.lock-            "version": "==2.22.0"
Pipfile.lock-        },

Looks like pipenv does the right thing － if there's * in Pipfile, then Pipfile.lock gets a specific version 2.22.0 and a set of hashes.

dimaqq picture dimaqq on 8 Nov 2019

Hi Dima,

My problem occurred when I was committing a documentation only change to a project that I do not own or otherwise maintain. The original author specified that at least some of the dependencies were to be latest, i.e. “*” but the Pipfile.lock for those dependencies like yours below has hashes for the latest version dependencies – as a result the Travis build fails because the current latest version of the dependencies does not match that at the time that the Pipfile.lock was generated.

This behaviour means that any project with dependencies marked ‘*’ the Pipfile.lock will be incorrect and need to be regenerated every time that the external dependency is updated.

The closes analogy that I can come up with is telling people that they have to pay 25 euros (and that any denomination of notes and other currencies are acceptable) and at the same time specifying the exact serial numbers of the notes to be used.

I feel that this places an unreasonable maintenance burden on the package authors and a hurdle for contributors if any packages are marked as “latest” as happened in my case – a simple documentation change has been blocked due to an external dependency having been updated.

The Pipfile for the project in question:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = ""
pyquery = ""
fake-useragent = ""
parse = ""
"bs4" = ""
"w3lib" = ""
pyppeteer = ""
"rfc3986" = ""
requests-html = {path = "."}
lxml = {path = "c:/Users/Gadget/Downloads/lxml-4.4.1-cp38-cp38-win_amd64.whl"}

[dev-packages]
twine = ""
requests-file = ""
pytest = ""
e1839a8 = {path = ".",editable = true}
sphinx = ""
mypy = ""
pytest-asyncio = ""
psutil = ""
white = ""

[scripts]
tests = "pytest -v -m ok"

Which does not include the failing package pluggy which is a dependency of pytest which is a dependency for pytest-asyncio (so 3rd hand)

But Pifile.lock has:
"pytest": {
"hashes": [
"sha256:062027955bccbc04d2fcd5d79690947e018ba31abe4c90b2c6721abec734261b",
"sha256:117bad36c1a787e1a8a659df35de53ba05f9f3398fb9e4ac17e80ad5903eb8c5"
],
"index": "pypi",
"version": "==3.4.2"
},
"pytest-asyncio": {
"hashes": [
"sha256:286b50773e996c80d894b95afaf45df6952408a67a59979ca9839f94693ec7fd",
"sha256:f32804bb58a66e13a3eda11f8942a71b1b6a30466b0d2ffe9214787aab0e172e"
],
"index": "pypi",
"version": "==0.8.0"
},

"pluggy": {
"hashes": [
"sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff"
],
"version": "==0.6.0"
},

However the current versions are:
pytest-asyncio: Version: 0.10.0
pytest: Version: 5.2.2
pluggy: Version: 0.13.0

Hope that clarifies things.

Steve

From: Dima Tisnek notifications@github.com
Sent: 08 November 2019 06:59
To: pypa/pipenv pipenv@noreply.github.com
Cc: Steve (Gadget) Barnes gadgetsteve@hotmail.com; Mention mention@noreply.github.com
Subject: Re: [pypa/pipenv] Outlaw packages with version =* from the lock file (#3997)

@GadgetStevehttps://github.com/GadgetSteve your link An example of this happening: psf/requests-html#338 points to a pull request where the changes don't include any dependencies.

Perhaps if you included a Pipfile and Pipfile.lock or snippets therefrom, your concern would be easier to see?

Here's what I get if I include "any version" of a dependency:

grep -C 6 requests Pipfile*

Pipfile-[requires]

Pipfile-python_version = "3.7"

Pipfile:requests = "*"

Pipfile.lock- "requires": {

Pipfile.lock- "python_version": "3.7",

Pipfile.lock: "requests": "*"

Pipfile.lock- },

Pipfile.lock: "requests": {

Pipfile.lock- "hashes": [

Pipfile.lock- "sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4",

Pipfile.lock- "sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31"

Pipfile.lock- ],

Pipfile.lock- "version": "==2.22.0"

Pipfile.lock- },

Looks like pipenv does the right thing － if there's * in Pipfile, then Pipfile.lock gets a specific version 2.22.0 and a set of hashes.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pypa/pipenv/issues/3997?email_source=notifications&email_token=ABKVUWWS7PLU5BKQHFJAWMDQSUE3PA5CNFSM4JCYHTMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDO5S3Q#issuecomment-551410030, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKVUWR2R7UEYSC4MOEAS7DQSUE3PANCNFSM4JCYHTMA.

GadgetSteve picture GadgetSteve on 9 Nov 2019

The lock file in the referenced psf/requests-html project only includes the hash for the source tarball of pluggy==0.6.0, but on Travis CI it is attempting to install the Python 3 binary wheel leading to the error about mismatched hashes.

Creating the environment locally using the following commands works as expected (only showing relevant output:)

> git clone [email protected]:psf/requests-html.git
Cloning into 'requests-html'...
...
> cd requests-html
> pipenv --python 3.6.7 # Same python version used in Travis CI
Creating a virtualenv for this project…
...
> pipenv install --dev --verbose
Installing dependencies from Pipfile.lock (7312a6)…
...
Found link https://files.pythonhosted.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl#sha256=e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5 (from https://pypi.org/simple/pluggy/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.6.0
  Found link https://files.pythonhosted.org/packages/11/bf/cbeb8cdfaffa9f2ea154a30ae31a9d04a1209312e2919138b4171a1f8199/pluggy-0.6.0.tar.gz#sha256=7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff (from https://pypi.org/simple/pluggy/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*), version: 0.6.0
...
Checked 2 links for project 'pluggy' against 1 hashes (1 matches, 0 no digest): discarding 1 non-matches:
  https://files.pythonhosted.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl#sha256=e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5 (from https://pypi.org/simple/pluggy/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*)
Using version 0.6.0 (newest of versions: 0.6.0)
Collecting pluggy==0.6.0
...

As can be seen in output above both the wheel and the source tar ball is found, and the source is used since it has the matching hash. If you set the PIP_ONLY_BINARY environment variable to :all: the source tarball is ignored and it will fail locally the same as it does within Travis CI.

So, either the Pipfile.lock needs to be updated to include the hash for the wheel so it can be installed within Travis CI; or the Travis CI build environment needs to be configured to be able to install using the source tarball (I'm not that familiar with Travis CI so not sure what this entails.) But this is not an issue with Pipenv.

rmbrad picture rmbrad on 13 Nov 2019

Awesome investigation, @rmbrad

I wonder if pipenv purposefully keeps only the hash of the file it used during lock.

Or was it the case that when deps were locked, tarball was published but wheel was not?

I think (but need to verify) that other systems stamp all sources for given version and pipenv does too: https://github.com/oberdev/cgbot/blob/a48c79b42bcd4a2d91981bd91f3bbef25f1d6c53/Pipfile.lock#L35 where hashes for wheels for all platforms were stamped...

dimaqq picture dimaqq on 13 Nov 2019

@dimaqq Thanks

I wonder if pipenv purposefully keeps only the hash of the file it used during lock.

I believe current versions of Pipenv should store hashes for all versions/platforms.

Or was it the case that when deps were locked, tarball was published but wheel was not?

Yeah, I'm pretty sure this is the case. The Pipfile.lock was last updated March 21, 2018, the wheels were added to PyPi on April 15, 2018.

rmbrad picture rmbrad on 13 Nov 2019
👍1

Thanks @rmbrad , so I can conclude that this issue was not caused by Pipenv.

Any more thoughts @GadgetSteve ?

frostming picture frostming on 13 Nov 2019
👍2

As Dima said – an awesome investigation from @rmbradhttps://github.com/rmbrad

Looking at the build error message (https://travis-ci.com/psf/requests-html/builds/132282995) and expanding the install.1 tab I see that the pipenv version is 2018.11.26 – nearly a year ago but the latest release so an outdated pipenv (and the pip install pipenv --upgrade-strategy=only-if-needed install) is probably not at fault.

The fact that even on the retry the cached wheel is being used (see line 208 of the build). My guess is that when pipenv is looking for alternatives is does not issue the --no-cache-dir instruction to pip so if there is a single cached copy that comes back as the only option.

Is that a possibility for why this build fails?

Steve

From: Frost Ming notifications@github.com
Sent: 13 November 2019 04:08
To: pypa/pipenv pipenv@noreply.github.com
Cc: Steve (Gadget) Barnes gadgetsteve@hotmail.com; Mention mention@noreply.github.com
Subject: Re: [pypa/pipenv] Outlaw packages with version =* from the lock file (#3997)

Thanks @rmbradhttps://github.com/rmbrad , so I can conclude that this issue was not caused by Pipenv.

Any more thoughts @GadgetStevehttps://github.com/GadgetSteve ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pypa/pipenv/issues/3997?email_source=notifications&email_token=ABKVUWTXNIUAMSXZVWXM32TQTN4QTA5CNFSM4JCYHTMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED4Z3HI#issuecomment-553229725, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKVUWQWTLJG5ETE43XSS6LQTN4QTANCNFSM4JCYHTMA.

GadgetSteve picture GadgetSteve on 14 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

johnjiang picture johnjiang  ·  3Comments
Californian picture Californian  ·  3Comments
xi picture xi  ·  3Comments
jacebrowning picture jacebrowning  ·  3Comments
ipmb picture ipmb  ·  3Comments