Pipenv: Pipfile.lock doesn't preserve some `extras` information
I also filed a possibly related issue here https://github.com/kennethreitz/pipenv/issues/1013
Environment
- Linux
- pipenv is running on python 3.4.3
- Pipenv version = 8.3.1
Expected result
Should be able to reinstall extra dependencies from a Pipfile.lock
In this example raven[flask] in python 2 depends on Blinker and Flask because of the flask extra, and depends on contextlib2 because its python < 3.4; so these dependencies should be installed.
https://github.com/getsentry/raven-python/blob/master/setup.py
Actual result
pipenv install raven[flask] works as expected but after recreating the virtual environment, installing from Pipfile.lock doesn't install all extra dependencies and some extra dependencies are absent from the Pipfile.lock.
Steps to replicate
Showing pipenv install raven[flask] installs all expected packages
chris $ pipenv --python 2.7
Virtualenv already exists!
Removing existing virtualenvβ¦
Creating a virtualenv for this projectβ¦
Using /my/version/of/python2.7 to create virtualenvβ¦
β Running virtualenv with interpreter /my/version/of/python2.7
New python executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python2.7
Also creating executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /home/chris/.local/share/virtualenvs/demos-8_UOIBMV
Creating a Pipfile for this projectβ¦
chris $ pipenv install raven[flask]
Installing raven[flask]β¦
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββCollecting raven[flask]
Using cached raven-6.3.0-py2.py3-none-any.whl
Collecting contextlib2; python_version < "3.2" (from raven[flask])
Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Collecting blinker>=1.1; extra == "flask" (from raven[flask])
Collecting Flask>=0.8; extra == "flask" (from raven[flask])
Using cached Flask-0.12.2-py2.py3-none-any.whl
Collecting itsdangerous>=0.21 (from Flask>=0.8; extra == "flask"->raven[flask])
Collecting Jinja2>=2.4 (from Flask>=0.8; extra == "flask"->raven[flask])
Using cached Jinja2-2.9.6-py2.py3-none-any.whl
Collecting Werkzeug>=0.7 (from Flask>=0.8; extra == "flask"->raven[flask])
Using cached Werkzeug-0.12.2-py2.py3-none-any.whl
Collecting click>=2.0 (from Flask>=0.8; extra == "flask"->raven[flask])
Using cached click-6.7-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask>=0.8; extra == "flask"->raven[flask])
Installing collected packages: contextlib2, blinker, itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask, raven
Successfully installed Flask-0.12.2 Jinja2-2.9.6 MarkupSafe-1.0 Werkzeug-0.12.2 blinker-1.4 click-6.7 contextlib2-0.5.5 itsdangerous-0.24 raven-6.3.0
Adding raven[flask] to Pipfile's [packages]β¦
Locking [dev-packages] dependenciesβ¦
Locking [packages] dependenciesβ¦
Updated Pipfile.lock (812aac)!
chris $ pipenv graph
blinker==1.4
Flask==0.12.2
- click [required: >=2.0, installed: 6.7]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.9.6]
- MarkupSafe [required: >=0.23, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.12.2]
raven==6.3.0
- contextlib2 [required: Any, installed: 0.5.5]
chris $
Showing pipenv install doesn't install some extra dependencies
chris $ pipenv --python 2.7
Virtualenv already exists!
Removing existing virtualenvβ¦
Creating a virtualenv for this projectβ¦
Using /my/version/of/python2.7 to create virtualenvβ¦
β Running virtualenv with interpreter /my/version/of/python2.7
New python executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python2.7
Also creating executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /home/chris/.local/share/virtualenvs/demos-8_UOIBMV
chris $ pipenv install
Installing dependencies from Pipfile.lock (812aac)β¦
π ββββββββββββββββββββββββββββββββ 8/8 β 00:00:02
To activate this project's virtualenv, run the following:
$ pipenv shell
chris $ pipenv graph
blinker==1.4
Flask==0.12.2
- click [required: >=2.0, installed: 6.7]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.9.6]
- MarkupSafe [required: >=0.23, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.12.2]
raven==6.3.0
- contextlib2 [required: Any, installed: ?]
chris $
Showing pipenv install --skip-lock installs expected dependencies but pipinstall upgrade uninstalls some.
chris $ pipenv --python 2.7
Virtualenv already exists!
Removing existing virtualenvβ¦
Creating a virtualenv for this projectβ¦
Using /my/version/of/python2.7 to create virtualenvβ¦
β Running virtualenv with interpreter /my/version/of/python2.7
New python executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python2.7
Also creating executable in /home/chris/.local/share/virtualenvs/demos-8_UOIBMV/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /home/chris/.local/share/virtualenvs/demos-8_UOIBMV
chris $ pipenv install --skip-lock
Installing dependencies from Pipfileβ¦
π ββββββββββββββββββββββββββββββββ 1/1 β 00:00:05
To activate this project's virtualenv, run the following:
$ pipenv shell
chris $ pipenv graph
blinker==1.4
Flask==0.12.2
- click [required: >=2.0, installed: 6.7]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.9.6]
- MarkupSafe [required: >=0.23, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.12.2]
raven==6.3.0
- contextlib2 [required: Any, installed: 0.5.5]
chris $ pipenv update
Updating all dependencies from Pipfileβ¦
Found 9 installed package(s), purgingβ¦
Uninstalling blinker-1.4:
Successfully uninstalled blinker-1.4
Uninstalling click-6.7:
Successfully uninstalled click-6.7
Uninstalling contextlib2-0.5.5:
Successfully uninstalled contextlib2-0.5.5
Uninstalling Flask-0.12.2:
Successfully uninstalled Flask-0.12.2
Uninstalling itsdangerous-0.24:
Successfully uninstalled itsdangerous-0.24
Uninstalling Jinja2-2.9.6:
Successfully uninstalled Jinja2-2.9.6
Uninstalling MarkupSafe-1.0:
Successfully uninstalled MarkupSafe-1.0
Uninstalling raven-6.3.0:
Successfully uninstalled raven-6.3.0
Uninstalling Werkzeug-0.12.2:
Successfully uninstalled Werkzeug-0.12.2
Environment now purged and fresh!
Locking [dev-packages] dependenciesβ¦
Locking [packages] dependenciesβ¦
Updated Pipfile.lock (812aac)!
Installing dependencies from Pipfile.lock (812aac)β¦
π ββββββββββββββββββββββββββββββββ 8/8 β 00:00:02
To activate this project's virtualenv, run the following:
$ pipenv shell
All dependencies are now up-to-date!
chris $ pipenv graph
blinker==1.4
Flask==0.12.2
- click [required: >=2.0, installed: 6.7]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.9.6]
- MarkupSafe [required: >=0.23, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.12.2]
raven==6.3.0
- contextlib2 [required: Any, installed: ?]
chris $
Pipfile
[[source]]
name = "pypi"
url = "https://pypi.python.org/simple"
verify_ssl = true
[dev-packages]
[packages]
raven = {extras = ["flask"]}
[requires]
python_version = "2.7"
Pipfile.lock
{
"_meta": {
"hash": {
"sha256": "79f6cd7950e9c9c7dfc51d1b941d4faa0a554593b2af880134c66358f5812aac"
},
"host-environment-markers": "REDACTED",
"pipfile-spec": 6,
"requires": {
"python_version": "2.7"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"blinker": {
"hashes": [
"sha256:471aee25f3992bd325afa3772f1063dbdbbca947a041b8b89466dc00d606f8b6"
],
"version": "==1.4"
},
"click": {
"hashes": [
"sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d",
"sha256:f15516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b"
],
"version": "==6.7"
},
"flask": {
"hashes": [
"sha256:0749df235e3ff61ac108f69ac178c9770caeaccad2509cb762ce1f65570a8856",
"sha256:49f44461237b69ecd901cc7ce66feea0319b9158743dd27a2899962ab214dac1"
],
"version": "==0.12.2"
},
"itsdangerous": {
"hashes": [
"sha256:cbb3fcf8d3e33df861709ecaf89d9e6629cff0a217bc2848f1b41cd30d360519"
],
"version": "==0.24"
},
"jinja2": {
"hashes": [
"sha256:2231bace0dfd8d2bf1e5d7e41239c06c9e0ded46e70cc1094a0aa64b0afeb054",
"sha256:ddaa01a212cd6d641401cb01b605f4a4d9f37bfc93043d7f760ec70fb99ff9ff"
],
"version": "==2.9.6"
},
"markupsafe": {
"hashes": [
"sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665"
],
"version": "==1.0"
},
"raven": {
"hashes": [
"sha256:cb644fb12ee886a341041dcd533540dfc82619a50bf0b7c587af070054bd2c7f",
"sha256:f3e465a545dcdb6a387d1fcb199d08f786ba3732d7ce6aa681718b04da6aedf1"
],
"version": "==6.3.0"
},
"werkzeug": {
"hashes": [
"sha256:e8549c143af3ce6559699a01e26fa4174f4c591dbee0a499f3cd4c3781cdec3d",
"sha256:903a7b87b74635244548b30d30db4c8947fe64c5198f58899ddcd3a13c23bb26"
],
"version": "==0.12.2"
}
},
"develop": {}
}
macbeth322
All 12 comments
Thanks for the full report @Grets, that was really helpful π
FYI, this is not related to #1013, even if it might look like it. Good thinking, but those a really distinct issues, I can assure it.
Now, for this issue:
From what I can see, the "extra" part itself is respected: Flask and Blinker are installed everytime. That's some good news.
The issue occurs with contextlib2, which has a condition on the python version. We've seen a lot of issues related to python-version-specific dependencies.
And I'm pretty sure this is exactly the issue here, here what happens:
pipenvruns with python 3.4.3- Your virtualenv runs with python 2.7
- The first install you did (which is performed pre-lock), use the venv python/pip, effectively 2.7
- The lock that was performed in that same install used the "root" python/pip, which is 3.4.3, excluding
contextlib2as its not a Python 3 dependency. - When you did the re-install from the lockfile, it used the lock mentioned above, which did not include
contextlib2, which is why it wasn't installed (the packages are essentially installed one-by-one withpip install --no-deps, as the dependency should all be in the lockfile, in theory, but not in this case due to the lock bug) - Doing
pipenv install --skip-lockgoes through without issue because it correctly uses the python/pip from the virtualenv, and process the dependency likepip(we don't put the--no-depsflag when doing--skip-lock, as we don't use the lockfile).
So yeah, that's the known issue of "wrong python version used for locking" we've been hunting for a while.
But seriously, @Grets, you have no idea how much that report just helped me out figuring a few things out. This was a known issue, but you gave me enough information to clear up a bunch of things. Thank you!
One additional thing I would ask, to get rid of that 0.1% of doubt I have about the cause of the issue: With the exact same setup, could you redo all of this, and at the end, do pipenv lock --clear --verbose and give the output as well as the resulting Pipfile.lock? That would be greatly appreciated.
vphilippon
on 1 Nov 2017
Thanks for the response @vphilippon, I went ahead and ran the pipenv lock --clear --verbose and I will put the results at the bottom of this. I also reinstalled my version of pipenv using python2.7 and retried all the steps I did before and now it seems to work. It generates the exact same Pipfile, but the Pipfile.lock now has the entry for contextlib2 with the marker for python_version < 3.2. Thanks again for your help, and I am glad I was able to provide some insight.
pipenv lock --clear --verbose output using pipenv with python3.4
Using pip: -i https://pypi.python.org/simple
ROUND 1
Current constraints:
Finding the best candidates:
Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done
Using pip: -i https://pypi.python.org/simple
ROUND 1
Current constraints:
raven[flask]
Finding the best candidates:
found candidate raven[flask]==6.3.0 (constraint was <any>)
Finding secondary dependencies:
New dependencies found in this round:
adding ['blinker', '>=1.1', '[]']
adding ['flask', '>=0.8', '[]']
adding ['raven', '==6.3.0', "['flask']"]
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable
ROUND 2
Current constraints:
blinker>=1.1; extra == "flask"
Flask>=0.8; extra == "flask"
raven[flask]==6.3.0
Finding the best candidates:
found candidate blinker==1.4 (constraint was >=1.1)
found candidate flask==0.12.2 (constraint was >=0.8)
found candidate raven[flask]==6.3.0 (constraint was ==6.3.0)
Finding secondary dependencies:
New dependencies found in this round:
adding ['click', '>=2.0', '[]']
adding ['itsdangerous', '>=0.21', '[]']
adding ['jinja2', '>=2.4', '[]']
adding ['werkzeug', '>=0.7', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable
ROUND 3
Current constraints:
blinker>=1.1; extra == "flask"
click>=2.0
Flask>=0.8; extra == "flask"
itsdangerous>=0.21
Jinja2>=2.4
raven[flask]==6.3.0
Werkzeug>=0.7
Finding the best candidates:
found candidate blinker==1.4 (constraint was >=1.1)
found candidate click==6.7 (constraint was >=2.0)
found candidate flask==0.12.2 (constraint was >=0.8)
found candidate itsdangerous==0.24 (constraint was >=0.21)
found candidate jinja2==2.9.6 (constraint was >=2.4)
found candidate raven[flask]==6.3.0 (constraint was ==6.3.0)
found candidate werkzeug==0.12.2 (constraint was >=0.7)
Finding secondary dependencies:
jinja2==2.9.6 not in cache, need to check index
jinja2==2.9.6 requires Jinja2==2.9.6, MarkupSafe>=0.23
click==6.7 not in cache, need to check index
click==6.7 requires click==6.7
itsdangerous==0.24 not in cache, need to check index
itsdangerous==0.24 requires itsdangerous==0.24
werkzeug==0.12.2 not in cache, need to check index
werkzeug==0.12.2 requires Werkzeug==0.12.2
New dependencies found in this round:
adding ['click', '==6.7,>=2.0', '[]']
adding ['itsdangerous', '==0.24,>=0.21', '[]']
adding ['jinja2', '==2.9.6,>=2.4', '[]']
adding ['markupsafe', '>=0.23', '[]']
adding ['werkzeug', '==0.12.2,>=0.7', '[]']
Removed dependencies in this round:
removing ['click', '>=2.0', '[]']
removing ['itsdangerous', '>=0.21', '[]']
removing ['jinja2', '>=2.4', '[]']
removing ['werkzeug', '>=0.7', '[]']
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 3: not stable
ROUND 4
Current constraints:
blinker>=1.1; extra == "flask"
click==6.7,>=2.0
Flask>=0.8; extra == "flask"
itsdangerous==0.24,>=0.21
Jinja2==2.9.6,>=2.4
MarkupSafe>=0.23
raven[flask]==6.3.0
Werkzeug==0.12.2,>=0.7
Finding the best candidates:
found candidate blinker==1.4 (constraint was >=1.1)
found candidate click==6.7 (constraint was >=2.0,==6.7)
found candidate flask==0.12.2 (constraint was >=0.8)
found candidate itsdangerous==0.24 (constraint was >=0.21,==0.24)
found candidate jinja2==2.9.6 (constraint was >=2.4,==2.9.6)
found candidate markupsafe==1.0 (constraint was >=0.23)
found candidate raven[flask]==6.3.0 (constraint was ==6.3.0)
found candidate werkzeug==0.12.2 (constraint was ==0.12.2,>=0.7)
Finding secondary dependencies:
markupsafe==1.0 not in cache, need to check index
markupsafe==1.0 requires MarkupSafe==1.0
jinja2==2.9.6 requires Jinja2==2.9.6, MarkupSafe>=0.23
werkzeug==0.12.2 requires Werkzeug==0.12.2
click==6.7 requires click==6.7
itsdangerous==0.24 requires itsdangerous==0.24
New dependencies found in this round:
adding ['markupsafe', '==1.0,>=0.23', '[]']
Removed dependencies in this round:
removing ['markupsafe', '>=0.23', '[]']
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 4: not stable
ROUND 5
Current constraints:
blinker>=1.1; extra == "flask"
click==6.7,>=2.0
Flask>=0.8; extra == "flask"
itsdangerous==0.24,>=0.21
Jinja2==2.9.6,>=2.4
MarkupSafe==1.0,>=0.23
raven[flask]==6.3.0
Werkzeug==0.12.2,>=0.7
Finding the best candidates:
found candidate blinker==1.4 (constraint was >=1.1)
found candidate click==6.7 (constraint was >=2.0,==6.7)
found candidate flask==0.12.2 (constraint was >=0.8)
found candidate itsdangerous==0.24 (constraint was >=0.21,==0.24)
found candidate jinja2==2.9.6 (constraint was >=2.4,==2.9.6)
found candidate markupsafe==1.0 (constraint was >=0.23,==1.0)
found candidate raven[flask]==6.3.0 (constraint was ==6.3.0)
found candidate werkzeug==0.12.2 (constraint was ==0.12.2,>=0.7)
Finding secondary dependencies:
werkzeug==0.12.2 requires Werkzeug==0.12.2
click==6.7 requires click==6.7
itsdangerous==0.24 requires itsdangerous==0.24
jinja2==2.9.6 requires Jinja2==2.9.6, MarkupSafe>=0.23
markupsafe==1.0 requires MarkupSafe==1.0
------------------------------------------------------------
Result of round 5: stable, done
Pipfile.lock generated from the same Pipfile as in my first comment but using pipenv with python2.7 this time and everything is as expected
{
"_meta": {
"hash": {
"sha256": "79f6cd7950e9c9c7dfc51d1b941d4faa0a554593b2af880134c66358f5812aac"
},
"host-environment-markers": "REDACTED",
"pipfile-spec": 6,
"requires": {
"python_version": "2.7"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"blinker": {
"hashes": [
"sha256:471aee25f3992bd325afa3772f1063dbdbbca947a041b8b89466dc00d606f8b6"
],
"version": "==1.4"
},
"click": {
"hashes": [
"sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d",
"sha256:f15516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b"
],
"version": "==6.7"
},
"contextlib2": {
"hashes": [
"sha256:f5260a6e679d2ff42ec91ec5252f4eeffdcf21053db9113bd0a8e4d953769c00",
"sha256:509f9419ee91cdd00ba34443217d5ca51f5a364a404e1dce9e8979cea969ca48"
],
"markers": "python_version < '3.2'",
"version": "==0.5.5"
},
"flask": {
"hashes": [
"sha256:0749df235e3ff61ac108f69ac178c9770caeaccad2509cb762ce1f65570a8856",
"sha256:49f44461237b69ecd901cc7ce66feea0319b9158743dd27a2899962ab214dac1"
],
"version": "==0.12.2"
},
"itsdangerous": {
"hashes": [
"sha256:cbb3fcf8d3e33df861709ecaf89d9e6629cff0a217bc2848f1b41cd30d360519"
],
"version": "==0.24"
},
"jinja2": {
"hashes": [
"sha256:2231bace0dfd8d2bf1e5d7e41239c06c9e0ded46e70cc1094a0aa64b0afeb054",
"sha256:ddaa01a212cd6d641401cb01b605f4a4d9f37bfc93043d7f760ec70fb99ff9ff"
],
"version": "==2.9.6"
},
"markupsafe": {
"hashes": [
"sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665"
],
"version": "==1.0"
},
"raven": {
"hashes": [
"sha256:cb644fb12ee886a341041dcd533540dfc82619a50bf0b7c587af070054bd2c7f",
"sha256:f3e465a545dcdb6a387d1fcb199d08f786ba3732d7ce6aa681718b04da6aedf1"
],
"version": "==6.3.0"
},
"werkzeug": {
"hashes": [
"sha256:e8549c143af3ce6559699a01e26fa4174f4c591dbee0a499f3cd4c3781cdec3d",
"sha256:903a7b87b74635244548b30d30db4c8947fe64c5198f58899ddcd3a13c23bb26"
],
"version": "==0.12.2"
}
},
"develop": {}
}
Also, the Pipfile.lock generated using pipenv lock --clear --verbose using pipenv with python3.4 was the same Pipfile.lock that I posted in my original comment.
macbeth322
on 1 Nov 2017
@macbeth322 Allright, that confirms it, thanks!
vphilippon
on 1 Nov 2017
I believe I stumbled upon a similar issue. I'm trying to deploy my app to docker with pipenv installing requirements. I added an extras package yesterday and now none of the dependencies to airflow exist in my lock file (notably dill). Correct me if I'm wrong, but the following should never happen, correct?
"default": {
"airflow": {
"extras": [
"crypto"
]
},
I can send you my Pipfile, Pipfile.lock, and/or other files if they would be relevant to the issue at hand.
sethwoodworth
on 8 Nov 2017
@sethwoodworth I only really have one question about this -- in the root directory of your project, do you have a subdirectory called airflow? See this for context: https://github.com/kennethreitz/pipenv/issues/1045#issuecomment-342901237
techalchemy
on 9 Nov 2017
😄1
@techalchemy Yes! I have precisely that. A folder named airflow and a folder named alembic as the case might be. I can move the folders back to their current values when PR #958 is merged.
sethwoodworth
on 9 Nov 2017
👍1
Nailed it! Seen this one enough by now to know right away whatβs up there
techalchemy
on 9 Nov 2017
🎉1
This is fixed in #958
techalchemy
on 19 Nov 2017
958 fixes the relative path issue, but does it also fix the correct python version being used in lock?
extras_require={
...
':python_version<"3.2"': ['contextlib2'],
},
helderco
on 23 Nov 2017
This is a dupe of #857, though.
helderco
on 23 Nov 2017
@helderco the local file thing and the python version thing are separate. The former is fixed, but dependency resolution with regard to the latter isnβt fixed yet
techalchemy
on 23 Nov 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
erinxocon
Β·
3Comments
leileigong
Β·
3Comments
jakul
Β·
3Comments
hynek
Β·
3Comments
randName
Β·
3Comments