Pipenv: Dependency resolution order difference between pip and pipenv

[c17r]

Took me awhile to track down what I was seeing as well as come up with the smallest test example.

So if you have a requirement of pytest-django<2.10 which in it's setup.py has an install_requires entry of pytest>=2.5. You also have a requirement of pytest<2.10.

Pip on it's own handles this just fine. Pipenv ends up with the pytest at the current release version.

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:51:42] $ ll
.
[-rw-r--r-- christian staff     121 Mar 29 18:29]  Pipfile
[-rw-r--r-- christian staff      31 Mar 29 18:51]  req.txt

0 directories, 2 files

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:51:43] $ cat Pipfile 
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
pytest = "<2.10"
pytest-django = "<2.10"

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:51:44] $ cat req.txt 
pytest<2.10
pytest-django<2.10

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:51:46] $ pip freeze

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:52:27] $ pip install -r req.txt
Collecting pytest<2.10 (from -r req.txt (line 1))
  Using cached pytest-2.9.2-py2.py3-none-any.whl
Collecting pytest-django<2.10 (from -r req.txt (line 2))
  Using cached pytest_django-2.9.1-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest<2.10->-r req.txt (line 1))
  Using cached py-1.4.33-py2.py3-none-any.whl
Installing collected packages: py, pytest, pytest-django
Successfully installed py-1.4.33 pytest-2.9.2 pytest-django-2.9.1

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:52:35] $ pip freeze
py==1.4.33
pytest==2.9.2
pytest-django==2.9.1

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:52:39] $ pip freeze

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:52:55] $ pipenv install
No package provided, installing all dependencies.
Pipfile found at /Users/christian/tmp/test_3/Pipfile. Considering this to be the project home.
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
⠹Locking [packages] dependencies...
⠦Updated Pipfile.lock!
Installing dependencies from Pipfile.lock...
Collecting appdirs==1.4.3 
  Using cached appdirs-1.4.3-py2.py3-none-any.whl
Collecting packaging==16.8 
  Using cached packaging-16.8-py2.py3-none-any.whl
Collecting py==1.4.33 
  Using cached py-1.4.33-py2.py3-none-any.whl
Collecting pyparsing==2.2.0 
  Using cached pyparsing-2.2.0-py2.py3-none-any.whl
Collecting pytest==3.0.7 
  Using cached pytest-3.0.7-py2.py3-none-any.whl
Collecting pytest-django==2.9.1 
  Using cached pytest_django-2.9.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools==34.3.3 in /Users/christian/.virtualenvs/test_3-9pMWYqV9/lib/python3.6/site-packages 
Collecting six==1.10.0 
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: appdirs, six, pyparsing, packaging, py, pytest, pytest-django
Successfully installed appdirs-1.4.3 packaging-16.8 py-1.4.33 pyparsing-2.2.0 pytest-3.0.7 pytest-django-2.9.1 six-1.10.0

⠙

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:53:17] $ pip freeze
appdirs==1.4.3
packaging==16.8
py==1.4.33
pyparsing==2.2.0
pytest==3.0.7
pytest-django==2.9.1
six==1.10.0

[test_3-9pMWYqV9][christian@WorkBookPro:~/tmp/test_3]
[18:53:23] $ 

[c17r]

this is also apparent now that Django has released 1.11. I have Django = "<1.11.0" in my pipfile. Other requirements have as their requirements django, Django, django>=1.8, Django>=1.8. Pipenv will end up installing 1.11 instead of 1.10.7 which is annoying as there are breaking changes.

[nateprewitt]

Hey @c17r, sorry this has slipped through the cracks. I'm working my way through the backlog of tickets that's piled up in the last week.

Could you provide an example Pipfile for me with the Django issues so I can test a few things out. Thanks!

[c17r]

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
django-environ = "*"
Django = "<1.11.0"

is a good short example, you'll end up with 1.11

[c17r]

Oddly

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
django-extensions = "*"
Django = "<1.11.0"

does what you'd expect

[nateprewitt]

Ok, I'm now convinced we can be doing better here. The issue is when a package isn't specified with an exact version (e.g. ==1.10.8), we choose the first downloaded copy of the package. This works in the second example because Django isn't a (non-test) requirement for django-extensions. It is however a requirement for django-environ which doesn't have a specified version requirement so 1.11.0 is set first and can't be overwritten.

I've opened #305 to hopefully address this better. I've done some initial testing and we seem to be getting the desired results but I'd welcome you, or anyone else with an inclination, to test it out before we merge it.

[c17r]

I tested #305 and all my issues are working now.

[nateprewitt]

@c17r Great! We'll get this slotted for 3.6.0 then.

[c17r]

I found something else somewhat along these lines:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
Django = "<1.11.0"

[dev-packages]
django-debug-toolbar = "*"

You end up with Django listed in both default (==1.10.7) and develop (==1.11) section of the lock file. If you do pipenv update --dev, you'll end up with Django 1.11

[nateprewitt]

This piece operates differently in pip that I expected. We download and evaluate dependencies for packages and dev-packages separately, so they have no knowledge of what the other contains.

I think a solution here is to check the contents of packages when generating dev-packages and only write dependencies to dev-packages when they aren't present in packages. This should still get us all of our dependencies, and not overwrite specifics in packages. This is still flawed though because a specific requirement in dev-packages won't hold weight if it's filled by a packages dependency.

pip doesn't really handle dependency conflicts right now, which means we'll have to roll our own solution. We've been avoiding that thus far but I'm not seeing a way to avoid it here. We'd need to compare every package in the Pipfile and choose the most specific option for each dependency. It would likely mean raising an exception for conflicting Pipfile requirements (e.g. packages: (pytest==1.0.0, dev-packages: pytest==0.6.0).

@kennethreitz, any thoughts on this? It'll be a semi-significant extension but is functionality pip somewhat provides that we've "broken".

[c17r]

This is still flawed though because a specific requirement in dev-packages won't hold weight if it's filled by a packages dependency.

But is that really an issue? When you install/update, it's either packages or packages AND dev-packages, never just dev-packages. If a dependency is needed by both, in my opinion, it should be listed in packages.

[nateprewitt]

But is that really an issue? When you install/update, it's either packages or packages AND dev-packages, never just dev-packages. If a dependency is needed by both, in my opinion, it should be listed in packages.

Yeah, I think it definitely could be. If some_package has a dependency flask-oauth specified as a dependency, we'll install flask-oauth at the latest version for packages. Then when we get to dev-packages and the user has specified flask-oauth==0.5.2, the approach listed above will decide that flask-oauth is already covered and ignore the pinned requirement. That's the same problem you're experiencing, just in reverse.

[Diggsey]

Would it be possible to compare the output of pip freeze against pipfile.lock after a pipenv install, and warn on any packages where the versions don't exactly match?

[techalchemy]

was this fixed by 56aaa86a ?

[kennethreitz42]

i think it's safe to say yes.