Pipenv: `pipenv lock -r` incorrectly includes dev dependencies by default in requirements output

[idoru]

Describe your environment

1. Linux
2. Python version: 3.6.3
3. Pipenv version: 9.0.0

Expected result

When I run pipenv lock -r I see requirements compatible dependencies listed on standard output for non-dev dependencies, like so:

flask==0.10.1 --hash=sha256:4c83829ff83d408b5e1d4995472265411d2c414112298f2eb4b359d9e4563373
gunicorn==19.3.0 --hash=sha256:a5179cde922d2b4e045ee5b11e87323ee77d363ae40294fc8252f25d6a0eaf06  --hash=sha256:8bc835082882ad9a012cd790c460011e4d96bf3512d98a04d3dabbe45393a089
itsdangerous==0.24 --hash=sha256:cbb3fcf8d3e33df861709ecaf89d9e6629cff0a217bc2848f1b41cd30d360519
jinja2==2.7.2 --hash=sha256:310a35fbccac3af13ebf927297f871ac656b9da1d248b1fe6765affa71b53235
markupsafe==0.21 --hash=sha256:c6465cd6ed2b96509ef0100e7fff8718ed52c2affab1860ed5a9b67f604dd59a
werkzeug==0.10.4 --hash=sha256:a438aa8c3f513a5cf3dda02a62f9e022b7598e9fbfeba14ba2d6dd957c8ab436  --hash=sha256:9d2771e4c89be127bc4bac056ab7ceaf0e0064c723d6b6e195739c3af4fd5c1d

Actual result

Output contains dev dependencies, and due to other issues such as #1137 / #1000 can lead to an unusable requirements (due to duplicates), like so:

flask==0.10.1 --hash=sha256:4c83829ff83d408b5e1d4995472265411d2c414112298f2eb4b359d9e4563373
gunicorn==19.3.0 --hash=sha256:a5179cde922d2b4e045ee5b11e87323ee77d363ae40294fc8252f25d6a0eaf06  --hash=sha256:8bc835082882ad9a012cd790c460011e4d96bf3512d98a04d3dabbe45393a089
itsdangerous==0.24 --hash=sha256:cbb3fcf8d3e33df861709ecaf89d9e6629cff0a217bc2848f1b41cd30d360519
jinja2==2.7.2 --hash=sha256:310a35fbccac3af13ebf927297f871ac656b9da1d248b1fe6765affa71b53235
markupsafe==0.21 --hash=sha256:c6465cd6ed2b96509ef0100e7fff8718ed52c2affab1860ed5a9b67f604dd59a
werkzeug==0.10.4 --hash=sha256:a438aa8c3f513a5cf3dda02a62f9e022b7598e9fbfeba14ba2d6dd957c8ab436  --hash=sha256:9d2771e4c89be127bc4bac056ab7ceaf0e0064c723d6b6e195739c3af4fd5c1d
click==6.7 --hash=sha256:29f99fc6125fbc931b758dc053b3114e55c77a6e4c6c3a2674a2dc986016381d  --hash=sha256:f15516df478d5a56180fbf80e68f206010e6d160fc39fa508b65e035fd75130b
coverage==4.4.2 --hash=sha256:d1ee76f560c3c3e8faada866a07a32485445e16ed2206ac8378bd90dadffb9f0  --hash=sha256:007eeef7e23f9473622f7d94a3e029a45d55a92a1f083f0f3512f5ab9a669b05  --hash=sha256:17307429935f96c986a1b1674f78079528833410750321d22b5fb35d1883828e  --hash=sha256:845fddf89dca1e94abe168760a38271abfc2e31863fbb4ada7f9a99337d7c3dc  --hash=sha256:3f4d0b3403d3e110d2588c275540649b1841725f5a11a7162620224155d00ba2  --hash=sha256:4c4f368ffe1c2e7602359c2c50233269f3abe1c48ca6b288dcd0fb1d1c679733  --hash=sha256:f8c55dd0f56d3d618dfacf129e010cbe5d5f94b6951c1b2f13ab1a2f79c284da  --hash=sha256:cdd92dd9471e624cd1d8c1a2703d25f114b59b736b0f1f659a98414e535ffb3d  --hash=sha256:2ad357d12971e77360034c1596011a03f50c0f9e1ecd12e081342b8d1aee2236  --hash=sha256:e9a0e1caed2a52f15c96507ab78a48f346c05681a49c5b003172f8073da6aa6b  --hash=sha256:eea9135432428d3ca7ee9be86af27cb8e56243f73764a9b6c3e0bda1394916be  --hash=sha256:700d7579995044dc724847560b78ac786f0ca292867447afda7727a6fbaa082e  --hash=sha256:66f393e10dd866be267deb3feca39babba08ae13763e0fc7a1063cbe1f8e49f6  --hash=sha256:5ff16548492e8a12e65ff3d55857ccd818584ed587a6c2898a9ebbe09a880674  --hash=sha256:d00e29b78ff610d300b2c37049a41234d48ea4f2d2581759ebcf67caaf731c31  --hash=sha256:87d942863fe74b1c3be83a045996addf1639218c2cb89c5da18c06c0fe3917ea  --hash=sha256:358d635b1fc22a425444d52f26287ae5aea9e96e254ff3c59c407426f44574f4  --hash=sha256:81912cfe276e0069dca99e1e4e6be7b06b5fc8342641c6b472cb2fed7de7ae18  --hash=sha256:079248312838c4c8f3494934ab7382a42d42d5f365f0cf7516f938dbb3f53f3f  --hash=sha256:b0059630ca5c6b297690a6bf57bf2fdac1395c24b7935fd73ee64190276b743b  --hash=sha256:493082f104b5ca920e97a485913de254cbe351900deed72d4264571c73464cd0  --hash=sha256:e3ba9b14607c23623cf38f90b23f5bed4a3be87cbfa96e2e9f4eabb975d1e98b  --hash=sha256:82cbd3317320aa63c65555aa4894bf33a13fb3a77f079059eb5935eea415938d  --hash=sha256:9721f1b7275d3112dc7ccf63f0553c769f09b5c25a26ee45872c7f5c09edf6c1  --hash=sha256:bd4800e32b4c8d99c3a2c943f1ac430cbf80658d884123d19639bcde90dad44a  --hash=sha256:f29841e865590af72c4b90d7b5b8e93fd560f5dea436c1d5ee8053788f9285de  --hash=sha256:f3a5c6d054c531536a83521c00e5d4004f1e126e2e2556ce399bef4180fbe540  --hash=sha256:dd707a21332615108b736ef0b8513d3edaf12d2a7d5fc26cd04a169a8ae9b526  --hash=sha256:2e1a5c6adebb93c3b175103c2f855eda957283c10cf937d791d81bef8872d6ca  --hash=sha256:f87f522bde5540d8a4b11df80058281ac38c44b13ce29ced1e294963dd51a8f8  --hash=sha256:a7cfaebd8f24c2b537fa6a271229b051cdac9c1734bb6f939ccfc7c055689baa  --hash=sha256:309d91bd7a35063ec7a0e4d75645488bfab3f0b66373e7722f23da7f5b0f34cc  --hash=sha256:0388c12539372bb92d6dde68b4627f0300d948965bbb7fc104924d715fdc0965  --hash=sha256:ab3508df9a92c1d3362343d235420d08e2662969b83134f8a97dc1451cbe5e84  --hash=sha256:43a155eb76025c61fc20c3d03b89ca28efa6f5be572ab6110b2fb68eda96bfea  --hash=sha256:f98b461cb59f117887aa634a66022c0bd394278245ed51189f63a036516e32de  --hash=sha256:b6cebae1502ce5b87d7c6f532fa90ab345cfbda62b95aeea4e431e164d498a3d  --hash=sha256:a4497faa4f1c0fc365ba05eaecfb6b5d24e3c8c72e95938f9524e29dadb15e76  --hash=sha256:2b4d7f03a8a6632598cbc5df15bbca9f778c43db7cf1a838f4fa2c8599a8691a  --hash=sha256:1afccd7e27cac1b9617be8c769f6d8a6d363699c9b86820f40c74cfb3328921c
flake8==3.5.0 --hash=sha256:c7841163e2b576d435799169b78703ad6ac1bbb0f199994fc05f700b2a90ea37  --hash=sha256:7253265f7abd8b313e3892944044a365e3f4ac3fcdcfb4298f55ee9ddf188ba0
flask==0.12.2 --hash=sha256:0749df235e3ff61ac108f69ac178c9770caeaccad2509cb762ce1f65570a8856  --hash=sha256:49f44461237b69ecd901cc7ce66feea0319b9158743dd27a2899962ab214dac1
flask-testing==0.6.2 --hash=sha256:f25effd266fce9b16482f4ce3423d5a7d25534aab77bc83caace5d9637bf0df0
itsdangerous==0.24 --hash=sha256:cbb3fcf8d3e33df861709ecaf89d9e6629cff0a217bc2848f1b41cd30d360519
jinja2==2.10 --hash=sha256:74c935a1b8bb9a3947c50a54766a969d4846290e1e788ea44c1392163723c3bd  --hash=sha256:f84be1bb0040caca4cea721fcbbbbd61f9be9464ca236387158b0feea01914a4
markupsafe==1.0 --hash=sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665
mccabe==0.6.1 --hash=sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42  --hash=sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f
pluggy==0.6.0 --hash=sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff
py==1.5.2 --hash=sha256:8cca5c229d225f8c1e3085be4fcf306090b00850fefad892f9d96c7b6e2f310f  --hash=sha256:ca18943e28235417756316bfada6cd96b23ce60dd532642690dcfdaba988a76d
pycodestyle==2.3.1 --hash=sha256:6c4245ade1edfad79c3446fadfc96b0de2759662dc29d07d80a6f27ad1ca6ba9  --hash=sha256:682256a5b318149ca0d2a9185d365d8864a768a28db66a84a2ea946bcc426766
pyflakes==1.6.0 --hash=sha256:08bd6a50edf8cffa9fa09a463063c425ecaaf10d1eb0335a7e8b1401aef89e6f  --hash=sha256:8d616a382f243dbf19b54743f280b80198be0bca3a5396f1d2e1fca6223e8805
six==1.11.0 --hash=sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb  --hash=sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9
tox==2.9.1 --hash=sha256:8af30fd835a11f3ff8e95176ccba5a4e60779df4d96a9dfefa1a1704af263225  --hash=sha256:752f5ec561c6c08c5ecb167d3b20f4f4ffc158c0ab78855701a75f5cef05f4b8
virtualenv==15.1.0; python_version != '3.2' --hash=sha256:39d88b533b422825d644087a21e78c45cf5af0ef7a99a1fc9fbb7b481e5c85b0  --hash=sha256:02f8102c2436bb03b3ee6dede1919d1dac8a427541652e5ec95171ec8adbc93a
werkzeug==0.12.2 --hash=sha256:e8549c143af3ce6559699a01e26fa4174f4c591dbee0a499f3cd4c3781cdec3d  --hash=sha256:903a7b87b74635244548b30d30db4c8947fe64c5198f58899ddcd3a13c23bb26

Steps to replicate

Run pipenv lock -r with this Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
Flask = "==0.10.1"
gunicorn = "==19.3.0"
itsdangerous = "==0.24"
Jinja2 = "==2.7.2"
MarkupSafe = "==0.21"
Werkzeug = "==0.10.4"

[requires]
python_version='3.6'

[dev-packages]
tox = "*"
coverage = "*"
"flake8" = "*"
flask-testing = "*"

[nateprewitt]

Sorry I redacted my previous response because it was operating a false assumption from reading the title. We'll look into what's causing this, I'm not immediately sure why you'd end up with duplicate entries for the same package. This may be another issue with our recent file splitting changes.

The inclusion of dev dependencies is likely related to #1172.

[idoru]

Thanks @nateprewitt !

Yes I agree #1172 looks like it fixes the issue, however, when I looked at the code it seems there were two problems with the test behavior (false positive on testing dev-deps, and did not properly check for exclusion of dev-deps by default).

Also it looks like the return values from merge_deps is being misappropriated, and that the code in cli.py could have been cleaned up to simply rely on the second return value for outputting requirements, since they are already determined in merge_deps.

To this end I've created #1178 which I feel would be an improvement, let me know what you think!

[sobolevn]

The same here for me. Here's my Pipfile.lock: https://github.com/wemake-services/wemake-django-template/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/Pipfile.lock

As a workaround I have downgraded to 8.3.2.
I have also put my main dependencies into the dev section: https://github.com/wemake-services/wemake-django-template/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/Pipfile#L39

Nothing else was able to solve it.

[techalchemy]

@idoru thanks for the suggestions, I haven't had a chance to look at your PR but both of the sections in question just underwent quite a bit of cleanup (thus the current issue with including dev depenedencies which I introduced). During the cleanup/slight refactor I didn't want to alter the logic too much, but I believe you are correct about relying on the second value for requirements.

WRT the test itself, it wasn't really having false positive testing on dev-deps, I was just unsure about how the CLI treatment of the virtualenv would interact with the environment. Since requests is isntalled by default with pipenv, I felt it was best to not assume the tests ran totally insulated from a given CI environment, so I swapped to a different test case.

@sobolevn this is interesting. I actually ran into the same issue today. Definitely something we should try to address.

[sobolevn]

@techalchemy any progress on this one? Maybe something I can help with?

This issue is hitting me hard.

[techalchemy]

@sobolevn the main issue is resolved in master but the conflict between dev and non-dev resolution may not be. I haven’t had a chance to focus on this just yet

[kennethreitz42]

this is intended behavior

[sobolevn]

Could you please explain why this behavior is intended?
It does not make sense to me.

[kennethreitz42]

it is a flat lock file that includes all dependencies

[hynek]

This has been fixed in 9.0.1 – I think Ken is confused here. But closing is the right action in any case. :)

[techalchemy]

@hynek I don't think he is confused, I think we may have broken the original API as he designed it because of the nature of lockfiles in general.

It is in fact _because_ we split dev and regular dependencies that we are experiencing the issue that @sobolevn detailed regarding resolution and which required putting main dependencies into the dev section as well -- granted that flattening is not the only way to fix this, but it is the simplest. That is probably why it was designed that way. I'm not actually even sure if it is possible to maintain a deterministic dependency graph with two independent sections... would such a graph have to be flat by nature?

[uranusjr]

I think the flat structure makes more sense. The lock file is supposed to be able to reflect an exact snapshot of the virtualenv, so it can deterministically reproduce it on another machine. Since Python always flattens dependencies (unlike, say, Node), the lock file should naturally also be flat. It doesn’t have to be, theoretically, but that would complicate things unnecessarily IMO.

There is nothing wrong with keeping two graphs. It’s just both need to be flat and self-contained. The two graphs should be regular, and regular + dev (instead of regular and dev). This way pipenv install --dev can work solely on the latter graph, providing a consistent behaviour.

[techalchemy]

@uranusjr I really like this idea, it seems simple and elegant.

[tklarryonline]

Thank you, @sobolevn, for your solution. We had spent lots of time debugging this problem 😿 .