Pipenv: [Cli Docs?] Conflicting Info in regards to `pipenv lock --dev --requirements`

[gtalarico]

Overview

Docs, Actual Results, and CLI Help are in conflict:
Should --dev include only dev dependencies? Or both dev + default dependencies?

Actual Results

pipenv lock --dev --requiremnts

Outputs only dev dependencies

Docs

States that --dev should output only dev dependencies

If you wish to generate a requirements.txt with only the development requirements you can do that too! ... pipenv lock -r --dev

https://pipenv.readthedocs.io/en/latest/advanced/#generating-a-requirements-txt

CLI usage

States --dev should includes both dev + default dependencies:

  -d, --dev           Install *both* develop and default packages.  [env var: PIPENV_DEV]

pipenv, version 2018.11.26

[gtalarico]

@frostming please confirm: is CLI docs incorrect and expected behaviour is dev deps only

[frostming]

Let me try to clarify it. We re-organized the commands and options to reduce repetitions. And the --dev option is taken from a common option, which is documented as "install both develop and default packages", whereas, as you found, behaves differently between commands.

In the context of lock -r, it is likely that people use it to generate the old-fashioned requirements.txt and dev-requirements.txt, so it is more expected the two sections are generated separately. Now what we should do is to replace the common option to a specific one with the correct document string to reflect what it does. Contributions are welcomed.

It is just my opinion and it might be wrong. Kenneth should have some words for it.

[frostming]

@gtalarico I just changed my mind and modified the previous comment a lot, sorry for the trouble.

[gtalarico]

I understand the logic, but using the same exact --dev flag for opposite behaviors smells funny to me
🙊

Edit

Did i get this right?

| Command| Flags| Default Deps | Dev Deps |
| ------------- | ------------- | ------------- | ------------- |
| pipenv install | none | ✅| ❌|
| pipenv install | --dev | ✅| ✅|
| pipenv lock | none | ✅| ✅ |
| pipenv lock | --requirements | ✅| ❌|
| pipenv lock | --requirements --dev | ❌| ✅|

• installing only dev dependencies is possible using (ok, edge case use)

pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt

[jxltom]

@gtalarico You are right as shown in the table. Currently --dev has different logic for install and lock -r

Install only dev packages is not a typical usage but you can do that by pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt.

By default, pipenv lock is locking both default and dev. I think --dev is not working for pipenv lock --dev (correct me if i'm wrong) which needs enhancement.

[gtalarico]

install only dev packages is not a typical usage but you can do that by pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt.

I think that's good enough given how unusual of a use case that is.

By default, pipenv lock is locking both default and dev

Updated the table above to reflect this

[techalchemy]

@gtalarico the api is backwards when you dump requirements, I didn't like that when it was implemented but it's probably too late to undo it

@jxltom it's technically impossible to lock dev packages on their own in any meaningful way with the current implementation of locking. Even if we keep the dev dependencies separate, they can't ever be allowed to _conflict_ with the production dependencies, which means we currently just overwrite dev dependencies as they are generated if they conflict.

It's not possible to generate them independently because by nature the dev dependencies _require_ the default dependencies to exist and be up to date first, otherwise there is no way to be sure they are not in conflict.

[jxltom]

@techalchemy Thanks for explanation. I feels same for that locking default and dev deps independently is not possible and also it is not a typical use. Maybe we could consider removing --dev options for pipenv lock otherwise it will confuse users.

[frostming]

@jxltom I doubt if there is a way to do this since --dev is needed by lock -r.

[jxltom]

@frostming Oh, yes, I missed that... Thanks for the kind reminder. I don't know whether click has this kind of support which accepts only lock or lock -r -d but not lock -d. The other way is to raise a warning manually for user when pipenv lock --dev is used.

[gtalarico]

Maybe we could propose a change... ?
Remove --dev flag due to its amiguitiy and opposite behavior of install dev flag.

Use lock --requirements and lock --requirements-dev to generate requirements txt and requirements dev txt

[layoaster]

In my case, I'm using the old-fashioned requirements.txt to build Docker images.

Therefore, it's extremely needful to have pipenv lock -r generating only default deps and pipenv lock -r -d generating both (default & development) dependencies together so I can build separate images for development and production.

As workaround, I have to manually add -r ./requirements.txt to my requirements_dev.txt which minimizes extra work but still far from optimal.

Honestly, I don't know if there is a better way to work with Docker images while using/adopting pipenv since this is the first time I use pipenv. if you know a better way, please point me to the right direction.

[philtonite]

@layoaster I struggled with this while trying to build docker images too - also the first time I'm using pipenv.

I found a way to use the Pipfile.lock and this command: pipenv install --ignore-pipfile --system --deploy --dev when building the image.

Currently I'm installing all the packages in the image, as I use the same image to test and run in production.

[nickwilliams-eventbrite]

Yet another very frustrating roadblock for us. We're trying to work around #1356 and #3586, which are basically killing us. We decided to just convert the Pipfile.lock to a requirements.txt because we know we can rely on pip install to actually fail when it fails. But pipenv lock -r --dev was not working as expected/documented. So then I came across this issue.

We completely disagree with the assertion that the documentation is wrong. It makes no sense for --dev to work differently here than it does in install, et. al. If you see a use case for dev-only requirements files, then add --dev-only, or, at the very least, PLEASE give us a --default-and-dev option so we can generate one file with everything. What should be a simple pipenv install --dev --deploy for us has now become the following, and it's horribly frustrating:

    pipenv lock --requirements > .tmp_requirements.txt
    pipenv lock --requirements --dev > .tmp_requirements_dev.txt
    pip install -r .tmp_requirements.txt
    pip install -r .tmp_requirements_dev.txt
    rm .tmp_requirements*.txt

[ncoghlan]

+1 from me for @frostming's proposal (cc @techalchemy).

We just ran into this in a docker build after switching to pip-sync to work around #3057 (since pip-sync removes non-matching packages, unlike pipenv sync, it's more sensitive to transitive dependencies being missing from the locked output).

In our case, we can work around it by removing the production/dev distinction (as our production deployments don't go through pipenv, they use a separate pip-compile invocation), but the general case needs --dev to work the same way it does in other commands.

[ncoghlan]

(Sorry about that - this isn't closed yet, there was just a magic comment in the PEEP 006 PR merge)