Pipenv: Allow creating requirements.txt files without dev-packages

[tuukkamustonen]

To install a set of pinned packages in remote systems, there is:

pipenv install --system  # won't install dev dependencies

However, this requires both Pipfile and Pipfile.lock to be present, and also pipenv to be installed in the target system. It's way more inconvenient than just being able to (maybe first copy package wheel into remote system and) say pip install <mypackage> (and letting pip to pull in the pinned dependencies specified in setup.py's install_requires).

(Or is there some benefit in using pipenv install --system instead?)

To be able to generate pinned requiremets.txt that excludes development packages, add --no-dev option to pipenv lock, that seems to by default include also dev packages. Or maybe even change to default to match that of pipenv install, and exclude dev-packages by default?

As per https://github.com/kennethreitz/pipenv/issues/209#issuecomment-278185133 this is already supported in code:

from pipenv.project import Project
from pipenv.utils import convert_deps_to_pip

pfile = Project(chdir=False).parsed_pipfile
requirements = convert_deps_to_pip(pfile['packages'], r=False)

Although there is workaround for this need already, supporting it right in the cli would/could be nice.

[erinxocon]

I think we could add a method that doesn't add dev packages tot he requirements file. It's tough because normally you'd just have everything in your requirements.txt or maybe had two for production and dev dependencies. The reason pipenv install --system instead of just pip is that with the PIpfile.lock, you get a deterministic build every time you install.

[tuukkamustonen]

The reason pipenv install --system instead of just pip is that with the PIpfile.lock, you get a deterministic build every time you install.

Yeah, but you get the same result if you depend on requirements.txt (read into install_requires) that was created by pip-compile from requirements.in (from pip-tools package). Right? I don't see any difference.

maybe had two for production and dev dependencies.

Yeah, it's pretty common practice to have separate requirements.txt and dev-requirements.txt (or rather requirements.in and dev-requirements.in if using pip-tools).

[techalchemy]

@tuukkamustonen I think if you prefer using piptools with pip-compile directly you are certainly able to do that, pipenv uses pip-tools to build lockfiles so technically you can do that yourself, if you like. If you don't see the difference between pipenv and using pip-compile from pip-tools then you likely wont get anything out of using pipenv.

[piotr-dobrogost]

@tuukkamustonen
Btw, you should not read requirements.txt into install_requires – https://caremad.io/posts/2013/07/setup-vs-requirement/

[tuukkamustonen]

@techalchemy Wasn't aware of the internal use of pip-tools. Neat.

I'm not sure what you mean though - pip-tools doesn't support reading Pipfile, so I cannot mix-and-match pipenv and pip-tools usage directly from cli. With some wrapper code surely, but I wouldn't want to write that. And I would like to use (or at least try) Pipfile as it seems to be the next blessed thing(tm).

@piotr-dobrogost I'm not doing that for libraries (that I use as dependency in other projects) - merely for the final deliverables (web apps, in my case) that I deploy to remote servers.

[piotr-dobrogost]

Related – #916 (feature request: pipenv lock -r without the dev packages)

[erinxocon]

There is indeed a --deploy option now for not installing dev packages.

[tuukkamustonen]

@erinxocon Ok so my initial assumption:

pipenv install --system # won't install dev dependencies

Was not correct. The above line _does_ install dev packages. So it actually requires --deploy flag also. But the documentation for it only states:

Abort if the Pipfile.lock is out–of–date, or Python version is wrong.

Should it be updated?

Also, this ticket was about supporting dev-package exclusion in pipenv lock -r reqs.txt (through something like pipenv lock --no-dev -r reqs.txt). Would it be possible to add such option as well?

[techalchemy]

The documentation is out of date. I actually knew we had a —deploy flag but didn’t have a chance to figure out what it does. If you had a PR with updates to the documentation I’m sure it would be accepted

As for lock -r it only locks things which are installed. I realize that is not entirely helpful but the last piece of functionality doesn’t exist yet.

[tuukkamustonen]

As for lock -r it only locks things which are installed.

So things that show up in pip list? Doesn't match the behavior

mkdir foo
cd foo
pipenv install jinja2  # installs also markupsafe as dependency
pipenv uninstall jinja2
pipenv lock -r

Yields:

markupsafe==1.0 --hash=sha256:a6be69091dac236ea9c6bc7d012beab42010fa914c459791d627dad4910eb665
jinja2==2.9.6 --hash=sha256:2231bace0dfd8d2bf1e5d7e41239c06c9e0ded46e70cc1094a0aa64b0afeb054  --hash=sha256:ddaa01a212cd6d641401cb01b605f4a4d9f37bfc93043d7f760ec70fb99ff9ff

Do you mean that it should exclude jinja2 from the output?

I will look into PR once I understand how things are supposed to work :|

the last piece of functionality doesn’t exist yet.

Can we re-open this ticket then?

[erinxocon]

So here's the syntax for the new functionality

pipenv lock -r will give you just the default and vcs deps

pipenv lock -r -d will give you just the development deps and development vcs deps

Will update the documentation.

[wooyek]

@erinxocon I'm confused. Should it work in 9.0.0?

Calling pipenv lock -r gives me all requirements, including the development ones.

[nateprewitt]

@wooyek this was a regression in 9.0.0 and will be fixed in the next release.

[gregbrowndev]

Is it now not possible to generate a _requirements.txt_ with both production and dev dependencies in one file?

For context I'm developing inside a Docker container so want to install everything into the system environment not in a virtual env. I've avoided using pipenv install --dev --system in the Dockerfile, since it lead to a bootstrapping issue where the lock file was out of sync with the Pipfile so the image didn't build properly, so I had to remove all of that from my Dockerfile and go into the container and install using pipenv install --dev within a virtual env just so the lock file would update in my local project.

[uranusjr]

-r -d

[gregbrowndev]

-r -d

@uranusjr thanks but that creates a requirements file with just the development dependencies. I think the best solution is to use --system option and ensure the lock file is up-to-date.