Pipenv: Add the option to create other sections in Pipfile

[marctc]

A common pattern in Django projects is spliting requirements in differents files. We do this in our company following the recommendations of the Two Scoops of Django of @pydanny and @audreyr. In cookiecutter-django you can see this patterns: https://github.com/pydanny/cookiecutter-django/tree/master/%7B%7Bcookiecutter.project_slug%7D%7D/requirements

So, my thought is that pipenv needs to provide more options besides the regular pipenv install and pipenv install --dev. Maybe something like pipenv install requests --section=production and then pipenv install --section=production to install all the production related packages.

[kennethreitz42]

not happening. explicitly designed for just two groups.

[kennethreitz42]

e..g this is an explicit design decision, inspired by composer, not an accident.

[kennethreitz42]

[default] is production.

[marctc]

Ok, got it.

I just wanted to separate dev packages from test packages but it's not a big deal.

[kennethreitz42]

yeah, that's the idea, it's not a big deal. the fact that the composer community has been able to pull this off gives me faith that we'll be able to as well.

keep in mind, the pipfile spec allows for other groups to be added in the future, so we can add this later, if demand is really very high, but so far, it's been smooth sailing.

[kennethreitz42]

e.g. only having two groups is a superior experience (imo) to having arbitrary groups.

[marctc]

keep in mind, the pipfile spec allows for other groups to be added in the future, so we can add this later, if demand is really very high, but so far, it's been smooth sailing.

agree with that

[borfast]

Here's a use case for having more than two separate sections: when we want to install certain packages only in production and not in other environments. For example, packages that are only used in production, like gunicorn, or whitenoise, or some email library from a provider like Mailgun, Sendgrid, etc.

[techalchemy]

Don't you test your production packages?

[pydanny]

If you want to separate requirements this much, think on the following:

• consider staying with pip and virtualenv. It's low level enough and works great. Especially if you want to keep debugging tools that can be a risk out of production.
• Remember that standards and practices evolve over time. So maybe pipenv has invalidated the approach we specify in Two Scoops of Django.

[borfast]

@techalchemy, sure, I test the production packages, but perhaps I don't want to do it on my local machine and only on a CI server or something.

@pydanny, indeed, things evolve. I did separate the production packages before reading Two Scoops of Django, though :) Great stuff, by the way!

In the end, I agree with what others said: it's not a big deal. It means I will have to install packages I don't need in my local environment but it's not a show-stopper.

[techalchemy]

@pydanny Pipenv is actually very well suited to keep debugging tools out of production, since if you install them with pipenv install --dev then you have to forcibly and intentionally install your dev dependencies with the --dev flag again in the future. Its just having things that only belong in production that pipenv does not currently offer

[patcon]

Just want to add a use-case of how the two groups are less-than-ideal, although workable:

I would suggest that "groups" are for humans. they are for helping us organize are minds and the projects our minds build. in rubygems land, groups can be arbitrarily named. assuming that prod and dev as the distinction is not something that ecosystem tries to impose.

Example

One project I'm interested in doesn't even use python on the production website, but has some processing tasks that happen locally to build assets. The two groups it wants to make distinction between are "devserver" (a mock api server that runs in python only locally) and "processing" (a bunch of long-compiling scientific packages for doing geospatial work). Forcing a project like this in the dev/prod dichotomy makes less sense.

Conclusion

Basically, I feel that the distinction is more opinionated than need be, and prevents the tool from speaking human in the way that users might like it to. In my example, we need to have docs (in the CLI instructions and the code comments) to explain this, instead of just having intuitive names of the groups that can go in a command line flag :)

Anyhow, thanks so so much for all your hard work. Just wanted to provide a counter-point. Everything is work-aroundable, but I simply would contest that it shouldn't need to be -- it seems a reasonable ask. (fwiw, as someone who comes from php land, i feel rubygems and maybe go is the package management ecosystem to emulate, not composer)

Y'all are awesome! :heart:

[kennethreitz42]

Composer has been successful with just two groups. That's what inspired this decision.

[borfast]

I also worked with PHP for a very long time and, just like Patrick, don't really think it's the best thing to emulate.

@kennethreitz, how would Composer handle the scenario @patcon described?

[uranusjr]

I’m not very involved in the PHP community, but this seems to sum up their opinions: https://matthiasnoback.nl/2014/04/theres-no-such-thing-as-an-optional-dependency/

And really I agree with it. When developing an application (which is what Pipenv is for), the dev/prod combination is theoretically enough, and usually you even only need one.

In practice, however, I personally think additional groups are handy to have, especially if you need to use services that may not be able to install everything. For example I might be building a website using a Python static site generator, with some documentation written with Sphinx, use some plugins to auto-generate some stuff locally, a test runner to make sure the plugins work correctly, and a task runner to help automating the deployment. But not every service can install everything, especially specialised ones.

It is not impossible to work around (requirements files are always available), but this means my environments are not as reproducible as a straight Pipenv-managed one can be. I feel Python is used for too many things (compared to, say, PHP) that conditional deployment could be a handy thing to have. Not theoretically necessary, but also not totally redundant.

[kennethreitz42]

My ethos is "solve the 95% use-case". This is the 95% use-case.

[gsemet]

pipenv is used on real life with just two groups and it works great. default group only describe the running dependencies, while all the doc/api autogenation, lint, checks, unit tests dependencies, belongs to the dev group.
The only reason to add more group is that if there were incompatibility in two dev dependencies. For example the sphinx generator and pylint has major, unsolvable dependencies issue. I never saw that in real life.

[borfast]

What I don't get is why impose a limitation that brings no clear advantage (aside from the obvious simpler implementation but I wouldn't say it's such a big difference), when lifting that limitation would make life easier for many people, even if they are just 5%. But hey, it's not my project and I don't want to become a PITA by insisting on this _ad nauseam_ :)

[techalchemy]

The implementation factor is a huge advantage though. As a maintainer I can say that anywhere our codebase is complex or hard to follow I’m reluctant to accept any PR I didn’t personally test by hand and trace just because things are much more likely to break. Too much code like that and the project becomes a maintenance nightmare.

For end users, if you’re new to python and you see that you can have 5 sections you start thinking maybe you should. Then every time you try to use pipenv you can’t remember what section you put a package in. You don’t use cat Pipfile because you’re new, you have to open it in an editor and this is a slow process. You get frustrated. Eventually you just start guessing and accidentally wind up installing the same package in a brand new section you just created. Now you have 6 sections. Ultimately you decide the whole process is overly complicated and you don’t feel like dealing with it anymore so you go back to some other thing that isn’t pipenv or maybe it isn’t even python.

[uranusjr]

Yes, implementation is a big factor. As a contributor I went through multiple episodes that went like “maybe I should come up with something to try persuading Kenneth and Dan” and after 15 minutes my head started hurting so I stopped. For anyone strongly pushing this issue, I encourage you to get yourself familiar this whole dependency resolution thing is done, and try to come up with a workable implementation. Hopefully one of us will eventually come up with something, but before that, take my words and drop it, this conversation won’t go anywhere.

[borfast]

I totally agree with the implementation complexity weighing considerably on the decision of implementing something. Not so sure about the multiple sections, though. But yes, it's dropped :)

[jonathanverner]

Being able to have sections for docs/tests/... as in https://github.com/pypa/pipenv/issues/726#issuecomment-387707042 would be nice, but one can sort of live without this. However, how should one handle the case when one codebase covers multiple related but separate applications, each having different dependencies. For example, I am developing an application, which has a web, gui and cli interface. The gui needs the PyQt5 package but I wouldn't want to install it on the server serving the web interface (and in some environments, perhaps I couldn't, even if I wanted).

[srkn]

Being able to have sections for docs/tests/... as in #726 (comment) would be nice, but one can sort of live without this. However, how should one handle the case when one codebase covers multiple related but separate applications, each having different dependencies. For example, I am developing an application, which has a web, gui and cli interface. The gui needs the PyQt5 package but I wouldn't want to install it on the server serving the web interface (and in some environments, perhaps I couldn't, even if I wanted).

Same situation can be applied to API development, I need set different requirements for client and server side of the API.
Beside this need, I think implementing this idea is really complicates the deployment process, for example how pipenv will decide to relate the requirements unders the each section with the related files or folders?
First idea that came to my mind is creating new env for each section and then adding a support (can be a flag that represents the section name? eg. pipenv switch "section_name") into pipenv to switch between envs easily without having to create the envs on your own. With this way we can tackle the problem that deciding to which env will be related with which files