pipenv is not usable for projects with production-only dependencies

[maclauss]

Hi, we're running into some issues that make pipenv (which I really like by the way), unfortunately unusable for us.

The current "dev-packages" and "packages" mechanism, while enough for many simple use cases, does not allow to express dependencies that are only available in production, be it because of system dependencies (e.g. picamera can't be installed on a mac or windows), hardware dependencies (GPU dependencies on a laptop without a GPU), or simply because we do not want that dependency in dev.

I know there are a number of closed issues that relate to this very problem, however I don't think any of them mention packages that are production-only by design. This is a major hurdle and having a separate workflow for production seems to defeat the purpose of a package manager, at least for us.

Any chance a solution might be implemented in the short-term @kennethreitz ?

[techalchemy]

The current "dev-packages" and "packages" mechanism, while enough for many simple use cases, does not allow to express dependencies that are only available in production

Pipenv is a tool for developing applications; if you have dependencies that you use in your application but you don't want to install in development, that's not something pipenv is going to allow you to express exactly. I suspect based on your other feedback that, with a better understanding of your use case, we can probably figure out what I think of as two core issues:

1. What is it that you need to accomplish, and what are the barriers? Typically your challenges are addressed with environment markers which is likely what I will suggest here (but we will come back to the GPU point) -- can you say more about the specific packages you are looking to avoid installing in development, and the differences between the dev system and the prod system in terms of OS, architecture, kernel (if linux),

2. How can we make it more intuitive to you and other users what it is that is required to accomplish this?

Onto the details!

be it because of system dependencies (e.g. picamera can't be installed on a mac or windows),

This is handled by using environment markers (outlined in the docs here). You can simply modify your pipfile entry so if it looks like:

package = "*"

it becomes

package = {version = "*",  markers = "os_name=='posix'"}

You can find the full list of specifiers that are valid in this section here.

hardware dependencies (GPU dependencies on a laptop without a GPU), or simply because we do not want that dependency in dev.

This is a bit more complicated, but as I mentioned above if you can share more information about the package in question and the systems (dev / prod) and their differences it would be more helpful for understanding the use case and whether anything is possible here currently -- if it is, we can point you in the right direction. If not, I can surface this in a PyPA discussion more broadly and see about changing Pep 508

I don't think any of them mention packages that are production-only by design

Right, everything is assumed to be a production dependency by default because if you're developing an application the assumption is you need to have its production dependencies installed, otherwise you might wind up deploying a different set of dependencies than you had installed locally and the behavior may be different.

If this still doesn't accomplish what you want, you probably would want to consider whether your project should be a python package (if it isn't already) and then you can define the gpu-specific dependencies as extras in your setup.cfg or setup.py. Part of why we haven't supported this in particular is because our design choice was to make it as difficult as possible to be out of sync with production while you develop locally. This ensures that deployments reproduce your development environment and therefore your dev environment behaviors. For the large majority of use cases this is the most sane default.

If you really need a short-term workaround, you can always just move the gpu dependency to your dev section, but I would really like to understand more about the use case

Thanks for the report!

[maclauss]

Thanks @techalchemy for the prompt and very detailed reply.

1. I'll give a try to the markers right away but it assumes the OS is different between prod and dev (which is luckily almost always the case for us). For the GPU issue, I could see this happening with "tensorflow" and "tensorflow-gpu". We're not having this issue at the moment because we need to build tensorflow tailored to the machine anyway (CPU, GPU, CUDA / cuDNN versions, etc...), but I could see that someone who simply wants to use the prebuilt "tensorflow-gpu" on their inference / training machines and "tensorflow" on their laptop could run into these problems. I'm not looking for a specific answer but that could be helpful if someone finds that post.

2. A google search redirected me to solutions that were not satisfying. I have no excuse for not seeing the marker example in this page https://docs.pipenv.org/advanced/#specifying-basically-anything

[maclauss]

The marker that worked for me was: picamera = { version = "*", markers = "platform_system=='Linux'" }

Closing this ticket, thanks @techalchemy

[dmillard]

I am also using tensorflow. Unfortunately, my dev and prod setups both run linux and are separated effectively only by the presence of GPUs. Having read PEP 508, I don't believe there is enough information in the markers language to distinguish these systems. Is there some other way that future versions of pipenv can select dependencies? Perhaps through a prod-only section, or similar.

e.g.

...
[packages]
foo = "*"

[prod-packages]
tensorflow-gpu = "*"

[dev-packages]
tensorflow = "*"
...

where [prod-packages] are only installed if --dev is missing?

[dmillard]

@techalchemy, is it possible to reopen this issue, since I don't think that the given solution actually solves the problem?