Poetry: support for testing / staging environments

[yunti]

• [ X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X ] I have searched the documentation and believe that my question is not covered.

Feature Request

We use several requirements files for testing staging etc.. including dev and production. Can poetry support such a setup?

[jacebrowning]

I believe that was some intention to keep number of environments small to minimize complexity.

Can you provide some examples of dependencies that you need in your staging environment but not production?

[hangtwenty]

@yunti You might want to practice 5 Whys on this and/or revisit 12-Factor Application and related ideas... Do you _really_ need to have any divergence between requirements for different envs? We all know why there's a 'dev' option, that is relevant to keeping test runners etc out of production if they can be. Unclear how/why there would be a general need for staging/etc.

I'm not looking for an answer in the abstract. I'm nudging you to go through literally each difference in these dependency lists, and see if there is even 1 that truly could not be installed in the other envs. If so, and you can bring it up concretely, it would be very useful to the thread.

[hangtwenty]

Even if there are concrete valid use cases -- would extras not give you what you're looking for?

[mozartilize]

for example, in local development, I need IPython and pytest, so maybe in pyproject.toml I can add them to dev dependencies, but when it's in CI test for production deployment, properly I dont need to install IPython.

Even if there are concrete valid use cases -- would extras not give you what you're looking for?

As it was discussed before, extras doesn't mean for this purpose (sdispater's comment, #759)

I think Poetry should support something like group of ruby's gemfile

[saidbouras]

In the best world, we don't need such a thing as multiples environment but as a developer in a corporate company, we are forces to develop in ..Windows !! And there is python packages that don't work on Windows and have a replacement package just for that shitty OS.

Example : kerberos, we have to get kerberos-sspi in Windows to replace the original package and such we have a divergence on dependencies.

[albireox]

A related case here is defining dependencies for building documentation. For example, I normally define Sphinx and some sphinx-related packages as docs dependencies. For me those are dev dependencies because I don't want the user to have to install them. But I do want a service such as ReadTheDocs to be able to do pip install .[docs] (for example, by defining docs as an extra in the RTD YAML file) to install those dependencies.

Right now the only way of achieving the above behaviour is by adding the docs dependencies to tools.poetry.dependencies as optional

Sphinx = {version = "^2.2.1", optional = true}
sphinx_readable_theme = {version = "^1.3.0", optional = true}

and then define them in tool.poetry.extras, but then doing poetry install or poetry update ignores those dependencies and I need to remember to do poetry install -E docs to install them. Apart from the fact that it's very verbose and really anti-intuitive.

I think the multitude of examples and issues about how this limitations breaks so many workflows is proof that this behaviour should be reconsidered.

[sangaline]

I'm not looking for an answer in the abstract. I'm nudging you to go through literally each difference in these dependency lists, and see if there is even 1 that truly could not be installed in the other envs. If so, and you can bring it up concretely, it would be very useful to the thread.

This is a concrete issue when testing against different Python implementations. If you have mypy as a development dependency for type checking, then you're unable to install the development environment with PyPy because typed_ast is incompatible. If you want to test against Python 3.5, then you can't install black. Adding your linting development dependencies as optional production dependencies and then needing to run extra commands in your normal development environment feels like a major workaround. It would be more natural for these development dependencies to be opt-out rather than opt-in.

It would be nice if we could do something like this on CI.

# When testing against PyPy.
poetry install --ignore mypy

# When testing against Python 3.5.
poetry install --ignore black

Another alternative would be specifying

mypy = {version = "^0.740.0", optional = true}

in pyproject.toml, and then having poetry install succeed with a warning if mypy can't be installed.

[AdrianVandierAst]

I have another use case: a package that is protected because it is an important part of the intelectual property of the company. The package is not available for development but should be installable on production.
Extras would fit this case but poetry checks for extras dependencies even if the extras are not selected (I suppose the goal is to create a correct .lock file). I think dependencies group may fit my need. As a workaround currently, i use a local path dependency for that protected package and we put a fake package there...

[funkyfuture]

As it was discussed before, extras doesn't mean for this purpose

well, the dev-depencies could be seen as an extra. and of course, @yunti's use-case can be addressed with extras.

[fredrikaverpil]

Cross-posting this from #1644:

Oh yes, this is exactly what I've been missing in poetry. Groups as well as individual deps that I need to install on their own or together with the production code.

I use poetry to setup multiple tox environments with isolated builds in tox.ini, and I currently need to wait for ~100 deps to be installed for each (!!!) tox env. The total installation time alone is many times longer than what it takes to run all tests. This feature would cut a large chunk out of the total wait time, and would probably be more valuable to me than speeding up the general installation time, which is something I understand is being worked on at the moment.

[mjpieters]

I'm also looking to reduce CI/CD installation times. Linting takes 10 seconds, running the test suite 16 seconds, but installing poetry and all the dev dependencies takes 2 minutes 20 seconds. The CI/CD steps don't need a lot of the dev dependencies, the test step doesn't need the linting dependencies, and the linting step doesn't need the testing dependencies. Yet, a developer will need _all_ the dev dependencies.

Extras don't really fit this case, as there is overlap in what each use-case needs to install and extras are listed in the distributed package (where they don't belong, these are _dev_ dependencies).

We basically have two fixed groups of dependencies right now: install dependencies and dev dependencies. Can we please have support for _any number_ of groups, with install and dev being the base defaults? Together with a simple group inclusion syntax, that would cover all our needs here.

The alternative right now is to maintain separate requirements files just for the CI/CD environment for use with pip install -r ... (really less than ideal as these will become outdated), or external tooling (poetry plugin or something that works from the pyproject.toml file and / or poetry export, these may require additional install time too).

---

Side note: I see several calls here for this feature because they have dependencies that don't work in certain environments. Those cases (@sangaline named mypy not working on pypy and black not working on Python 3.5, and @saidbouras mentioned kerberos and kerberos-sspi as libraries that you swap out when on Windows) are already fully supported in Poetry. You want to use Python and environment marker specifications for those.

E.g. here's a black dependency that will only be installed when running Python 3.6 or newer, on CPython (and not on pypy or Python 3.5):

[tool.poetry.dev-dependencies.black]
version = "19.10b0"
allow-prereleases = true
python = "^3.6"
markers = "platform_python_implementation == 'CPython'"

[mjpieters]

I note that this is very closely related to #1644, which specifically calls for group support.

[fredrikaverpil]

...and #1644 is now part of the roadmap, slated for v1.2: https://github.com/python-poetry/poetry/issues/1856 🥳❤️