Poetry: Support dependencies groups

[pawamoy]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Feature Request

As @mozartilize stated in this comment, groups from Ruby's gemfile could be a great addition and a solution to our complex workflows and different environments (local, production, ci, docs, etc.).

Instead of supporting more tool.poetry.*-dependencies sections, we could add support for a groups property on dependencies. It would be more flexible than rigid sections.

I also understand there is the extras functionality, but using extras changes the behavior of poetry install, as one needs to now specify the extras to install all the dependencies. Also, extras are opt-in only, and often times we need opt-out.

[tool.poetry.dependencies]
python = "^3.6"
requests = "*"

[tool.poetry.dev-dependencies]
bandit = { version = "^1.5", groups = ["ci"] }
black = { version = "*", allows-prereleases = true, groups = ["ci"] }
coverage = { version = "=5.0a8", allows-prereleases = true, groups = ["ci"] }
flake8 =  { version = "^3.6", groups = ["ci"] }
ipython =  { version = "^7.2", groups = ["local"] }
isort = { version = "^4.3", extras = ["pyproject"], groups = ["ci"] }
jinja2-cli =  { version = "^0.7.0", groups = ["local"] }
pylint = { git = "https://github.com/PyCQA/pylint", groups = ["ci"] }
pytest =  { version = "^4.3", groups = ["ci"] }
pytest-cov =  { version = "^2.8", groups = ["ci"] }
pytest-sugar =  { version = "^0.9.2", groups = ["ci"] }
pytest-xdist =  { version = "^1.26", groups = ["ci"] }
recommonmark =  { version = "^0.6.0", groups = ["docs"] }
safety =  { version = "^1.8", groups = ["ci"] }
sphinx =  { version = "^1.8", groups = ["docs"] }
sphinx-rtd-theme =  { version = "^0.4.2", groups = ["docs"] }
toml =  { version = "^0.10.0", groups = ["docs"] }

Also:

• default group would be attributed to dependencies without groups (requests in the above example)
• each dependencies would be attributed its own group as well, something like /bandit for bandit, etc.

Then we would have the following options for the poetry install command:

• --groups: explicitely specify which groups to install (these and only these groups, no implicit default group)
• --with-groups or --with: default group plus the specified groups
• --without-groups or --without: all groups minus the specified groups

Usage would be:

# in CI
poetry install --groups ci

# or in specific CI jobs
poetry install --groups /bandit  # bandit reads the code, we don't need anything else
poetry install --with /safety    # safety needs production dependencies to be installed

# locally, all three lines are equivalent
poetry install  # installs everything, as usual
poetry install --with local,ci,docs
poetry install --groups default,local,ci,docs

# on readthedocs
poetry install --with docs  # sphinx-autodoc needs production dependencies to be installed

1007 could then probably be closed in favor of this issue.

[pawamoy]

I didn't show any use-case for --without, any ideas?

Neither for multiple groups on dependencies. This one must be clarified/specified more thoroughly I guess:

default-dep = "*"
group-dep = { version = "*", groups = ["hello", "world"] }

Then what happens when I do:

poetry install --with hello
# since we do not include the "world" group, do we include group-dep or not? 

poetry install --without hello
# since we exclude the "hello" group, but keep the other ones including "world", do we exclude group-dep or not?

From the group page mentioned above (for --without):

Install all gems, except those in the listed groups. Gems in at least one non-excluded group will still be installed.

So if we follow the same logic:

poetry install --without hello
# will still install group-dep since it's in the non-excluded "world" group

poetry install --with hello
# should be the inverse? so it would not include group-dep since "world" is non-included?

[thejohnfreeman]

@pawamoy I'll try to answer your questions as I understand them.

1. What are the use-cases for --without?

The first is excluding documentation dependencies from my test environment in continuous integration. I have a library, project-template-python, that fails on AppVeyor during installation of dependencies because some of them report An error occurred when reading setup.py or setup.cfg: 'Attribute' object has no attribute 'id'. Even though none of these warnings halts the installation or breaks the environment for testing, and even though I specified $ErrorActionPreference = "Continue", AppVeyor halts the test as a failure. I know that at least one of these dependencies is pyobjc-core, depended on through sphinx-autobuild, a documentation dependency, and I suspect that the others are related to non-test dependencies too. Instead of figuring out how to get AppVeyor to ignore these warnings, I'd rather tell Poetry to skip their installation.

This adds a performance benefit, too, as @rmorshea put it:

A common use case for having dev-extras is bifurcated requirements for local development vs remote CI/CD pipelines. Given how slow dependency resolution is, being able to reduce the number of packages you install in your CI/CD pipeline can shave a lot of time off each build.

The second use case, which led me down a rabbit hole until I landed here, is... related to a conversation you and I had almost a year ago! (Wow, I honestly didn't notice either me or you in that thread until I started writing this comment.) Here's my example:

• I have a library, xpring-py, that depends on a library, fastecdsa, that requires libgmp to be installed on the system.
• libgmp is not a dependency available in PyPI, which means I cannot install it into the ReadTheDocs (RTD) build environment.
• These days RTD installs dependencies with pip, but if it could use Poetry, and if Poetry could exclude dependencies via --without, then I could use Poetry to install all the installable dependencies in the RTD environment and let autodoc mock the rest.

1. I think it's a good place to start if dependencies can only exist in one group. It will be easy to extend from one group to many groups if a use case arises, but it will be a breaking change to go the other direction.

2. However, if we must have multiple groups, then given:

group-dep = { version = "*", groups = ["hello", "world"] }
optional-dep = { version = "*", groups = ["hello", "world"], optional = true }

here is the behavior I would expect:

poetry install # group-dep in, optional-dep out
poetry install --with hello # both in
poetry install --with world # both in
poetry install --with hello,world # both in
poetry install --without hello # group-dep in, optional-dep out
poetry install --without world # group-dep in, optional-dep out
poetry install --without hello --with world # both in
poetry install --without world --with hello # both in
poetry install --without hello,world # both out

Let me just remark that the Ruby semantics are very poorly documented, but let me try to take their rule and tweak it to match these semantics as simply and concisely as possible:

Install all gems, except those that are optional or in the --without groups, but including those in the --with groups. Gems in at least one non-excluded group will still be installed.

I hope this is intuitive.

[kkozmic]

This issue is pretty important to me.
Is there any way I can help move it forward?
Is it something that the maintainers see as a priority?

[matthijskooijman]

How would this feature relate to the "extras" feature? Would this be able to replace extras, or is there something that extras provides that these groups could not? It seems like groups would be somewhat of a superset? The way you specify what extras/groups are installed is different, but once you select the same set of extras/groups to install, would there be other differences?

It seems that extras are not currently really working or documented well (#1145, #1076), so maybe this could be replacement for extras with clearer and more complete semantics?

Note that where I say "extras" above, I refer to the "clusters of dependencies"-interpretation of that, not the "extras to apply to dependencies"-interpretation. See https://github.com/python-poetry/poetry/issues/1076#issuecomment-639569322 for some discussion of the difference. I believe the "extras to apply to dependencies"-interpretation is clearer and would not need to be replaced.

[Cielquan]

How would this feature relate to the "extras" feature? Would this be able to replace extras, or is there something that extras provides that these groups could not? It seems like groups would be somewhat of a superset? The way you specify what extras/groups are installed is different, but once you select the same set of extras/groups to install, would there be other differences?

It seems that extras are not currently really working or documented well (#1145, #1076), so maybe this could be replacement for extras with clearer and more complete semantics?

I would not replace the extras system but add the group system on top.

Reason: (Blow I added a shorted version of pyproject.toml like I use and like I want to use.)

So the thing is that I use the extras system to split my development dependencies into categories. Then I can install mypackage[docs] into the venv which tests and builds my docs. I can install mypackage[testing] into the venvs testing my code. etc.
Of cause I could add the testing, docs and pre-commit dependencies directly to the dev dependencies, but then each and every venv created for testing will have multiple dependencies it does not need. This increases time for installation and space. (I use tox for automation)

So I think a good idea would be to keep the [tool.poetry.extras] section for extras meant to enhance usability of your package to the user like toml for coverage to enable pyproject.toml as config source. The group feature could then be a development thing to split up development dependencies for certain tasks.

And I would only allow group assignments for optional dependencies. I see no use for the feature for mandatory dependencies.

Currently:

...
    [tool.poetry.dependencies]
        python = "^3.6.1"
        importlib-metadata = {version = "^1.6", python = "<3.8"}
        mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
        enhancing-dep = {version = "^5.1.1", optional = true}
        # Testing
        pytest = {version = "^5.4.2", optional = true}
        # Docs
        sphinx = {version = "^3", optional = true}
        # Code check
        pre-commit = {version = "^2.4", optional = true}
        mypy = {version = "0.770", optional = true}
        pylint = {version = "^2.4", optional = true}

    [tool.poetry.dev-dependencies]
        devtools = {version = "^0.5", extras = ["pygments"]}

    [tool.poetry.extras]
        better-pkg-usability = ["enhancing-dep"]
        testing = ["pytest"]
        docs = ["sphinx"]
        pre-commit = ["pre-commit", "mypy", "pylint"]
...

How I would like it:

...
    [tool.poetry.dependencies]
        python = "^3.6.1"
        importlib-metadata = {version = "^1.6", python = "<3.8"}
        mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
        enhancing-dep = {version = "^5.1.1", optional = true}
        # Testing
        pytest = {version = "^5.4.2", optional = true, group = ["testing"]}
        # Docs
        sphinx = {version = "^3", optional = true, group = ["docs"]}
        # Code check
        pre-commit = {version = "^2.4", optional = true, group = ["pre-commit"]}
        mypy = {version = "0.770", optional = true, group = ["pre-commit"]}
        pylint = {version = "^2.4", optional = true, group = ["pre-commit"]}

    [tool.poetry.dev-dependencies]
        devtools = {version = "^0.5", extras = ["pygments"]}

    [tool.poetry.extras]
        better-pkg-usability = ["enhancing-dep"]
...

With this a user could install the extras to enhance usability like always: pip install mypackage[better-pkg-usability].
And for the docs testing env I could run poetry install --with docs and so on.

Note that where I say "extras" above, I refer to the "clusters of dependencies"-interpretation of that, not the "extras to apply to dependencies"-interpretation. See #1076 (comment) for some discussion of the difference. I believe the "extras to apply to dependencies"-interpretation is clearer and would not need to be replaced.

Both are extras but only viewed from another perspective. The "clusters of dependencies"-extra for your project is an "extras to apply to dependencies"-extra for someone else using your package.

Also in setuptools the first is specified in extras_require section and the latter just inside [] after the dependency in install_requires section.

PS: The extras for dependencies like this devtools = {version = "^0.5", extras = ["pygments"]} are fine as they are IMHO.

[fredrikaverpil]

Oh yes, this is exactly what I've been missing in poetry. Groups as well as individual deps that I need to install on their own or together with the production code.

I use poetry to setup multiple tox environments with isolated builds in tox.ini, and I currently need to wait for ~100 deps to be installed for each (!!!) tox env. The total installation time alone is many times longer than what it takes to run all tests. This feature would cut a large chunk out of the total wait time, and would probably be more valuable to me than speeding up the general installation time, which is something I understand is being worked on at the moment.

[SimonBiggs]

I really liked this idea so I implemented it as a temporary work around until it lands in poetry actual.

The idea, is you flag the groups with a comment:
https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/pyproject.toml#L33-L35

And then the following code takes those comments and writes them into tool.poetry.extras:
https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/scripts/propegate-extras.py#L25-L51

[matthijskooijman]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are? It seems that both are just specifying sets (I'm intentionally not saying "groups" here, to emphasize I'm talking about the generic concept of sets rather than the proposed "groups" feature) of dependencies, the difference seems to be only in the way these sets are addressed an their default.

@Cielquan wrote:

So I think a good idea would be to keep the [tool.poetry.extras] section for extras meant to enhance usability of your package to the user like toml for coverage to enable pyproject.toml as config source. The group feature could then be a development thing to split up development dependencies for certain tasks.

What I read there is that "extras" are things that you can add to your dependencies (default omitted) and "groups" are things that you can omit from your (dev) dependencies (default included).

The implementation @SimonBiggs follows a similar flow, where "groups" are things in the default dependency list that are annotated, resulting in an entry in the "extras" section as well.

So I wonder if maybe the "extras" feature should just be somewhat
generalized. It already allows specifying sets, so what if we would
enhance extras with a way to make extras installed by default? And then
add a -with and --without option to include or exclude these sets?
Since extras dependencies are currently already specified with their
version and optional=true in the main dependencies list and the name is
duplicated in the extras section, making installed-by-default extras
could be a matter of just omitting the optional tag (which I think is
pretty much what optional does right now, if it's false or omitted, the
package is installed by default, otherwise only when requested).

E.g. taking the example from @Cielquan, that could become as below. Note
that I moved a bunch of dependencies to dev-dependencies, as I think
that's where they would belong?

[tool.poetry.dependencies]
    python = "^3.6.1"
    importlib-metadata = {version = "^1.6", python = "<3.8"}
    mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
    enhancing-dep = {version = "^5.1.1", optional = true}

[tool.poetry.dev-dependencies]
    devtools = {version = "^0.5", extras = ["pygments"]}
    # Testing
    pytest = {version = "^5.4.2"}
    # Docs
    sphinx = {version = "^3"}
    # Code check
    pre-commit = {version = "^2.4"}
    mypy = {version = "0.770"}
    pylint = {version = "^2.4"}

[tool.poetry.extras]
    better-pkg-usability = ["enhancing-dep"]
    pre-commit = ["pre-commit", "mypy", "pylint"]
    docs = ["sphix"]
    testing = ["pytest"]

However, this does end up with the information spread out, so now I've
writen it down, I'm not so sure I like this approach much. I can also
imagine using the "groups" attribute, as suggested before, and then
still use "optional" to indicate installed-or-omitted by default. This
could become something like:

[tool.poetry.dependencies]
    python = "^3.6.1"
    importlib-metadata = {version = "^1.6", python = "<3.8"}
    mandatory-dep = {version = "^1.5.3", extras = ["some-extra"]}
    enhancing-dep = {version = "^5.1.1", optional = true, groups = ["better-pkg-usability"]}

[tool.poetry.dev-dependencies]
    devtools = {version = "^0.5", extras = ["pygments"]}
    pytest = {version = "^5.4.2", groups = ["testing"]}
    sphinx = {version = "^3", groups = ["docs"]}
    pre-commit = {version = "^2.4", groups = ["pre-commit"]}
    mypy = {version = "0.770", groups = ["pre-commit"]}
    pylint = {version = "^2.4", groups = ["pre-commit"]}

Compared to the original example by @Cielquan, this:

• Renames "group" to "groups", since it contains a list.
• Removes the "optional=true" from all dependencies that should be installed by default (e.g. all but enhancing-dep).
• Moves the dev dependencies to the dev-dependencies section, so they get installed by default without --no-dev.
• Removes the "extras" section, in favor of adding a "groups" field on the ehancing dep.

The idea here is that everything in the dependencies section is installed always, except for the enhancing-dep which needs --with better-pkg-usability (or mypackage[better-pkg-usability]). Everything in dev-dependencies is installed when --no-dev is omitted, but can be granularly chosen using e.g. --without pre-commit.

Essentially, this would mean that "extras" and "groups" are just the same thing. More specifically:

• Any dependency foo that has optional=true and group=bar is equivalent to specyfing bar = ["foo"] in the extra section (and vice versa).
• Running poetry install --with foo installs all dependencies that include foo in their groups (iow, ignores "optional=true" for those dependencies).
• Running poetry install --without foo omits all dependencies that include foo in their groups (iow, adds "optional=true" for those dependencies).
• Installing mypackage[foo] should do the same as running poetry install --with foo in the mypackage directory.
• It would be nice of installing mypackage[no-foo] would be equivalent to running poetry install --without-foo, but I suspect that this requires changes to setuptools or what not.

One thing that still needs some thought is how --with and --without interact when both are specified and match a single dependency (i.e. which gets precedence?)

This seems to offer a generic group feature, that cleanly maps onto the existing "extras" mechanism (as exposed to other packaging tooling such as pip), without duplicating information.

One could even go one step further and say that dev is just a group. So:

• Any dependencies in the dev-dependencies section are equivalent to regular dependencies with the "dev" group added.
• poetry install --no-dev is equivalent to poetry install --without dev
• poetry install is equivalent to poetry install --with dev

There might be some value in still storing dev dependencies in a separate section of the pyproject.toml file as now (easier to distinguish them), though just having a single list with explicit groups = ["dev"] would make it more obvious that dev deps are just a group.

As a last extra step, you could consider allowing negated groups, e.g.:

    some-package = {version = "^1.5.3", groups = ["!foo"]}

This would be a package that is normally installed, unless --with foo is
given. I can typically see this used for production-only dependencies,
something like:

    some-production-package = {version = "^1.5.3", groups = ["!dev"]}

This would be installed only when --no-dev or --without dev is specified
(which is slightly different from the foo example above, but that's because
--with dev is usually implied by poetry install.

Maybe this negation is one step too far and you should just add a "production"
group for this, though.

[SimonBiggs]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are?

So, within a given production dependency definition currently an "extra" parameter is allowed. When that is defined within the dependency parameters it refers to a pypi extra for that dependency, not for your package, eg. alldeps within scikit-learn refers to an extra of scikit-learn, not your package.

Let's say, your package is called "mypackage", and we want our package to have an optional dependency of scikit-learn within a group called ml (short for machine learning). And we want to make sure that when scikit-learn installs it installs with all of its own dependencies. I would expect this would look like:

[tool.poetry.dependencies]
scikit-learn = {version = "*", extras = ["alldeps"], groups = ["ml"]}

That way, running pip install mypackage[ml] would also install scikit-learn, and when scikit-learn is installed it would be installed as if pip install scikit-learn[alldeps] had been run.

Importantly, when groups is defined, I believe this implies an implicit optional=true. So optional should actually be replaced by the groups parameter, and no longer used.

[matthijskooijman]

Re-reading this discussion, I am still wondering what the essential difference between these new "groups" and the existing "extras" are?

So, within a given production dependency definition currently an "extra" parameter is allowed. When that is defined within the dependency parameters it refers to a pypi extra for that dependency, not for your package, eg. alldeps within scikit-learn refers to an extra of scikit-learn, not your package.

Yeah, I realize that that is what the extras in the tools.poetry.dependencies section mean, but I was referring to the other use of extras (i.e. the tool.poetry.extras section). But I think there is no essential difference between the proposed "groups" and the existing "extras" in this sense, except that the "groups" feature would be a feature superset of the existing "extras" (in that everything you can do with tool.poetry.extras can also be expressed using groups, and with my proposal above, one can in fact be mapped onto the other).

Importantly, when groups is defined, I believe this implies an implicit optional=true. So optional should actually be replaced by the groups parameter, and no longer used.

I do not think this is true. If you do this, then this "groups" concept becomes again the same as the existing "extras" concept, except with a different name. If you do this, you lose the ability to have dependencies that have a group set and are installed by default, but can be omitted using poetry install --without some-group (unless you require an explicit optional=false for this case, but that would be confusing IMHO).

It might be true that optional=true is useless without also specifying one or more groups (since IIUC, optional=true is not installed by default, and without a group, there is no way to cause such a dependency to become used).

[pawamoy]

@matthijskooijman I really like what you described above. I think it's very sensible.

• "group" dependencies can be written into the dev-dependencies section instead of the dependencies one (or both): it's less confusing (to me)
• the tool.poetry.extras section is automatically populated from groups: easier, less error-prone
• grouped dependencies don't have to be optional anymore, meaning new contributors don't have to read the pyproject.toml file to know what extras they should install using poetry install -E .... Everything just gets installed, like when doing poetry install.

I like it!

[matthijskooijman]

"group" dependencies can be written into the dev-dependencies section instead of the dependencies one (or both): it's less confusing (to me)

Do you mean "dependencies in the dev group" here? Or maybe I do not understand what you say exactly?

the tool.poetry.extras section is automatically populated from groups: easier, less error-prone

Do you mean that something would actually write them to the tool.poetry.extras section in the pyproject.toml file? Or just that this happens virtually, in memory, by poetry when interpreting the file?

[SimonBiggs]

I was referring to the other use of extras (i.e. the tool.poetry.extras section). But I think there is no essential difference between the proposed "groups" and the existing "extras" in this sense

Yup, I was not intending there to be any difference in the way I was wanting to use them. The aim would be to minimise inherent duplication of the package names and make a cleaner config API.

I personally was not proposing any feature changes, instead just what I see as an improved API.

I found manually managing the following lists was tedious and error prone, this is in contrast to adding group flags which I found to be simple and hard to make a mistake:

https://github.com/pymedphys/pymedphys/blob/ec837162da76016223e59ce0ab121147a00d0329/pyproject.toml#L98-L106

[pawamoy]

Do you mean "dependencies in the dev group" here? Or maybe I do not understand what you say exactly?

What I mean is that currently, to define extras, you have to put the dependencies in the tool.poetry.dependencies section, and set them as optional. You can also duplicate them in dev-dependencies if you want them to be installed by poetry install. With this feature, one could put the dependencies in tool.poetry.dev-dependencies instead.

Do you mean that something would actually write them to the tool.poetry.extras section in the pyproject.toml file? Or just that this happens virtually, in memory, by poetry when interpreting the file?

The latter, it would happen in memory, while building the wheel, etc., the pyproject.toml file would not be modified.

[Cielquan]

After some time and based on the discussion above from a week ago I changed my idea:

Currently we have the dev section with dev dependencies and the normal dependency section which includes the package's dependencies and optional dependencies which can be included via the extras section. IMO the extras section should be used to add additional dependencies which enable some additional functionality for the package and NOT the development. But currently the extras are _misused_ (by me e.g.) to also group dev dependencies. I use this to not always install e.g. the linter tools when I run unit tests (with tox) like I mentioned earlier.

So the primary thing I would like to achieve with the "group" feature is to add an extra like feature to the dev dependencies also.

With this you have the normal dependency section including the must have dependencies for the package to run and the optional ones which can be included via the extras like it is now. No change needed here.
The dev dependency section would be allowed to also have optional dependencies in it. Those would be, just like the extras from the normal section, grouped in another section like tool.poetry.groups to separate those from the package enhancing dependencies in the tool.poetry.extras section.
So this would be just a copy of the extras functionality to the dev dependencies to split them from the runtime dependencies, which is my personal primary goal here. This change shouldn't be that much work because it just copies existing behavior.

This basic change/step can be enhanced in multiple ways:

1. Change group behavior to opt-out instead of opt-in
   The extras for the runtime deps are opt-in and need to be explicitly named to be install, like it should be. But for the dev-dependencies I think it should be the other way around and the groups should be a opt-out feature. The reason for this is that if a new developer joins the team who likes to call pytest, linter etc. by hand instead of using for example a makefile or tox he/she would use only one venv I think. And to install all needed dependencies into this venv for developing the package he/she would have to look into the pyproject.toml file and add all the optional dev deps (groups) to the install command to have all the tools installed. So from a general perspective I think it would be good to install all dev dependencies and only opt-out some if explicitly said so in the command.
   This is a change I would recommend and like to see.

2. Relocate the groups definition (dev dependencies not runtime dependencies)
   So next thing could be to remove/relocate the tool.poetry.group section into the dev dependencies, because in conjunction with No. 1 all dev deps are installed by default and there is no optional=true anymore. So one could directly set the group(s) which should include the dependency at the dependency itself instead of writing them in a designated section. This is a design decision because you could say for both ways that they are more clear and better to read.
   I personally am fine with both ways.

3. Relocate the extras definition (runtime dependencies not dev dependencies)
   Next one could also remove/relocate the tool.poetry.extras section into the dependencies, like in No.2, because all runtime deps which are extras are also optional=true. So one could directly set the extras which should include the dependency instead of writing optional=true. This is also a design decision because you could say for both ways that they are more clear and better to read.
   I personally am fine with both ways.

4. Opt-in dev dependencies
   Also possible would be to leave the option to set optional=true in the dev dependency section to make a dev dependency opt-in counter to the default behavior of opt-out (see No.1). But I personally see no real use case for this and I think it would over-complicate things.
   Change my mind.

DISCLAIMER: The ideas above are not all mine but also taken from earlier comments. I just combined them to something I would like to see.

[anthem-bmk]

Could this be simplified by mirroring the extras functionality in dev-dependencies? extras:dependencies::dev-extras:dev-dependencies. It could work exactly the same and we could have a [tool.poetry.dev-extras] that defines the extras for dev-dependencies. You could then install them with:

poetry install  # installs main and dev dependencies
poetry install -E asdf  # installs main extra named "asdf"
poetry install -DE zxcv  # installs dev-extra named "zxcv"

My use case is to make it so poetry install just installs what's needed for most devs in our group. If you want to do something extra (e.g. do some debugging with dephell, or build docs) those dependencies right now are in extras, which feels a little awkward because it's exposed as a public extras

[Cielquan]

Could this be simplified by mirroring the extras functionality in dev-dependencies? extras:dependencies::dev-extras:dev-dependencies. It could work exactly the same and we could have a [tool.poetry.dev-extras] that defines the extras for dev-dependencies. You could then install them with:

poetry install  # installs main and dev dependencies
poetry install -E asdf  # installs main extra named "asdf"
poetry install -DE zxcv  # installs dev-extra named "zxcv"

My use case is to make it so poetry install just installs what's needed for most devs in our group. If you want to do something extra (e.g. do some debugging with dephell, or build docs) those dependencies right now are in extras, which feels a little awkward because it's exposed as a public extras

Well thats exactly what I meant with the basic change .. only difference is that I named your dev-extras group.

[anthem-bmk]

Ah, gotcha. That's what I thought at first but I started reading the numbered points below and they seem to argue for a more complicated setup with groups and with opt-out instead of opt-in, etc.

But I personally see no real use case for this and I think it would over-complicate things.
Change my mind.

My current use case might be a good example. Let's say you want to build your docs in CI. Most devs won't need to install sphinx and some plugins, which could add quite a few dependencies and take an extra minute or two to install. If you need to test a change to the config or do some debugging you could always poetry install -DE sphinx.

Or you might typically install with poetry, but want to support installing with pip. But you can't do pip install -e without a setup.py, which makes debugging a huge pain. You could convert it to setup.py using dephell, but in most cases you wouldn't need to do that, so there's another set of dependencies.

I think having a new developer type poetry install is a much better experience than poetry install --without dephell,sphinx,etc

I do think groups has its place as well and could add some cool functionality, but personally would like to see a basic dev-extras supported that won't be broken by additional features.

[Cielquan]

Well I see where you are coming from, because I do almost the same.

I use tox for automating creation and testing of my docs, unit testing, etc. (local and in CI) and for the docs venv I specify the docs extra which is like poetry install -E docs which would become poetry install -DE docs like you suggested.

But what I also do is I always create a venv via tox which i call dev into which I install ALL dependencies including package extras and dev-extras so I can do all I want from the command line, because this venv is set for the project interpreter in my IDE. Like this I can call tox -e docs to let tox create my docs like specified in my tox.ini or I can type the command myself to completely change the arguments etc.

My thought was that if someone works very basic with a venv like my dev venv one should not have to look into the pyproject.toml to get all needed dev dependencies. But both ways are completely valid.

My pyproject.toml example:
https://github.com/Cielquan/python_test/blob/b0b137084598e01fe0e40734976f6fbdc7d47565/pyproject.toml

My tox.ini example:
https://github.com/Cielquan/python_test/blob/b0b137084598e01fe0e40734976f6fbdc7d47565/tox.ini

EDIT: fix typo

[matthijskooijman]

TBH, I think the last few comments seem to overly complicate things (though maybe I also just did not completely understand them).

In any case, my main point is that things should IMHO be simple and regular. Currently, the "dev" dependencies are somewhat special, but I'd rather seem them generalized into "just another group" than adding more special stuff for them (such as a tool.poetry.dev-extras section). Also, I'd rather make things as regular as possible, so each dependency is essentially the same (can be part of a group or not, is installed by default or not), without arbitrarily limiting options to cater for the most common usecase, and without changing defaults for opt-in/opt-out/optional based on whether a dependency is a dev dependency.

Anyway, I'm also curious what the devs think of this, maybe not too much point in ongoing discussion here without some input from their end.

[Cielquan]

Anyway, I'm also curious what the devs think of this, maybe not too much point in ongoing discussion here without some input from their end.

Well thats a very good point TBH.

[nextmat]

In any case, my main point is that things should IMHO be simple and regular. Currently, the "dev" dependencies are somewhat special, but I'd rather seem them generalized into "just another group" than adding more special stuff for them (such as a tool.poetry.dev-extras section). Also, I'd rather make things as regular as possible, so each dependency is essentially the same (can be part of a group or not, is installed by default or not), without arbitrarily limiting options to cater for the most common usecase, and without changing defaults for opt-in/opt-out/optional based on whether a dependency is a dev dependency.

I totally agree with this.

Generalizable logic which allows common cases to be easy beats nested special case logic every time.

For precedent here is how Bundler handles this, which works quite well.

[Cielquan]

Anyway, I'm also curious what the devs think of this, maybe not too much point in ongoing discussion here without some input from their end.

Well thats a very good point TBH.

Just realized that this topic was put onto the roadmap #1856 für version 1.2 a while ago.

So when when 1.1 leaves beta and releases this issue should get the devs attention.

[roxchkplusony]

Could groups also influence the poetry build target, not just poetry install?