Poetry: Extras with dev-dependences are not recognised by `pip install`

Created on 10 Nov 2020 · 4Comments · Source: python-poetry/poetry

[x] I am on the latest Poetry version.
[ ] ~I have searched the issues of this repo and believe that this is not a duplicate.~ _This is a duplicate to #1145._
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS 10.15.7
Poetry version: 1.1.4
Python version: 3.8.6

Issue

This is a duplicate of #1145 but, given how confusing the original thread has become, I opted to created a new issue. I'm experiencing the exact same problem as @devkral (original thread's OP), namely: pip install .[docs] does not install extras listed under [tool.poetry.extras] when the corresponding package are listed under tool.poetry.dev-dependencies.

As @paveldedik noted in the original thread, this issue disappears when the extra dependencies are moved to tool.poetry.dependencies.

Setup

Here is my pyproject.toml (real world project, thus the verbosity):

[tool.poetry]
# ...

[tool.poetry.dependencies]
python = ">=3.6, <3.9"
attrs = "~20.3.0"
click = "~7.1.2"
click-plugins = "~1.1.1"
matplotlib = "~3.3.2"
scipy = "~1.5.4"
Shapely = {extras = ["vectorized"], version = "~1.7.1"}
svgwrite = "~1.4"
toml = "~0.10.2"
svgpathtools = {git = "https://github.com/abey79/svgpathtools", rev = "vpype"}

[tool.poetry.dev-dependencies]
pytest = "^6.1.2"
black = "^20.8b1"
isort = "^5.6.4"
# the following must be moved to [tool.poetry.dependencies] to be recognised by pip
Sphinx = { version = "^3.3.0", optional = true }
sphinx-click = {git = "https://github.com/abey79/sphinx-click", rev = "vpype-docs", optional = true}
sphinx-autodoc-typehints = { version = "^1.11.1", optional = true }
sphinx-rtd-theme = { version = "^0.5.0", optional = true }

[tool.poetry.extras]
docs = ["Sphinx", "sphinx-click", "sphinx-autodoc-typehints", "sphinx-rtd-theme"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Expected behaviour

$ pip install .[docs]
[...] output indicating that pip tries to install sphinx, sphinx-click, sphinx-autodoc-typehints, sphinx-rtd-theme

Actual behaviour

pip install .[docs] does not attempt to install docs packages.

Workaround

Move docs packages to [tool.poetry.dependencies], then pip install .[docs] will correctly attempt (and succeed in this case) to install said packages.

Bug Triage

Source

abey79

All 4 comments

Looks like a feature request to me. Not a bug report. As far as I know _poetry_ never meant to allow _extras_ (i.e. optional dependencies) to be listed in dev-dependencies.

From my point of view, if there is a bug, it is that poetry should complain when optional dependencies are listed in dev-dependencies.

sinoroc on 10 Nov 2020

I should have clarified my use-case upfront. I'm trying to get my poetry-migrated project's .readthedocs.yml to work and, unfortunately, arbitrary installation commands are not allowed. Only requirements.txt or pip installation method (with optional extras) are possible. Since I'd rather avoid having a docs-only requirements.txt (pyproject.toml should have it all), I'm left with the pip + extras option. Although the workaround is somewhat unclean (CI stuff is listed in _dependencies_), at least everything is in pyproject.toml and there are no real detrimental side effects.

I understand the point that extras being somehow user-facing (through inclusion in distribution metadata), taking them from _dev-dependencies_ is meaningless as the latter should be discarded from distribution. IMO this should be clarified in the documentation and, well, some error reporting wouldn't be a bad thing either. I guess clarifying this in #1145 as well would be good as well since the issue is still opened.

abey79 on 10 Nov 2020

I totally understand. If it can comfort you in your decision, placing such dependencies that are only needed to build the documentation (or run tests, for another example) in an _extra_ is a very common pattern and that would be my recommendation as well (I usually name that _extra_ dev_doc to make it clear that is something for development only). See examples:

But of course, in _poetry_ it is not that simple. Since by default, for your own development workflow, poetry install will not install this dev_doc extra, so you can not build documentation immediately. You would need poetry install --extras dev_doc first.

Related discussion from the point of view of _readthedocs_ (there are suggestions in there):

https://github.com/readthedocs/readthedocs.org/issues/4912

One of the noteworthy suggestions, I have not looked into it, I do not know if it is recommendable:

https://github.com/chrahunt/rtd-poetry/

sinoroc on 10 Nov 2020

👍1

I totally agree with @sinoroc. The dev-dependencies are only meant for dependencies during development with poetry. So if you use pip for installing the package, only the normal dependencies will be presented to pip to install.

What you can do is, defining to dependencies in the dependencies section and dev-dependencies section, like this:

[tool.poetry.dependencies]
python = ">=3.6, <3.9"
attrs = "~20.3.0"
click = "~7.1.2"
click-plugins = "~1.1.1"
matplotlib = "~3.3.2"
scipy = "~1.5.4"
Shapely = {extras = ["vectorized"], version = "~1.7.1"}
svgwrite = "~1.4"
toml = "~0.10.2"
svgpathtools = {git = "https://github.com/abey79/svgpathtools", rev = "vpype"}
Sphinx = { version = "^3.3.0", optional = true }
sphinx-click = {git = "https://github.com/abey79/sphinx-click", rev = "vpype-docs", optional = true}
sphinx-autodoc-typehints = { version = "^1.11.1", optional = true }
sphinx-rtd-theme = { version = "^0.5.0", optional = true }

[tool.poetry.dev-dependencies]
pytest = "^6.1.2"
black = "^20.8b1"
isort = "^5.6.4"
Sphinx = { version = "^3.3.0"}
sphinx-click = {git = "https://github.com/abey79/sphinx-click", rev = "vpype-docs"}
sphinx-autodoc-typehints = { version = "^1.11.1"}
sphinx-rtd-theme = { version = "^0.5.0"}

[tool.poetry.extras]
docs = ["Sphinx", "sphinx-click", "sphinx-autodoc-typehints", "sphinx-rtd-theme"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Than poetry will install the dev-dependencies on poetry install. When using pip the optional packages will only gets installed if you select the corresponding extra.

fin swimmer