Poetry: How are you supposed to install optionals?

[dfee]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Question

I've got something similar to poetry's pyproject.toml file.

You can see my file here: aiohttp_session_ws

The problem is, now that black is optional, I can't get it to install. Should I only be using optional for packages that are "extras". And if so, why doesn't poetry install -E development then install black?

[sdispater]

Optional dependencies can only be opted-in by using extras, yes.

And regarding black not being installed: do you use at least Python 3.6? black is only compatible with Python 3.6 or later. And is ipdb installed when running poetry install -E development?

[dfee]

I was able to get black working (i had to set python = ... instead of optional = ....

However, I'm unable to get ipdb installed using poetry install -E development:

~/code/aiohttp_session_ws (aiohttp_session_ws/.venv)
$ poetry install -E development
Installing dependencies from lock file

Nothing to install or update


~/code/aiohttp_session_ws (aiohttp_session_ws/.venv)
$ pip list | grep ipdb
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[salotz]

This doesn't help for installing these packages with pip though on the consumer side. You would have to produce a 'requirements.txt' with named extras like in PEP 508.

[nickpresta]

EDIT I've submitted a PR that fixes the issue.

I have the same (or similar) issue:

pyproject.toml

[tool.poetry]
name = "mything"
version = "0.1.0"
description = "description"
authors = ["nick"]

[tool.poetry.dependencies]
python = "^3.6 || ^3.7"

[tool.poetry.dev-dependencies]
pytest = "^3"
black = { version = "^18.9b0", optional = true }
mypy = { version = "^0.641.0", optional = true }
typing-extensions = { version = "^3.6", optional = true }

[tool.poetry.extras]
developer = ["black", "mypy", "typing-extensions"]

When I run poetry install -v, I get:

$ poetry install -v
Creating virtualenv mything-py3.7 in /Users/npresta/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/npresta/Library/Caches/pypoetry/virtualenvs/mything-py3.7
Updating dependencies
Resolving dependencies... (0.6s)


Package operations: 7 installs, 0 updates, 0 removals

Writing lock file

  - Installing six (1.11.0)
  - Installing atomicwrites (1.2.1)
  - Installing attrs (18.2.0)
  - Installing more-itertools (4.3.0)
  - Installing pluggy (0.8.0)
  - Installing py (1.7.0)
  - Installing pytest (3.10.0)

When I run poetry show -v:

$ poetry show -v
Using virtualenv: /Users/npresta/Library/Caches/pypoetry/virtualenvs/mything-py3.7
appdirs           1.4.3  A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir".
atomicwrites      1.2.1  Atomic file writes.
attrs             18.2.0 Classes Without Boilerplate
black             18.9b0 The uncompromising code formatter.
click             7.0    Composable command line interface toolkit
more-itertools    4.3.0  More routines for operating on iterables, beyond itertools
mypy              0.641  Optional static typing for Python
mypy-extensions   0.4.1  Experimental type system extensions for programs checked with the mypy typechecker.
pluggy            0.8.0  plugin and hook calling mechanisms for python
py                1.7.0  library with cross-python path, ini-parsing, io, code, log facilities
pytest            3.10.0 pytest: simple powerful testing with Python
six               1.11.0 Python 2 and 3 compatibility utilities
toml              0.10.0 Python Library for Tom's Obvious, Minimal Language
typed-ast         1.1.0  a fork of Python 2 and 3 ast modules with type comment support
typing            3.6.6  Type Hints for Python
typing-extensions 3.6.6  Backported and Experimental Type Hints for Python 3.5+

The packages appdirs, black, click, mypy, mypy-extensions,toml,typed-ast,typing,typing-extensions` are red instead of green.

When I run poetry install -v -E developer I get:

$ poetry install -v -E developer
Using virtualenv: /Users/npresta/Library/Caches/pypoetry/virtualenvs/mything-py3.7
Installing dependencies from lock file

Nothing to install or update

and poetry run pip freeze shows:

$ poetry run pip freeze
atomicwrites==1.2.1
attrs==18.2.0
more-itertools==4.3.0
pluggy==0.8.0
py==1.7.0
pytest==3.10.0
six==1.11.0

HOWEVER, I found a manual fix. When I look at poetry.lock, there is a section that looks like this:

[extras]
developer = []

When I change that list to:

[extras]
developer = ["mypy", "black", "typing-extensions"]

and run poetry install -v -E developer I get:

$ poetry install -v -E developer
Using virtualenv: /Users/npresta/Library/Caches/pypoetry/virtualenvs/mything-py3.7
Installing dependencies from lock file


Package operations: 9 installs, 0 updates, 0 removals

  - Installing appdirs (1.4.3)
  - Installing click (7.0)
  - Installing mypy-extensions (0.4.1)
  - Installing toml (0.10.0)
  - Installing typed-ast (1.1.0)
  - Installing typing (3.6.6)
  - Installing black (18.9b0)
  - Installing mypy (0.641)

and poetry show works as expected (all 9 packages are green).

It seems the initial extras are not being written to the lock file when the dependencies appear as part of dev-dependencies.

When I move the requirements from dev-dependencies to dependencies, things work as expected:

$ cat pyproject.toml
[tool.poetry]
name = "mything"
version = "0.1.0"
description = "description"
authors = ["nick"]

[tool.poetry.dependencies]
python = "^3.6 || ^3.7"
black = { version = "^18.9b0", optional = true }
mypy = { version = "^0.641.0", optional = true }
typing_extensions = { version = "^3.6", optional = true }

[tool.poetry.dev-dependencies]
pytest = "^3"

[tool.poetry.extras]
developer = ["black", "mypy", "typing_extensions"]

$ poetry install -v -E developer
Using virtualenv: /Users/npresta/Library/Caches/pypoetry/virtualenvs/mything-py3.7
Installing dependencies from lock file


Package operations: 9 installs, 0 updates, 0 removals

  - Installing appdirs (1.4.3)
  - Installing click (7.0)
  - Installing mypy-extensions (0.4.1)
  - Installing toml (0.10.0)
  - Installing typed-ast (1.1.0)
  - Installing typing (3.6.6)
  - Installing black (18.9b0)
  - Installing mypy (0.641)
  - Installing typing-extensions (3.6.6)

I believe this is a bug. We should be able to define extras that depend on packages only in dev-dependencies. I have a straightforward fix that I'm working on and expect to submit today.

[sdispater]

Like I said in the associated PR this is not something that will be supported for the reasons explained here: https://github.com/sdispater/poetry/pull/606#issuecomment-437943927

[mikenerone]

As it appears to be the case that the use of extras is required for all optional dependencies in order in order to be able to opt-in, I think the phrasing of this comment in the docs at https://python-poetry.org/docs/pyproject/ need to be changed. Note the word "some", which implies "not all".

# A list of all of the optional dependencies, some of which are included in the
# below `extras`. They can be opted into by apps.

It's confusing at best.