Poetry: Extras of a dependency's dependency are not installed

[hukkinj1]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04
• Poetry version: 0.12.17 and 1.0.0b5
• Link of a Gist with the contents of your pyproject.toml file: https://github.com/hukkinj1/bug-poetry-dependency-extras/blob/master/pyproject.toml

Issue

Poetry does not install extras of a dependency's dependency, if the dependency's dependency is also referenced without the extra.

Here's the pyproject.toml. The generated poetry.lock lockfile is here.

Pyproject.toml requires PyOTA package, which requires requests[security]. The security extra is not installed though. This can be observed by the fact the the lockfile has no pyOpenSSL entry, which is a part of the security extra of requests.

Note that we remove requests = "*" from pyproject.toml, then the security extra will be installed as expected. It seems the issue only occurs when the package that needs extras is referenced elsewhere with no mention of the extras.

[sdispater]

This is a tricky one.

I can definitely reproduce and I know why that happens but actually fixing this is not trivial. Unfortunately, it's unlikely that it will be fixed before the 1.0.0 release because it will need a tweak in the resolution logic.

For the time being, a workaround is to declare your dependency on requests like the following:

requests = {"version": "*", extras = ["security"]}

[hukkinj1]

Thanks @sdispater for having a look.

There is a case where not even the workaround works. I'm not sure how related this is, maybe I should file another bug report for this...?

Anyways I'll try to briefly explain it here: Let's imagine a world where the PyOTA package we required has not been released on PyPI, but only on a git server. Then our pyproject.toml (with the workaround) will look something like:

requests = { version = "*", extras = [ "security" ] }
PyOTA = { git = "https://github.com/iotaledger/iota.py", tag = "2.1.0" }

Running poetry install will then create a perfect poetry.lock that includes the requests[security] extra. However, the next time we run poetry update what happens is, that the security extra gets removed (copy-pasting command line output here):

$ poetry update
Updating dependencies
Resolving dependencies... (3.8s)

Writing lock file


Package operations: 0 installs, 1 update, 4 removals

  - Updating PyOTA (0.0.0 3057a1b -> 2.1.0 2.1.0)
  - Removing cffi (1.13.2)
  - Removing cryptography (2.8)
  - Removing pycparser (2.19)
  - Removing pyopenssl (19.1.0)

Following poetry update calls will not bring the security extra back.

The obvious workaround for this is to require the packages of the security extra explicitly, but this gets very awkward if the extra contains many packages.

P.S. Thanks for poetry! Despite these issues it's still an amazing tool.

[sdispater]

The only way we can solve the extras issues currently is by always resolving the extra dependencies even though they were not opted in. That would fix the issue but it might introduce another which is that resolution conflict might be raised by extra dependencies even though they were never opted in.

I don't see a perfect solution for this at the moment. The Python ecosystem is so unique and complex (some might say convoluted) that coming up with a perfect dependency resolver is almost impossible and will sometimes require the help of the end user by hinting it on how it should proceed.

[absassi]

@sdispater I haven't looked at how the code works, but do you really need to resolve all extras to fix this? It seems like you can keep a list of required extras when resolving a dependency, and if that dependency is required again by some other path you don't skip it or resolve it again from scratch, but just check if there are new extras and only resolve them. You may need to keep a list of required extras for each package to append to whenever you resolve an extra, and you may need some work on how you resolve a dependency in order to be able to do more work on it afterwards (i.e. resolve more extras), but seems doable.

Another (maybe simpler) alternative, is to consider requests[security] as a new virtual dependency, which has the same versions available as requests, but installs no package and depends on the concatenated lists of requirements for requests and for security extra. In other words, when you find requests = {version = "x", extras = ["security"]} you understand that as "requests[security]" = {version = "x"} and know that to find the metadata of requests[security], you look at the requests metadata, but take the combined requirements from requests and from its security extra instead. The current resolver should be able to resolve this correctly and without any false-positive conflicts. You can pre-create all these virtual extra packages as soon as you parse the metadata of requests if you wish, but only really require these virtual dependencies when you see them referenced somewhere.

[daniellehanks]

Having this problem when installing a google lib that does not require the grpcio-gcp extra on google-api-core (e.g. google-cloud-bigtable), then installing one that does (e.g. google-cloud-spanner). The grpcio-gcp extra does not get added unless the adds are done with the one requiring grpcio-gcp first.

[mvoitko]

Still relevant to me.
Poetry version 1.1.2

poetry install "google-cloud-bigquery[pandas]" 

google-cloud-bigquery is installed. pandas is not.

[abn]

@mvoitko can you please try 1.1.3? There is also an issue with nested extras that will be resolved later with 1.1.4.

[mvoitko]

@abn updating helped