Poetry: Poetry takes longer than expected on resolving awscli

[rexwangcc]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 10.13.6
• Poetry version: Poetry 0.12.16
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/rexwangcc/0d6c06cd9a854e2f3aa15959cb6b51f1

Issue

First of all, thank you for all of your work to build this great tool :) I have been using it for a while for my personal projects and recently started to experimentally apply Poetry on some of our projects at work.

The TOML file, as you could see from the gist, is relatively simple, except that one of the deps is a git dependency. It seems the Poetry resolver takes a very long time on awscli (actually it didn't even finish until I ran out of my patience and stopped it, after 400s+). I'm using a clean Python env installed by pyenv, so I suspect it must be something related to the strict requirements of awscli(especially around coloroma), but cannot make sure. I'm currently letting it run for even longer to see if it can run through, but 400 secs to me is almost a deal breaker in the working development process.

Thanks!

stdout of poetry install -vv FYI:

1: fact: botocore (1.12.170) depends on python-dateutil (>=2.1,<3.0.0)
   1: fact: botocore (1.12.170) depends on jmespath (>=0.7.1,<1.0.0)
   1: fact: botocore (1.12.170) depends on docutils (>=0.10)
   1: fact: botocore (1.12.170) depends on urllib3 (>=1.20,<1.26)
   1: selecting botocore (1.12.170)
   1: derived: jmespath (>=0.7.1,<1.0.0)
   1: derived: python-dateutil (>=2.1,<3.0.0)
   1: selecting jmespath (0.9.4)
   1: fact: python-dateutil (2.8.0) depends on six (>=1.5)
   1: selecting python-dateutil (2.8.0)
   1: fact: boto3 (1.9.170) depends on botocore (>=1.12.170,<1.13.0)
   1: fact: boto3 (1.9.170) depends on jmespath (>=0.7.1,<1.0.0)
   1: fact: boto3 (1.9.170) depends on s3transfer (>=0.2.0,<0.3.0)
   1: selecting boto3 (1.9.170)
   1: fact: awscli (1.16.180) depends on botocore (1.12.170)
   1: fact: awscli (1.16.180) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.180) depends on docutils (>=0.10)
   1: fact: awscli (1.16.180) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.180) depends on PyYAML (>=3.10,<=5.1)
   1: fact: awscli (1.16.180) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.180)
   1: fact: awscli (1.16.179) depends on botocore (1.12.169)
   1: fact: awscli (1.16.179) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.179) depends on docutils (>=0.10)
   1: fact: awscli (1.16.179) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.179) depends on PyYAML (>=3.10,<=5.1)
   1: fact: awscli (1.16.179) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.179)
   1: fact: awscli (1.16.178) depends on botocore (1.12.168)
   1: fact: awscli (1.16.178) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.178) depends on docutils (>=0.10)
   1: fact: awscli (1.16.178) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.178) depends on PyYAML (>=3.10,<=5.1)
   1: fact: awscli (1.16.178) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.178)
   1: fact: awscli (1.16.177) depends on botocore (1.12.167)
   1: fact: awscli (1.16.177) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.177) depends on docutils (>=0.10)
   1: fact: awscli (1.16.177) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.177) depends on PyYAML (>=3.10,<=3.13)
   1: fact: awscli (1.16.177) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.177)
   1: fact: awscli (1.16.176) depends on botocore (1.12.166)
   1: fact: awscli (1.16.176) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.176) depends on docutils (>=0.10)
   1: fact: awscli (1.16.176) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.176) depends on PyYAML (>=3.10,<=3.13)
   1: fact: awscli (1.16.176) depends on s3transfer (>=0.2.0,<0.3.0)

...(1k+ lines)

   1: derived: not awscli (1.16.0)
   1: fact: awscli (1.15.85) depends on botocore (1.10.84)
   1: fact: awscli (1.15.85) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.15.85) depends on docutils (>=0.10)
   1: fact: awscli (1.15.85) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.15.85) depends on PyYAML (>=3.10,<=3.13)
   1: fact: awscli (1.15.85) depends on s3transfer (>=0.1.12,<0.2.0)
   1: derived: not awscli (1.15.85)
   1: fact: awscli (1.15.84) depends on botocore (1.10.83)
   1: fact: awscli (1.15.84) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.15.84) depends on docutils (>=0.10)
   1: fact: awscli (1.15.84) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.15.84) depends on PyYAML (>=3.10,<=3.13)
   1: fact: awscli (1.15.84) depends on s3transfer (>=0.1.12,<0.2.0)
   1: derived: not awscli (1.15.84)
   1: fact: awscli (1.15.83) depends on botocore (1.10.82)
   1: fact: awscli (1.15.83) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.15.83) depends on docutils (>=0.10)
   1: fact: awscli (1.15.83) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.15.83) depends on PyYAML (>=3.10,<=3.13)
   1: fact: awscli (1.15.83) depends on s3transfer (>=0.1.12,<0.2.0)

^C   1: Version solving took 409.142 seconds.
   1: Tried 1 solutions.

[jacebrowning]

The explanation was posted somewhere, but I can't find it right now.

Apparently the awscli packages are not published with metadata about its dependencies so poetry literally has to download each version to figure out what it depends on. This is amplified by the fact that there are hundreds of releases of awscli on PyPI. The resolution time is greatly reduced if you give poetry a starting point for awscli.

Try adding awscli = "^1.16.180" under [tool.poetry.dependencies] in pyproject.toml.

[rexwangcc]

@jacebrowning Thanks, adding awscli = "^1.16.180" worked!

However it's still not clear why poetry logs showed:

 1: fact: awscli (1.16.180) depends on botocore (1.12.170)
   1: fact: awscli (1.16.180) depends on colorama (>=0.2.5,<=0.3.9)
   1: fact: awscli (1.16.180) depends on docutils (>=0.10)
   1: fact: awscli (1.16.180) depends on rsa (>=3.1.2,<=3.5.0)
   1: fact: awscli (1.16.180) depends on PyYAML (>=3.10,<=5.1)
   1: fact: awscli (1.16.180) depends on s3transfer (>=0.2.0,<0.3.0)
   1: derived: not awscli (1.16.180)

to me. I assumed poetry downloaded and looked into this version at the beginning of the resolve process 🤔

[rexwangcc]

Closing this since I worked around the issue by following the suggestions above.

[kdaily]

Hi @jacebrowning,

Checking in from the AWS CLI. I'm wondering what you're referring to that is missing that could aid Poetry in the dependency resolution - is it the lack of a pyproject.toml? Thanks!