Poetry: SolverProblemError when installing aiohttp

[Victor-Savu]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Lubuntu 18.04
• Poetry version: 0.11.1
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Victor-Savu/e26e28cffd506bc399b5e0fae19dee62

Issue

Poetry incorrectly identifies idna-ssl to be a dependency of aiohttp when the required python is 3.7.
Here is the result of poetry lock -vvv.

Here is a Dockerfile which can be used together with the linked pyproject.toml file to reproduce the error:

FROM python:3.7.0b5-alpine
RUN apk add --no-cache curl \
 && curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python3
ADD pyproject.toml .
RUN poetry lock -vvv

The setup.py of aiohttp only includes the dependency on idna-ssl for versions lower than 3.7:

if sys.version_info < (3, 7):
    install_requires.append('idna-ssl>=1.0')

install_requires is later on passed to setuptools.setup(...).

idna-ssl indeed only supports python versions <3.7 but it should not be included as a dependency to begin with.

[sdispater]

aiohttp should use environment markers to tell that idna-ssl is only needed for Python < 3.7 (see PEP 508).

Poetry never executes setup.py to get dependency information for obvious security reasons and rather inspect either wheels or sdist tarballs. In the sdist, there is a file (requires.txt) which content is:

attrs>=17.3.0
chardet<4.0,>=2.0
multidict<5.0,>=4.0
async_timeout<4.0,>=3.0
yarl<2.0,>=1.0
idna-ssl>=1.0

There is no environment marker so Poetry tries to resolve idna-ssl which is only compatible with Python <3.7, hence the conflict you encounter.

There is nothing Poetry can do and should do for packages like this. Poetry tries to enforce good practices and defining dependencies only compatible with a subset of python versions programmatically is not one.

The problem lies upstream so you should report it there.

[Victor-Savu]

@sdispater Thanks for the quick response! I was wondering if that was going to be the case :) I'll report (and submit PR) upstream.
Thanks for being awesome! :1st_place_medal:

[Victor-Savu]

@sdispater Your explanation helped me find a workaround at least until the problem is fixed upstream. You got me wondering where poetry gets the requirements from if there is no sdist ready to go (such as when you install from git). Spoiler: installing from git works :)

Dockerfile:

FROM python:3.7.0b5-alpine
RUN apk add --no-cache curl git \
 && curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python3
ADD pyproject.toml .
RUN poetry lock -vvv

pyproject.toml:
```toml[tool.poetry]
name = "poet-fail"
version = "0.1.0"
description = ""
authors = ["The Author [e-mail]"]

[tool.poetry.dependencies]
python = "3.7"
aiohttp = { git = "https://github.com/aio-libs/aiohttp.git", branch = "v3.3.2" }
```

[Victor-Savu]

In the sdist, there is a file (requires.txt) which content is:

I am sorry, I couldn't find the file you are referring to. Could you please share a link? It would really help me with reporting the bug upstream to aiohttp.

I checked the sdist here on the last link

I only found a few requirements files in the requirements directory, and those do specify the correct PEP 508 markers, as can be seen in the requirements/ci-wheel.txt:

attrs==18.1.0
async-generator==1.9
async-timeout==3.0.0
brotlipy==0.7.0
cchardet==2.1.1
chardet==3.0.4
coverage==4.5.1
cython==0.28.3
flake8==3.5.0
gunicorn==19.8.1
isort==4.3.4
pyflakes==2.0.0
multidict==4.3.1
pytest==3.6.0
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest-xdist==1.22.2
towncrier==18.5.0
tox==3.0.0
twine==1.11.0
yarl==1.2.4

# Using PEP 508 env markers to control dependency on runtimes:
aiodns==1.1.1; platform_system!="Windows"  # required c-ares will not build on windows
codecov==2.0.15; platform_system!="Windows"  # We only use it in Travis CI
uvloop==0.9.1; platform_system!="Windows" and implementation_name=="cpython" and python_version<"3.7" # MagicStack/uvloop#14
idna-ssl==1.0.1; python_version<"3.7"

Could it be that this file is not referenced correctly in the sdist?

[sdispater]

Sorry I was not clear, the requires.txt file is in the *.dist-info directory. And if Poetry can't find it it will inspect the wheel(s) if necessary.

It's a file generated by the python setup.py sdist command I think. Poetry avoids using the requirements.txt (or equivalent) since those do not necessarily represent the library's dependency.

Also, the reason it works from the git repository is that in this case poetry executes python setup.py egg-info to get the information. But even then it's not prefect in the case of aiohttp: if you generate the lock file from Python 3.7 but your project is also compatible with Python < 3.7 idna-ssl will never appear. That's why environment markers are so important.

[Victor-Savu]

I see now, thanks! I looked at the wheel and indeed the METADATA file reads:

Requires-Python: >=3.5.3
Requires-Dist: attrs (>=17.3.0)
Requires-Dist: chardet (<4.0,>=2.0)
Requires-Dist: multidict (<5.0,>=4.0)
Requires-Dist: async-timeout (<4.0,>=3.0)
Requires-Dist: yarl (<2.0,>=1.0)
Requires-Dist: idna-ssl (>=1.0)

So the markers don't make it to the wheel. I'm now ready to open the issue upstream. Thanks for the help!

[Victor-Savu]

Hmm.. there still seems to be an issue: The wheel is not yet released for python 3.7 So poetry should not really take it into consideration. (I was hinted by this issue https://github.com/aio-libs/aiohttp/issues/3111). I hadn't noticed this when I wrote the previous comment.

In this case, should poetry ignore the wheel so as to not install the wrong dependencies? I mean, if idna-ssl didn't request python <3.7 poetry would have installed it even though there was no indication anywhere that it was a dependency (again, because the wheel is not for py37).

[sdispater]

In this particular case, Poetry uses the sdist since there is not wheel matching and this sees idna-ssl (>=1.0).

There are still some improvements to be made in the logic Poetry uses when it needs to fallback on downloading the distributions (wheels and sdists). However, it should work with properly packaged and PEP-compliant packages.

[sdispater]

And it seems someone already opened a PR: https://github.com/aio-libs/aiohttp/pull/3114

[Victor-Savu]

This particular problem was solved upstream. Thanks for the support and thanks for your great work with poetry!

[Yanpas]

Had a problem with multidict:

 $ poetry update
Updating dependencies
Resolving dependencies... (0.2s)


Package operations: 7 installs, 0 updates, 0 removals

Writing lock file

  - Installing multidict (4.5.2)

[EnvCommandError]
Command ['pip', 'install', '--no-deps', 'multidict==4.5.2'] errored with
the following output:
Collecting multidict==4.5.2
  Could not find a version that satisfies the requirement multidict==4.5.
2 (from versions: 1.0.0a0, 1.0.1, 1.0.2, 1.0.3, 1.1.0b1, 1.1.0b2, 1.1.0b4
, 1.1.0b5, 1.1.0b6, 1.1.0b7, 1.1.0b8, 1.1.0, 1.2.0, 1.2.1, 1.2.2, 2.0.0,
2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.1.6, 3.0.0, 3.1.0, 3.1
.1, 3.1.3a0, 3.1.3, 3.1.4a1, 3.1.4a2, 3.1.4a3, 3.2.0a3, 3.2.0, 3.2.1a0, 3
.3.0, 3.3.1)
No matching distribution found for multidict==4.5.2

After rm -rf ~/.cache/pypoetry/ the error is gone.

toml file is simple:

[tool.poetry]
name = "xxx"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.6"
peewee = "^3.8"
aiohttp = "~3.5"

[tool.poetry.dev-dependencies]
pytest = "^3.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"