Poetry: Incorrect python version checking

[shenek]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Linux roadwarrior2 5.4.0-3-amd64 #1 SMP Debian 5.4.13-1 (2020-01-19) x86_64 GNU/Linux
• Poetry version: 1.0.5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/shenek/50cfa373695a917d91ca06873660b965

Issue

I tried to install dependency which uses setuptools and python_requires='>=3.6'.
But this seems to be in conflict with python requirement. Of my pyproject.toml.

[tool.poetry.dependencies]
python = "^3.6"

It causes:

poetry update
Updating dependencies
Resolving dependencies... (0.3s)

[SolverProblemError]
The current project's Python requirement (^3.6.0) is not compatible with some of the required packages Python requirement:
  - djangorestframework-simplejwt requires Python >=3.6,<3.9

Because djangorestframework-simplejwt (4.4.0) requires Python >=3.6,<3.9
 and no versions of djangorestframework-simplejwt match >4.4.0,<4.5.0, djangorestframework-simplejwt is forbidden.
So, because permission-service depends on djangorestframework-simplejwt (~4.4.0), version solving failed.
https://gist.github.com/shenek/50cfa373695a917d91ca06873660b965

And I'm using python version 3.7 which should match here.

[shenek]

As a workaround you can:

[tool.poetry.dependencies]
python = ">=3.6,<3.9"

[ZelphirKaltstahl]

I can confirm this. I have the same issue, but the workaround does not work for me either. Basically it is unusable for me currently.

[abn]

@shenek I might be mistaken, but isn't the solver doing what it is supposed to be doing here? Which is, raising an error because the dependencies have a stronger constraint on the python version?

The project python version ^3.6 implies a constraint >=3.6.0 and <4.0.0. This is much more relaxed than what the dependency requires >=3.6.0 and <3.9.0. Implying there is no version of the dependency that will work with all the versions the project specifies.

I believe that the workaround might be the right fix here.

Adding debug logs as it might be useful for further triage.

Updating dependencies
Resolving dependencies...
   1: fact: dep-error is 0.1.0
   1: derived: dep-error
   1: fact: dep-error depends on djangorestframework-simplejwt (~4.4.0)
   1: fact: dep-error depends on pytest (^5.2)
   1: fact: dep-error depends on pytest (^5.2)
   1: selecting dep-error (0.1.0)
   1: derived: pytest (^5.2)
   1: derived: djangorestframework-simplejwt (~4.4.0)
PyPI: 13 packages found for pytest >=5.2,<6.0
   1: fact: pytest (5.4.1) depends on py (>=1.5.0)
   1: fact: pytest (5.4.1) depends on packaging (*)
   1: fact: pytest (5.4.1) depends on attrs (>=17.4.0)
   1: fact: pytest (5.4.1) depends on more-itertools (>=4.0.0)
   1: fact: pytest (5.4.1) depends on pluggy (>=0.12,<1.0)
   1: fact: pytest (5.4.1) depends on wcwidth (*)
   1: fact: pytest (5.4.1) depends on importlib-metadata (>=0.12)
   1: fact: pytest (5.4.1) depends on atomicwrites (>=1.0)
   1: fact: pytest (5.4.1) depends on colorama (*)
   1: selecting pytest (5.4.1)
   1: derived: colorama (*)
   1: derived: atomicwrites (>=1.0)
   1: derived: importlib-metadata (>=0.12)
   1: derived: wcwidth (*)
   1: derived: pluggy (>=0.12,<1.0)
   1: derived: more-itertools (>=4.0.0)
   1: derived: attrs (>=17.4.0)
   1: derived: packaging (*)
   1: derived: py (>=1.5.0)
PyPI: No release information found for djangorestframework-simplejwt-1.5, skipping
PyPI: 1 packages found for djangorestframework-simplejwt >=4.4.0,<4.5.0
   1: fact: djangorestframework-simplejwt (4.4.0) requires Python >=3.6,<3.9
   1: derived: not djangorestframework-simplejwt (4.4.0)
   1: fact: no versions of djangorestframework-simplejwt match >4.4.0,<4.5.0
   1: conflict: no versions of djangorestframework-simplejwt match >4.4.0,<4.5.0
   1: derived: not djangorestframework-simplejwt (>4.4.0,<4.5.0)
   1: conflict: djangorestframework-simplejwt (4.4.0) requires Python >=3.6,<3.9
   1: ! djangorestframework-simplejwt (4.4.0) is partially satisfied by not djangorestframework-simplejwt (>4.4.0,<4.5.0)
   1: ! which is caused by "no versions of djangorestframework-simplejwt match >4.4.0,<4.5.0"
   1: ! thus: djangorestframework-simplejwt is forbidden
   1: ! djangorestframework-simplejwt (>=4.4.0,<4.5.0) is satisfied by djangorestframework-simplejwt (~4.4.0)
   1: ! which is caused by "dep-error depends on djangorestframework-simplejwt (~4.4.0)"
   1: ! thus: version solving failed
   1: Version solving took 0.078 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
The current project's Python requirement (>=3.6) is not compatible with some of the required packages Python requirement:
  - djangorestframework-simplejwt requires Python >=3.6,<3.9

Because djangorestframework-simplejwt (4.4.0) requires Python >=3.6,<3.9
 and no versions of djangorestframework-simplejwt match >4.4.0,<4.5.0, djangorestframework-simplejwt is forbidden.
So, because dep-error depends on djangorestframework-simplejwt (~4.4.0), version solving failed.

[shenek]

@abn
I'm not sure how the dependency solver should work. But it seems like a bit strange behavior.

Imagine that my deps look like this:

A = "*"
B = "<2.0.0"

and the A requirements are:

B = "<1.0.0"

So the packages are not installed because the A requirements are stricter than mine.

But currently there is a combination that would match the requirements.

A 9.9.9 (latest)
B 0.9.9

And the solver was unable to find it.

Note that I'm still considering python as an ordinary dependency here.

[abn]

I cannot comment on the expected behaviour with authority unfortunately. From the code, what I can tell is that the solver takes atleast some inspiration from https://github.com/dart-lang/pub/blob/master/doc/solver.md - someone from the core team will need to address that.

The above example might not be representative of the issue here. I believe that the scenario above will be solved without issue.

Considering the python version as just another dependency might be misleading here.

In original scenario, we are saying;

1. Package dep_error is compatible with all python versions >=3.6.0 <4.0.0
2. Package dep_error requires djangorestframework-simplejwt (~4.4.0)

However, since djangorestframework-simplejwt (~4.4.0) is only compatible with python versions >=3.6.0 <3.9.0, (1) is no longer true. One should infer that the dep_error will only work with python versions >=3.6.0 <3.9.0 by virtue of its mandatory dependency working only with those versions.

I guess poetry's default behaviour could be to handle this scenario by implicitly updating compatible python versions if the upper bound is not specified. Personally, I'd prefer it being explicit because the package will not work with python 3.9 unless the dependency releases a new version that is compatible with 3.9 - which would mean an update to the lock file anyway.

[ZelphirKaltstahl]

However, since djangorestframework-simplejwt (~4.4.0) is only compatible with python versions >=3.6.0 <3.9.0, (1) is no longer true. One should infer that the dep_error will only work with python versions >=3.6.0 <3.9.0 by virtue of its mandatory dependency working only with those versions.

Yes, that is what I also expected. Basically I expected Poetry to make every incompatibility narrow down the usable versions, until there is no usable version left for one of the specified packages and only then give an error, as there is no possibility to reconcile versions of packages.

However, the bug only happens since I upgraded Poetry. I don't know what version I had before, but something definitely changed, because before it worked just fine.

[abn]

@ZelphirKaltstahl was it the exact same dependency? Or something else, if so do you have a minimal pyproject.toml that you could share? It could also be that the broken behavior was fixed in the new release.

[ZelphirKaltstahl]

@abn Sorry for the late reply. I have something like 10-20 environments on my machine and I don't know which one it was. If I find a minimal example, I'll share it here.

[ZelphirKaltstahl]

I found one where the error happens:

[tool.poetry]
name = "envy the little env"
version = "0.1.0"
description = "my new env"
authors = ["me"]

[tool.poetry.dependencies]
python = "^3.6"
pandas = "^1.0.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

$ poetry --version
/home/user/.poetry/lib/poetry/_vendor/py2.7/subprocess32.py:149: RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads.
  "program uses threads.", RuntimeWarning)
Poetry version 1.0.5

[finswimmer]

Hello,

@abn is correct. poetry expect that all dependencies are defined in a way, that they are compatible to all python versions to which the project is compatible. Independent from the currently used python version. Moreover it expect to find a major version of a dependency that is compatible to the python version. And that's the part that is tricky with pandas.

pandas 1.0 needs a python >=3.6.1 < 4.0. For python 3.6.0 you can only use a version <1.0. So there are two ways to solve this.

1. Make your project compatible for python >=3.6.1 <4.0

[tool.poetry.dependencies]
python = "^3.6.1"

1. Define what version of pandas should used for python 3.6.0

[tool.poetry.dependencies]
python = "^3.6"
pandas = [
    { version = "^1.0", python = "^3.6.1" },
    { version = "<1.0", python = "<3.6.1" }
]

fin swimmer

[ZelphirKaltstahl]

Ah OK, I think that I understand it better now. I would like to add, that this behavior is unusual for anyone having used requirements.txt, pipenv or npm before. Which does not need to be a bad thing. Here is a suggestion: Perhaps we could take this as an example and put it in the readme, somewhere more visible, so that people do not come with different expectations and do not hit this problem.