Poetry: Failed to install packages that target a range of python versions if local python version is not fixed

[MarcBoissonneault]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

▶ poetry debug:info

Poetry
======

 * Version: 0.12.17
 * Python:  3.7.0


Virtualenv
==========

 * Python:         3.7.0
 * Implementation: CPython
 * Path:           /Users/user/projectX/.venv
 * Valid:          True


System
======

 * Platform: darwin
 * OS:       posix
 * Python:   /Users/user/.pyenv/versions/3.7.0

pyproject.toml

[tool.poetry]
name = "projectX"
version = "0.1.0"
description = ""
authors = ["user <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

When I try to install a dependency that target a range of python versions, that seems to contains the python version I target in my pyproject.toml it fails with the following error.

▶ poetry add kedro -vvv
Using virtualenv: /Users/user/projectX/.venv
PyPI: 7 packages found for kedro *
Using version ^0.15.1 for kedro

Updating dependencies
Resolving dependencies...
   1: fact: projectX is 0.1.0
   1: derived: projectX
   1: fact: projectX depends on kedro (^0.15.1)
   1: selecting projectX (0.1.0)
   1: derived: kedro (^0.15.1)
PyPI: 1 packages found for kedro >=0.15.1,<0.16.0
   1: fact: kedro (0.15.1) requires Python >=3.5, <3.8
   1: derived: not kedro (0.15.1)
   1: fact: no versions of kedro match >0.15.1,<0.16.0
   1: conflict: no versions of kedro match >0.15.1,<0.16.0
   1: ! kedro (>0.15.1,<0.16.0) is partially satisfied by not kedro (0.15.1)
   1: ! which is caused by "kedro (0.15.1) requires Python >=3.5, <3.8"
   1: ! thus: kedro is forbidden
   1: ! kedro (>=0.15.1,<0.16.0) is satisfied by kedro (^0.15.1)
   1: ! which is caused by "projectX depends on kedro (^0.15.1)"
   1: ! thus: version solving failed
   1: Version solving took 0.037 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
The current project must support the following Python versions: ^3.7
Because no versions of kedro match >0.15.1,<0.16.0
 and kedro (0.15.1) requires Python >=3.5, <3.8, kedro is forbidden.
So, because projectX depends on kedro (^0.15.1), version solving failed.

Exception trace:
 /Users/user/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/user/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/user/.poetry/lib/poetry/_vendor/py3.7/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/user/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/user/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/user/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/user/.poetry/lib/poetry/console/commands/add.py in handle() at line 139
   status = installer.run()
 /Users/user/.poetry/lib/poetry/installation/installer.py in run() at line 73
   self._do_install(local_repo)
 /Users/user/.poetry/lib/poetry/installation/installer.py in _do_install() at line 165
   ops = solver.solve(use_latest=self._whitelist)
 /Users/user/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/user/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 180
   raise SolverProblemError(e)

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

As you can see from the error and from the kedro pypi release page, the target python version is Python >=3.5, <3.8.
I specify python ^3.7 in my pyprojet.toml and as you can see when I run poetry debug:info my python version is * Python: 3.7.0 .

Other info, if I change the pyproject.toml:

[tool.poetry.dependencies]
python = "^3.7"

To:

[tool.poetry.dependencies]
python = "3.7"

It's working

[skorokithakis]

I had the same problem trying to install coverage 4.5.4 with python = ">=3.7". Changing it to "^3.7" resolved.

[jasperges]

I have the same when I want to add PySide2.

$ poetry add pyside2

[SolverProblemError]
The current project's Python requirement (^3.7) is not compatible with some of the required packages Python requirement:
  - pyside2 requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <3.9

Because no versions of pyside2 match >5.14.0,<6.0.0
 and pyside2 (5.14.0) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <3.9, pyside2 is forbidden.
So, because tester depends on pyside2 (^5.14.0), version solving failed.

When I use the 'fix' from @MarcBoissonneault (changing to python = "3.7") it works.

[nils-werner]

I have the same with pandas = "^0.25.1" and python = ">=3.5, <3.8"

...

[tool.poetry.dependencies]
python = ">=3.5, <3.8"
numpy = "^1.15.4"
scipy = "^1.2.0"
pandas = "^0.25.1"

...

[SolverProblemError]
The current project's Python requirement (>=3.5, <3.8) is not compatible with some of the required packages Python requirement:
  - pandas requires Python >=3.5.3
  - pandas requires Python >=3.5.3
  - pandas requires Python >=3.5.3

Because no versions of pandas match >0.25.1,<0.25.2 || >0.25.2,<0.25.3 || >0.25.3,<0.26.0
 and pandas (0.25.1) requires Python >=3.5.3, pandas is forbidden.
And because pandas (0.25.2) requires Python >=3.5.3, pandas is forbidden.
So, because pandas (0.25.3) requires Python >=3.5.3
 and thesis depends on pandas (^0.25.1), version solving failed.

[a-berg]

I am having this same problem with dataclasses in Python 3.6:

[SolverProblemError]
The current project's Python requirement (^3.6.0) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

[crflynn]

I'm pretty sure the bug is here:

https://github.com/python-poetry/poetry/blob/7d5934e5c8a7463a930bf46ba3882434953183ca/poetry/puzzle/provider.py#L500-L507

I'm testing using this test, but not quite sure what to change in order to fix it.

https://github.com/python-poetry/poetry/blob/7d5934e5c8a7463a930bf46ba3882434953183ca/tests/mixology/version_solver/test_python_constraint.py#L5-L19

FWIW I'm testing with root.python_versions = "^3.7" and add_to_repo(repo, "foo", "1.0.0", python="<3.9"), which is similar to the situation I'm experiencing when trying to install the fluent-logger package, which gives this result on poetry 1.0.3:

[SolverProblemError]
The current project’s Python requirement (^3.7) is not compatible with some of the required packages Python requirement:
  - fluent-logger requires Python >=2.7,!=3.0,!=3.1,!=3.2,!=3.3,<3.9

[teknico]

Same problem when trying to add the dataclasses backport package for Python 3.6:

[SolverProblemError]
The current project's Python requirement (^3.6) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because decred depends on dataclasses (^0.7), version solving failed.

Poetry version: 1.0.3

[crflynn]

I looked at this a bit deeper and I think that the Provider.incompatibilities_for() method is evaluating not difference.is_empty() to True, which returns an Incompatibility between ^3.7 and <3.9. The difference between these two constraints evaluating to >=3.9. This evaluation makes sense because in the test test_solver_triggers_conflict_for_dependency_python_not_fully_compatible_with_package_python
, there is a check for non-empty differences, but this is testing that a package intending to be installed is compatible with the lower bound of python versions for the current project.

It seems that it’s evaluating properly, but it also seems that the expected behavior from the end-user differs from how the solver deals with packages that have upper bounds which conflict with project python constraints. The workaround discovered here is to pin the python version, but it would also work to set an upper bound on the python version which matches that of the package.

I’m not sure what the best solution here is, but if this special case happens, it might be helpful to log a suggestion to set a strict upper bound on the current project to match that of the package if you want to keep the solver behavior the way it is. Otherwise, I think the solution is to code this particular edge case into the solver. However, this somewhat invalidates the project python restrictions as the actual restriction then becomes the intersection of the project and its dependencies, but is no longer manifested explicitly in the pyproject.toml file.

[MrGreenTea]

I just stumbled upon this myself when using a dependency that depends on the backport of dataclasses.

Installing version 0.6 of dataclasses works fine, but 0.7 has python_requires=">=3.6, <3.7" in setup.py and that seems to trip up poetry's resolver.

[tribals]

Just experienced the same strange behavior.

Seems like when you use tilde requirement for Python: ~3.7 this will work as expected.

IMO, the caret requirements ^4.2 is very hard to understand and use properly. I propose to discard them at all.

[and-semakin]

Also receive similar error with taskipy package:

$ poetry update
Updating dependencies
Resolving dependencies... (0.9s)

[SolverProblemError]
The current project's Python requirement (>=3.7) is not compatible with some of the required packages Python requirement:
  - taskipy requires Python >=3.6,<4.0
  - taskipy requires Python >=3.6,<4.0

Because no versions of taskipy match >1.2.0,<1.2.1 || >1.2.1,<2.0.0
 and taskipy (1.2.0) requires Python >=3.6,<4.0, taskipy is forbidden.
So, because taskipy (1.2.1) requires Python >=3.6,<4.0
 and incutils depends on taskipy (^1.2.0), version solving failed.

[finswimmer]

Hello,

the "issue" here is one of the most misunderstood of poetry's version resolving. poetry checks if the versions that are given in the pyproject.toml are always valid and not just with the currently used python version.

So, if one define that the project is compatible with python ^3.6 this means it's compatible with all python versions >=3.6 but <4.0.

Let's have a look at one of the examples above:

dataclasses requires Python >=3.6, <3.7

dataclass is not available for python >3.7 here.

Or this one:

[SolverProblemError]
The current project's Python requirement (>=3.7) is not compatible with some of the required packages Python requirement:
  - taskipy requires Python >=3.6,<4.0
  - taskipy requires Python >=3.6,<4.0

The python requirement for the project has no upper boundary but taskipy has.

A further requirement is, that there is one major version of a package that is compatible with all python version. If this is not the case, the cli is not able to resolve this and we have to help by defining the version in the pyproject.toml. See here how this looks like.

fin swimmer

[skorokithakis]

@finswimmer I will argue that, since this many people got confused, this is actually a bug that needs to be resolved somehow. We'd at least appreciate the proper way to do this, as currently no solution is posted. Maybe a solution could be included in the docs?

[sdispater]

@skorokithakis This won't be treated as a bug since it's not one but the intended behavior.

That being said I agree that, since it's confusing for some people, we should give guidance to resolve the issue. That's something that is in progress and will be available in the next feature release (1.1). Here is what it will look like:

[skorokithakis]

That looks great, offering a solution in the output is a great idea. I would add something to make clear that all versions need to be satisfied, otherwise you will get text like:

The current project must support the following Python versions: ^3.7
Because no versions of kedro match >0.15.1,<0.16.0 and kedro (0.15.1) requires Python >=3.5, <3.8, kedro is forbidden.

Which is confusing to a user, as above. The problem is in "must support the following Python versions", which users interpret as "any of", but is actually "all of". Maybe changing it to "The current project must support all of the following Python versions" would be acceptable.

[matthijskooijman]

I also ran into this problem just now. I hear that this is intentional, but I'm not sure why.

the "issue" here is one of the most misunderstood of poetry's version resolving. poetry checks if the versions that are given in the pyproject.toml are always valid and not just with the currently used python version.

I think this is the essence. Why do all python versions given in the pyproject.toml need to valid with all dependencies?

Or, to make it more specific, look at this example:

The current project's Python requirement (>=3.7) is not compatible with some of the required > packages Python requirement:

• taskipy requires Python >=3.6,<4.0

I can see that installing taskipy would limit the python versions that the project works with to < 4.0. And apparently the solution here, would be to modify the project's python requirement to be >=3.7,<4.0, making it a subset of the taskipy allowed python versions (did I get that right?).

But why does one need to do this manually? Isn't this something that poetry could sort out? If every library, and the project, declares what Python versions it works with, then poetry can figure out what the resulting supported Python range is? If this results in an empty set of Python versions, that's of course problematic, but that does not seem to be the case at hand.

I can see that there is some documentation value in a project's pyproject.toml listing the effective usable Python versions for the project, but there is also value in documenting the python requirement of the project itself, without any dependency-imposed requirements. Especially since the latter can easily change when updating dependencies, that means that you would end up artificially limiting the python versions for the project, and then having to reconsider (manually check) whether the requirements can be relaxed again whenever dependencies are updated.

If this is is indeed the rationale for this requirement, I guess that the proper solution could be to have two python version requirements for the project: One that shows the requirements of the project itself, and one that shows the requirements of the project and all its requirements combined (where the former is manually configured and the latter is automatically determined by poetry whenever the lock file changes).

Another thing I wondered: Is this same policy applied to normal dependencies, or just to the python version? In other words, if a project depends on package A ^1.0 and also on a package B that depends on A >= 1.0, will this also produce the same error (I suspect not, since then the micromanagement of dependency versions might have been way too complicated). I guess the documentation value of listing the effectively viable versions of a package might be lower than for the python version, so if there is indeed an inconsistency here, that could explain this.

[finswimmer]

Hello @matthijskooijman,

I think this is the essence. Why do all python versions given in the pyproject.toml need to valid with all dependencies?

at first place poetry is a tool for development. When you start with a new python project, the first question you have to answer is: What python versions will I support. The reasons for the answer can be different: You need to use a special syntax that is only available in on version, you know your target platform, you already know that you will need a specific package that is only available for python x.y, ....

Once you've answer this question and write it down to your pyproject.toml poetry will make sure that with the dependencies you try to add, this goal can be reached.

But why does one need to do this manually? Isn't this something that poetry could sort out? If every library, and the project, declares what Python versions it works with, then poetry can figure out what the resulting supported Python range is?

Might be possible, but that's not how should manage your python project. As I said, you first have to decide what python versions you want to support, because this is essential for the python syntax you will use. Having valid dependencies is only the second step. Not the other way round.

In other words, if a project depends on package A ^1.0 and also on a package B that depends on A >= 1.0, will this also produce the same error.

No, this will not produce an error. Because the version constraints for the package tell poetry from which versions it can choose. Which one it will choose depends on the other packages in the project and there dependencies.

fin swimmer

[skorokithakis]

That makes sense to me, it should probably just be a bit better documented because many users (including me) assume the opposite.

[nils-werner]

I @finswimmer, thanks for the explanation.

You have given a few examples for why we should decide/pin the Python version first, and then install dependencies. However, can't you just use these as arguments against version deciding/pinning, too?

What python versions will I support.

That's exactly my thought when starting out: What versions do I/does this library support? Most likely it is python>=3.6. Why do I have to go look at each individual dependency and adjust my Python version accordingly?

You need to use a special syntax that is only available in on version

Syntax features are available in a wide range of Python versions. If a user wants to use a specific syntax, say f-strings. That doesn't mean they have to pick python==3.6, but instead python>=3.6. If now a dependency comes along and requires python>=3.7, that's fine, f-strings are still there.

you already know that you will need a specific package that is only available for python x.y,

I am not saying I should not care, but why exactly do I have to know this? Say in a scenario where I want to use Django 3, shouldn't Poetry figure out that django>=3.0 in fact needs python>=3.6?

(Sidenote: the fact that poetry wants me to switch Python versions using poetry env use, but is nevertheless still able to create the correct environment, does not make sense to me, either.)

I absolutely get that the Python version should be pinned during deployment, but that's what poetry.lock does/should do, no?

The mental model I have: Python is just another dependency, just like any package from PyPI. In my mental model I should not have to treat it specially. If I and all my dependencies define their compatible version range (and I don't have a conflict somewhere), Poetry should be able to find a valid Python version that suits all. It should be flexible in pyproject.toml and pinned in poetry.lock. And I shouldn't have to pick a Python version prior to running poetry install. But that's just my mental model, maybe I am wrong somewhere and am just failing to see it. :-)

[sdispater]

@nils-werner I know it might be confusing but there is actually a good and logical reason behind the way Poetry behaves in those cases.

As an example, I will take the taskipy example mentioned by @matthijskooijman:

• The project declares being compatible with Python >=3.7
• This project depends on taskipy which declares being compatible only with Python >=3.6,<4.0.
• So, it means that taskipy would not be satisfied for Python >=4.0 (the difference between >=3.7 and >=3.6,<4.0)

At this point, you have two choices, either change the python requirement of your project to >=3.7,<4.0 (however, note that this is not what I would personally recommend because it would mean changing the declared compatible Python versions to accommodate the dependencies) or specify that the taskipy dependency should only be installed for Python >=3.7,<4.0 by using the python property or the markers property:

taskipy = {version = "^1.2.1", python = ">=3.7,<4.0"}

It will now work properly.

This will be better explained when the 1.1 version of Poetry is released:

On another note

I absolutely get that the Python version should be pinned during deployment, but that's what poetry.lock does/should do, no?

No, the lock file is system and Python version agnostic.

[matthijskooijman]

At this point, you have two choices, either change the python requirement of your project to >=3.7,<4.0 (however, note that this is not what I would personally recommend because it would mean changing the declared compatible Python versions to accommodate the dependencies)

Exactly.

or specify that the taskipy dependency should only be installed for Python >=3.7,<4.0 by using the python property or the markers property:

That does not really seem right either: Aren't you now specifying that for Python >= 4.0, taskipy is not needed? If you would be using Python 4.0, poetry would happily install without taskipy, thinking it has produced a working set of dependencies, while in reality your application will break due to a missing dependency?

[vpoulailleau]

That does not really seem right either

@matthijskooijman Exactly. The only thing we know is that the two projects aren't totally compatible. So we need to either restrict the possible python versions for our project or restrict the dependency usage. I agree this isn't satisfying, but I'd say that with @sdispater's solution we wait for the dependency to be compatible with our vision of our project :muscle:. We don't bend our vision to match those of our dependencies (though we have to keep in mind what you were saying about missing dependencies :cry:).

[matthijskooijman]

@sdispater's solution we wait for the dependency to be compatible with our vision of our project muscle. We don't bend our vision to match those of our dependencies (though we have to keep in mind what you were saying about missing dependencies)

But we do bend our vision, because we specify that taskipy is not needed for python >= 4.0, even when that is not true.

I would greatly prefer to just state the dependencies as they are in reality, so poetry can correctly deduce that my package is not installable for python >= 4.0 and can provide feedback to users that do want to install my package with python 4.0.

It might still be useful if this information is also shared with the current package maintainer, but I do not think it should prevent adding the dependencie. i.e. I can imagine that running poetry add will say "Adding package taskipy limits the effective supported python versions (declared >= 3.7, effective >= 3.7 <= 4.0) and then continue adding it. Maybe this information could be displayed on request as well, or maybe stored in an automatically updated field in pyproject.toml or poetry.lock?