Poetry: Dataclasses solver failure with conditional python marker

[mands]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Linux (Fedora 32):

• Poetry version:

$ poetry debug info

Poetry
Version: 1.0.5
Python:  3.8.2

Virtualenv
Python:         3.8.2
Implementation: CPython
Path:           /home/projects/poetry-dataclasses/poetry-demo/.venv
Valid:          True

System
Platform: linux
OS:       posix
Python:   /usr

pyproject.yaml

[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "^0.7", python = "~3.6.1"}
dacite = "^1.5.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Issue

Following on from #1413, the solution there seems to suggest user python version markers to specify when to install a conditional dependencty - in my case the dataclasses-backport on python 3.6.*. However this seems to result in a SolverProblemError resolving in certain cases even with the python specifier as indicated in my pyproject.yaml above .

Things seem to work fine until I also install the addional library dacite (https://github.com/konradhalas/dacite) - which also has a dependency on dataclasses when using python 3.6. The setup.py for it is as follows,

    install_requires=['dataclasses;python_version<"3.7"'],

Once this library is also installed, adding any other packages to my project results in the aforementioned SolverProblemError, e.g.

$ poetry add arrow
Using version ^0.15.6 for arrow

Updating dependencies
Resolving dependencies... (0.0s)

[SolverProblemError]
The current project's Python requirement (^3.6.1) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because poetry-demo depends on dataclasses (^0.7), version solving failed.

Commenting then uncommenting out either my use of dataclasses or dacite in my pyproject.toml to add additional libraries works, but it seems that both my projects and dacite's use of dataclasses with a conditional python marker conflict in poetry.

[tarkatronic]

I just ran into what I believe is this same bug, and was able to reliably reproduce it from this example. First, a working dependency specification:

[tool.poetry.dev-dependencies]
black = {version = "19.10b0", allow-prereleases = true, python = "^3.6"}

All operations (lock, install, show, etc) work as expected with the above. When adding "markers", modifying it to the following:

[tool.poetry.dev-dependencies]
black = {version = "19.10b0", allow-prereleases = true, python = "^3.6", markers = "platform_python_implementation == 'CPython'"}

I get the following error:

$ poetry lock
Updating dependencies
Resolving dependencies... (0.1s)

[SolverProblemError]
The current project's Python requirement (^3.5) is not compatible with some of the required packages Python requirement:
  - black requires Python >=3.6

Because tartufo depends on black (19.10b0) which requires Python >=3.6, version solving failed.

[kakulukia]

I think im also effected by the same error:

[SolverProblemError]
The current project's Python requirement (^3.8) is not compatible with some of the required packages Python requirement:
- django-bulma requires Python >=3.6,<3.9

Because no versions of django-bulma match >0.7.1,<0.8.0
and django-bulma (0.7.1) requires Python >=3.6,<3.9, django-bulma is forbidden.
So, because bulmatest depends on django-bulma (^0.7.1), version solving failed.

My Python version is within the set limits, but not accepted by poetry as a matching version. How can this be solved?

[P403n1x87]

I think I have the same problem with Coverage.py

~~~ bash
Using python3 (3.6.9)
Updating dependencies
Resolving dependencies... (0.0s)

[SolverProblemError]
The current project's Python requirement (>=3.6) is not compatible with some of the required packages Python requirement:

• coverage requires Python >=2.7, !=3.0., !=3.1., !=3.2., !=3.3., !=3.4.*, <4

Because no versions of coverage match >5.1
and coverage (5.1) requires Python >=2.7, !=3.0., !=3.1., !=3.2., !=3.3., !=3.4.*, <4, coverage is forbidden.
So, because austin-python depends on coverage (>=5.1), version solving failed.
~~~

[pawamoy]

I reproduced withi this very simple config:

[tool.poetry]
name = "hey"
version = "0.1.0"
description = ""
authors = ["Timothée Mazzucotelli <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.6"
dataclasses = { version = "^0.7", python = "<3.7" }

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[SolverProblemError]
The current project's Python requirement (^3.6) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because no versions of dataclasses match >0.7,<0.8
 and dataclasses (0.7) requires Python >=3.6, <3.7, dataclasses is forbidden.
So, because hey depends on dataclasses (^0.7), version solving failed.

[kakulukia]

I solved my problem by fixing the projects python version removing the ^
Because the upper Python version of the pack to be installed is limited and by default poetry projects use ^currentPythonVersion, poetry is creating this weird situation. @sdispater are you aware of this "bug"?

[pawamoy]

I think the problem here is that the environment marker is not taken into account when resolving.
Sure the project accepts >=3.6, <4.0 while dataclasses accepts only >=3.6, <3.7, but I'm explicitly asking to install dataclasses only when python is 3.6, so this is compatible, and Poetry should detect that :slightly_smiling_face:

[kakulukia]

Poetry could print a hint or offer to fix the project requirements with one of the available options that would allow to proceed.

[jcmuddle]

I have a workaround by including a package that depends on dataclasses.

[tool.poetry]
name = "test_poetry"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = { version = "^0.7", python = "^3.6.1, <3.7" }
dataclasses_json = "^0.4.5"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[jcmuddle]

Now without the additional package, just drop the patch version for dataclases.

[tool.poetry]
name = "test_poetry"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = { version="^0.7", python="^3.6, <3.7" }

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[kakulukia]

@jcmuddle Try setting your python version to 3.6.1 instead of ^3.6.1 and report .. this also should work.

[jcmuddle]

The following pyproject.toml does not install dataclasses:

[tool.poetry]
name = "test_poetry"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "3.6.1"
dataclasses = { version="^0.7", python="^3.6.1, <3.7" }


[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Is this what you meant @kakulukia ?
Here is the output from poetry update -vvv

Using virtualenv: /home/jmuddle/.conda/envs/test
Updating dependencies
Resolving dependencies...
   1: fact: test-poetry is 0.1.0
   1: derived: test-poetry
   1: fact: test-poetry depends on dataclasses (^0.7)
   1: selecting test-poetry (0.1.0)
   1: derived: dataclasses (^0.7)
PyPI: 1 packages found for dataclasses >=0.7,<0.8
   1: selecting dataclasses (0.7)
   1: Version solving took 0.009 seconds.
   1: Tried 1 solutions.

[jcmuddle]

@sdispater Firstly, thank you for the amazing poetry package and the work you and others have put into it. I saw that you released version 1.0.9 which contained the pull request #2526. However, with the latest version, I'm still seeing the same issue of dataclasses not being installed.

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = { version="^0.7", python="^3.6.1, <3.7" }

The above is the one failing; perhaps I'm doing something wrong here.

[kakulukia]

just add dataclasses as you would normally poetry add dataclasses .. this just worked for me using python 3.6.1

[theodcr]

Here is a more accurate way:

[tool.poetry.dependencies]
python = "^3.6.1 <3.7"
dataclasses = "^0.7"

It works and tells poetry exactly what we want: use Python 3.6 only.

[mands]

As the original submitter, I can still reproduce this on the latest Poetry (1.0.9), when running poetry add <anypackage> within a Python 3.8 venv.

For more context, I'm building an application that runs on any Python from >= 3.6.1, and < 4.0.0 that requires the dataclasses library. As dataclasses is only required when developing using / when a user is running on Python >= 3.6.1 and < 3.7.0, I've set a conditional marker - however I develop myself within a Python 3.8 venv. I don't believe that this is a user error, as I imagine Poetry supports such a configuration (Poetry's own pyproject.toml has a similar dataclasses constraint) so I'm not sure what's not working here.

The only way I can get it to work right now is by commenting the dataclasses package in my pyproject.toml, adding the additional package I want via poetry add, and then uncommenting dataclasses again.

My pyproject.toml is,

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "0.7", python = "~3.6.1"}

The output from running poetry add <anypackage> -vvv is

   1: fact: dataclasses (0.7) requires Python >=3.6, <3.7
   1: conflict: dataclasses (0.7) requires Python >=3.6, <3.7
   1: ! dataclasses (0.7) is satisfied by dataclasses (0.7)
   1: ! which is caused by "datapane depends on dataclasses (0.7)"
   1: ! thus: version solving failed
   1: Version solving took 0.420 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
The current project's Python requirement (^3.6.1) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because datapane depends on dataclasses (0.7) which requires Python >=3.6, <3.7, version solving failed.

[boukeversteegh]

Same problem as @mands

• Define a project that needs python >=3.6 / 3.6.1 / ^3.6 / ^3.6.1
• poetry add dataclasses --python ">=3.6, <3.7"
• installs fine
• then add any other package poetry add --dev black
• fails with

[SolverProblemError]
The current project's Python requirement (^3.6) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because betterproto depends on dataclasses (^0.7), version solving failed.

I can confirm the workaround above works. (Remove dataclasses, add new package, re-add dataclasses)

[dazza-codes]

Similar issue:

[SolverProblemError]
Because {some-lib} ({some-ver}) depends on dataclasses (^0.6) which doesn't match any versions, {some-lib} is forbidden.

where {some-lib} uses a pyproject.toml with

[tool.poetry.dependencies]
python = ">=3.6.1, <3.9.0"  # see tox below too
dataclasses = {version = "^0.6", python = "^3.6.1, <3.7"}

So, what's up with dataclasses (^0.6) which doesn't match any versions?

Also, there are some PyPi packages that have started to declare minimal python versions of 3.6.1 and this seems to force the consumer to also use that minimal version, but then another issue arises like:

[SolverProblemError]
The current project's Python requirement (>=3.6.1, <3.9.0) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

An example of a 3rd-party lib causing conflict with the minimal python version when the project is trying to use python = ">=3.6, <3.9.0", which should be OK with dataclasses, is the following.

[tool.poetry.dependencies]
python = ">=3.6, <3.9.0"  # see tox below too
dataclasses = {version = "^0.6", python = "^3.6.1, <3.7"}

[tool.poetry.dev-dependencies]
pre-commit = "^2.4"

But then:

[SolverProblemError]
The current project's Python requirement (>=3.6, <3.9.0) is not compatible with some of the required packages Python requirement:
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1
  - pre-commit requires Python >=3.6.1

This is impossible. My {some-lib} is allowed to be used in python >= 3.6, < 4.0 and I expect to be able to use dataclasses anytime that python is a 3.6.x version (and since pre-commit is only a dev-dep, it will never be installed when {some-lib} is installed, so dataclasses trumps pre-commit, and if the actual python is 3.6.0 I don't care if pre-commit cannot be installed), so the following solver error is some kind of a bug:

[SolverProblemError]
The current project's Python requirement (>=3.6, <3.9.0) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

So long as the dataclasses is restricted to a subset of the project python versions (i.e. the project python versions subsume all the versions allowed by dataclasses), it should pass OK. Consider even the extreme case where all of the dataclasses versions are entirely _outside_ the range of versions allowed by the project (i.e. no version intersection), it could still allow the inclusion of the dataclasses dependency and issue a warning that it is never going to be installed anywhere that the project is allowed. Whenever there is any version intersection between the project python versions and the dependency python versions, that means the dependency could be installed and should be allowed to pass without warning or error.

Somehow, poetry is using too-tight coupling of project python versions with dependency versions. Something about the sets of allowed dependency intersections needs to be relaxed for the package-spec. The actual package installation rules are another matter entirely, because the range of options are restricted by the actual python version involved at the time of installation.

In the case of dataclasses, there is an implicit, not explicit, understanding that the package is _not_ required for any python >= 3.7.0, so when the project declares that it can be used in a range of pythons > 3.6 but it only needs dataclasses for 3.6, it's like an implicit declaration that dataclasses is a "required option" only for py 3.6. Somehow the version-specs and the "optional" or "extra" specs are not fully capturing this implicit intention. i.e.

dataclasses = {version = "*", python = "~3.6.0", optional = true}

It might be cool if we could declare optional = true, required = "~3.6.0" or something to suggest that this package is normally optional but when the installation is on py3.6, then it is required.

IMO, maybe python should be fully backporting dataclasses into all current py3.6 releases, rather than providing this backport as a package. Give py3.6 this little extra battery that should be included.

In case it might help, python, dataclasses and pre-commit seem to play OK together using:

[tool.poetry.dependencies]
python = "^3.6"
dataclasses = {version = "*", python = "~3.6.0"}

[tool.poetry.dev-dependencies]
pre-commit = "*"

This combination of specs was arrived at by a combination of poetry commands and manual edits. When dataclasses version is specified using ^0.6, it seems to install only 0.6 and not allow an update to 0.7. (For consumers of this lib with these specs, there might be a problem with the * version for dataclasses, TBD.)

[cgarciae]

Having a lot of trouble with this. If a dependency itself depends on a restricted version of dataclasses you also get this error, even if you don't depend on dataclasses yourself.

[dekked]

This happens when adding black ^20.8b1 to dev requirements.

Unfortunately, I have to install black separately because of this issue :(

[abn]

Reproducible in 1.0.10.

$ [email protected] add --dry-run arrow
Using version ^0.16.0 for arrow

Updating dependencies
Resolving dependencies... (0.0s)

[SolverProblemError]
The current project's Python requirement (^3.6.1) is not compatible with some of the required packages Python requirement:
  - dataclasses requires Python >=3.6, <3.7

Because dataclasses (0.7) requires Python >=3.6, <3.7
 and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
So, because poetry-demo depends on dataclasses (^0.7), version solving failed.

This should be fixed in 1.1.0b2.

$ [email protected] add --dry-run arrow
Using version ^0.16.0 for arrow

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 0 updates, 0 removals, 5 skipped

  • Removing dataclasses (0.7): Skipped for the following reason: Not currently installed
  • Installing six (1.15.0): Skipped for the following reason: Already installed
  • Installing python-dateutil (2.8.1): Skipped for the following reason: Already installed
  • Installing arrow (0.16.0): Skipped for the following reason: Already installed
  • Installing dacite (1.5.1): Skipped for the following reason: Already installed

[xylix]

Had this bug with poetry 1.1.0b4.

With this https://github.com/robocorp/rpaframework/blob/master/packages/main/pyproject.toml pyproject toml running any poetry add or poetry remove

Using version ^0.782 for mypy

Updating dependencies
Resolving dependencies... (0.1s)

  SolverProblemError

  The current project's Python requirement (>=3.6,<4.0) is not compatible with some of the required packages Python requirement:
    - dataclasses requires Python >=3.6, <3.7, so it will not be satisfied for Python >=3.7,<4.0

  Because dataclasses (0.7) requires Python >=3.6, <3.7
   and no versions of dataclasses match >0.7,<0.8, dataclasses is forbidden.
  So, because rpaframework depends on dataclasses (^0.7), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties

    For dataclasses, a possible solution would be to set the `python` property to ">=3.6,<3.7"

    https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies,
    https://python-poetry.org/docs/dependency-specification/#using-environment-markers

Worked around the problem by manually modifying pyproject.toml.

[abn]

@xylix see #2950