poetry picks pypy dependencies when executed in cpython environment

[gragib]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 1.1.4
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/gragib/07f146d5b0c978674376c2e1d63edd79

Issue

If a dependency has one or more subdependency specific to a Python implementation, _poetry_ ignores the implementation details.

E.g. _slash_ has the following requirements:

IPython==1.2.1; implementation_name=='pypy'
IPython; implementation_name!='pypy'

If your project depends on _slash_, in a _cpython_ 3.7 environment, _poetry_ uses

ipython==1.2.1

instead of

ipython==*

[ARCHIVED] Issue

Compiling this file

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["None <[email protected]>"]

[tool.poetry.dependencies]
python = "3.7"
slash = "*"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

installs these packages

arrow==0.17.0
atomicwrites==1.4.0
attrs==20.3.0
Brotli==1.0.9
certifi==2020.11.8
chardet==3.0.4
colorama==0.4.4
confetti==2.5.3
dessert==1.4.1
emport==1.3.1
gossip==2.4.0
idna==2.10
ipython==1.2.1
Jinja2==2.11.2
libtmux==0.8.5
Logbook==1.5.3
MarkupSafe==1.1.1
munch==2.5.0
ordered-set==4.0.2
py==1.9.0
pyparsing==2.4.7
pyreadline==2.1
python-dateutil==2.8.1
raven==6.10.0
requests==2.25.0
sentinels==1.0.0
six==1.15.0
slash==1.12.0
SQLAlchemy==1.3.20
urllib3==1.26.2
vintage==0.4.1

Simply installing slash using pip installs these packages:

arrow==0.17.0
atomicwrites==1.4.0
attrs==20.3.0
backcall==0.2.0
Brotli==1.0.9
certifi==2020.11.8
chardet==3.0.4
colorama==0.4.4
confetti==2.5.3
decorator==4.4.2
dessert==1.4.1
emport==1.3.1
gossip==2.4.0
idna==2.10
ipython==7.19.0
ipython-genutils==0.2.0
jedi==0.17.2
Jinja2==2.11.2
libtmux==0.8.5
Logbook==1.5.3
MarkupSafe==1.1.1
munch==2.5.0
ordered-set==4.0.2
parso==0.7.1
pickleshare==0.7.5
prompt-toolkit==3.0.8
py==1.9.0
Pygments==2.7.2
pyparsing==2.4.7
python-dateutil==2.8.1
raven==6.10.0
requests==2.25.0
sentinels==1.0.0
six==1.15.0
slash==1.12.0
SQLAlchemy==1.3.20
traitlets==5.0.5
urllib3==1.26.2
vintage==0.4.1
wcwidth==0.2.5

My understanding is that the two lists should be identical. They are not.

[sinoroc]

_pip_ and _poetry_ use different algorithms / implementations for dependency resolution. So differences in the results are to be expected. As far as I know, there is not necessarily one single solution for the same input, _pip_ and _poetry_ make different choices.

Is there something particularly problematic in the example you gave?
Is it important for you to have the same results in both cases? How so?

[gragib]

The version of ipython selected by poetry is ancient. We use features in the newer versions of ipython.

[gragib]

This piece from requirements.txt of slash seems the root cause of the issue:

IPython==1.2.1; implementation_name=='pypy'
IPython; implementation_name!='pypy'

Is there any way to tell poetry to ignore specific implementations, e.g. pypy in this example?

[sinoroc]

I see. I do not know of any workaround.

Probably better is to file a bug against _slash_, if you believe their packaging is wrong.

On the other hand, _pip_'s dependency resolution seems wrong in that case. Were you using an old version of _pip_? They have a new dependency resolver (or maybe it is not released yet), that should not allow these kind of errors.

[gragib]

I am using 2020-resolver in pip.

[gragib]

The pyproject.toml above is a tiny fragment of my actual file, but it highlights the issue I am fighting. Adding

slash = {version = "*", markers = "implementation_name != 'pypy'"}

alleviates the issue, but as soon as I add another dependency that also requires slash, I end up in the same hole.

[sinoroc]

@gragib Something I must have missed, are you using _pypy_ or not?

[finswimmer]

Hello @gragib,

We use features in the newer versions of ipython.

if the project depends on ipython, than you should define it as a dependency. Especially if you rely on features, available from a certain version onward.

fin swimmer

[gragib]

I did that and dependency resolution never succeeded because poetry insists on ipython==1.2.1 for slash.

[sinoroc]

@gragib Are you using pypy or not?

[gragib]

I am not using pypy. Just cpython 3.6+.

[sinoroc]

OK. Then it looks like a bug to me as well. Poetry should not be insisting on ipython==1.2.1 since you are not on _pypy_.

[gragib]

My temporary workaround is to look at the dependency tree to determine the deepest dependency on slash and to add a marker to that dependency to ignore pypy implementation. Not ideal, but it works for now.

I hope this issue gets resolved soon. It would be a good idea to replace the title of this issue with something more accurate.

[sinoroc]

It would be a good idea to replace the title of this issue with something more accurate.

Usually Github allows one to edit the title of their own tickets.

[gragib]

I have updated the issue title and description.