Poetry: Dependency resolution failure: pip works but poetry cannot install turicreate 6.4.1

[malcolmgreaves]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name:
• Poetry version: 1.1.0b2 & 1.1.0rc1
• Link of a Gist with the contents of your pyproject.toml file: gist contains pyproject toml + result of poetry add -vvv turicreate=6.4.1

Issue

poetry does not correctly resolve the dependencies for turicreate whereas pip does.

turicreate version 6.4.1 depends on coremltools version 4.0b3. This can be seen by downloading the 3.8 wheel and inspecting:

$ pkginfo -f requires_dist turicreate-6.4.1-cp38-cp38-manylinux1_x86_64.whl
requires_dist: ['decorator (>=4.0.9)', 'numpy', 'pandas (>=0.23.2)', 'pillow (>=5.2.0)', 'prettytable (==0.7.2)', 'resampy (==0.2.1)', 'requests (>=2.9.1)', 'scipy (>=1.1.0)', 'six (>=1.10.0)', 'coremltools (==4.0b3)', 'tensorflow (>=2.0.0)', 'numba (<0.51.0)']

poetry add coremltools=4.0b3 works as expected. However, the problem comes with poetry add turicreate=6.4.1:

Updating dependencies
Resolving dependencies... (0.1s)

  SolverProblemError

  Because turicreate (6.4.1) depends on coremltools (3.3)
   and  depends on coremltools (4.0b3), turicreate is forbidden.
  So, because  depends on turicreate (6.4.1), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

In contrast, pip install turicreate=6.4.1 works as expected.

$ pip install turicreate==6.4.1
Collecting turicreate==6.4.1
  Downloading turicreate-6.4.1-cp38-cp38-manylinux1_x86_64.whl (92.0 MB)
     |████████████████████████████████| 92.0 MB 29 kB/s
...
Collecting coremltools==4.0b3
  Downloading coremltools-4.0b3-py3-none-any.whl (4.6 MB)
     |████████████████████████████████| 4.6 MB 2.3 MB/s
...

Why does poetry not find the correct dependency when pip does 😭 ?

[finswimmer]

Hello @malcolmgreaves,

how does your pyproject.toml looks like. Especially the entry for python is interesting. It looks like, depending on the python version the requirements for coremltools is different.

fin swimmer

[malcolmgreaves]

👋 @finswimmer -- the pyproject.toml is in the Gist link above (https://gist.github.com/malcolmgreaves/b7ab6812eedce1adb1a2d03737dace3b).

Originally, I thought this might be a py3.7/8 issue -- but there's the right pkginfo for the turicreate==6.4.1 package.

Does the non semantic versioning 2.0 format of coremltools matter?

[abn]

@malcolmgreaves the issue is the metadata associated with the package:

$ curl -sL https://pypi.org/pypi/turicreate/json | jq '.info.requires_dist'
[
  "decorator (>=4.0.9)",
  "numpy",
  "pandas (>=0.23.2)",
  "pillow (>=5.2.0)",
  "prettytable (==0.7.2)",
  "resampy (==0.2.1)",
  "requests (>=2.9.1)",
  "scipy (>=1.1.0)",
  "six (>=1.10.0)",
  "llvmlite (==0.31.0)",
  "coremltools (==3.3)",
  "tensorflow (>=2.0.0)"
]

Note the requirement specified for coremltools.

Additionally, the second issue you face is a debug log bug. :(

[finswimmer]

@abn yeah I saw this too. Where does pypi take this metadata from? (and go to bed man, it's late already! :smile: )

[abn]

@finswimmer haha. Probably a smart plan.

As for the metadata, it is what is publish during dist upload by the publisher. Twine, poetry etc.

[malcolmgreaves]

Wow! Thank you so much for debugging this @abn 🙏 Does this mean that it is turicreate packaging & publishing error? (If so, I can file an issue on their GH page.)

[abn]

@malcolmgreaves i think filing an issue with the project is the right approach here. Tools like poetry rely on this metadata to work correctly.

[abn]

We should at some point write an FAQ about how to debug these issues.

[malcolmgreaves]

@abn I filed an issue https://github.com/apple/turicreate/issues/3342 and a PR https://github.com/apple/turicreate/pull/3343 to fix it on turicreate's end. One of the project members is engaging on that GH issue thread about doing python version requirements in the package's published information. I think I'm a little out of my element commenting about "The Right Way" (tm) to do it -- if you have some time, could you add your 2 cents there?

[abn]

@malcolmgreaves done.

[malcolmgreaves]

Thank you so much for your help! 🙏