Poetry: Failure to detect dependencies of scikit-learn from private source

[gmacon]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [ ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS 10.13.6
• Poetry version: 0.12.16
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/gmacon/00003531106df72b0ae00415d32d4cf1

Issue

Poetry is failing to detect the dependencies of scikit-learn, but only when using a private source. If you modify the pyproject.toml by removing the tool.poetry.source entry, the dependencies (numpy, scipy, joblib) are correctly detected from PyPI.

This may be the same issue mentioned in https://github.com/sdispater/poetry/issues/607#issuecomment-437008459, but I'm not sure of that.

I had thought that this was caused by something strange that scikit-learn is doing, but I think that's less likely since it works correctly with PyPI.

Debug output

foo % python3.7 -m venv env
foo % . env/bin/activate
foo % poetry install -vvv                                                                                                                                                     Using virtualenv: /.../foo/env
Updating dependencies
Resolving dependencies...
   1: fact: foo is 0.1.0
   1: derived: foo
   1: fact: foo depends on scikit-learn (^0.21.2)
   1: selecting foo (0.1.0)
   1: derived: scikit-learn (^0.21.2)
nexus: 1 packages found for scikit-learn >=0.21.2,<0.22.0
   1: selecting scikit-learn (0.21.2)
   1: Version solving took 0.236 seconds.
   1: Tried 1 solutions.

Writing lock file


Package operations: 1 install, 0 updates, 0 removals

  - Installing scikit-learn (0.21.2)
foo % poetry debug:info

Poetry
======

 * Version: 0.12.16
 * Python:  3.7.3


Virtualenv
==========

 * Python:         3.7.3
 * Implementation: CPython
 * Path:           /.../foo/env
 * Valid:          True


System
======

 * Platform: darwin
 * OS:       posix
 * Python:   /usr/local/var/pyenv/versions/3.7.3/Python.framework/Versions/3.7

[dekim24]

I was having this same issue specifically with scikit-learn and was able to resolve it by updating poetry to the latest preview version (1.0.0b1)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

[earshinov]

I am using poetry 1.0.5, and it fails to install scikit-learn correctly.

I am using poetry inside docker, with python 3.8, and a private repo.

$ poetry add scikit-learn
Using version ^0.23.1 for scikit-learn

Updating dependencies
Resolving dependencies... (11.0s)An error occurred when reading setup.py or setup.cfg: 'Call' object has no attribute 's'
Resolving dependencies... (13.4s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  - Installing scikit-learn (0.23.1)

Scikit-learn's dependencies are missing in poetry.lock:

[[package]]
category = "main"
description = "A set of python modules for machine learning and data mining"
name = "scikit-learn"
optional = false
python-versions = "*"
version = "0.23.1"

[package.source]
reference = "our-private-repo"
url = "https://out-private-repo/simple"

An attempt to import sklearn in Python fails, because joblib, one of scikit-learn's dependencies, is not installed:

# python3.8
Python 3.8.3 (default, May 14 2020, 20:11:43)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/sklearn/__init__.py", line 80, in <module>
    from .base import clone
  File "/usr/local/lib/python3.8/dist-packages/sklearn/base.py", line 21, in <module>
    from .utils import _IS_32BIT
  File "/usr/local/lib/python3.8/dist-packages/sklearn/utils/__init__.py", line 23, in <module>
    from .class_weight import compute_class_weight, compute_sample_weight
  File "/usr/local/lib/python3.8/dist-packages/sklearn/utils/class_weight.py", line 7, in <module>
    from .validation import _deprecate_positional_args
  File "/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py", line 22, in <module>
    import joblib
ModuleNotFoundError: No module named 'joblib'

Please reopen this issue.

[dekim24]

@earshinov I re-encountered this issue as well on 1.0.5, and the workaround is to poetry add joblib and then the install should work. But I agree that this issue should be fixed if it's something on poetry's end.

[blaise17]

I'm also having this issue with a private source, poetry 1.0.10 and it not picking up any dependencies for scikit-learn in the lock file (scipy being the one causing me problems).

[dekim24]

I'm also having this issue with a private source, poetry 1.0.10 and it not picking up any dependencies for scikit-learn in the lock file (scipy being the one causing me problems).

I just tried to reproduce the issue in a new test project but couldn't - this is using python 3.6.10 and poetry 1.0.10, installing scikit-learn = "^0.23.2". which versions are you using / trying to install?

[blaise17]

I'm using python 3.7.5 and have tried both scikit-learn = "^0.23.1" and scikit-learn = "^0.23.2". I've also now tried it with poetry 1.0.9 (same effect). I've tried deleting my lock file and recreating it (same effect).

I do have an optional dependency that brings in scipy (which means scipy is present in the lock file, but listed as optional and only shown as a dependency of tensorflow), but scikit-learn lists no dependencies for its package. If I remove the optional dependency on tensorflow then scipy is not present in the lock file at all. Just for full disclosure. (And if I add in a scipy dependency explicitly then I get the same errors on the next missing scikit-learn dependency, which is joblist.)

[louwers]

We're also affected by this.

[louwers]

It seems that scikit-learn doesn't explicitly define it's dependencies: https://github.com/scikit-learn/scikit-learn/blob/master/setup.py

[gmacon]

Well, it does, just in a way that makes full use of the fact that setup.py is Python code. The install_requires parameter is set on line 249 based on computations performed in sklearn._build_utils.min_dependencies, where joblib is indeed declared as a dependency.

[louwers]

@gmacon My bad! It was an old version of scikit-learn that was causing problems, 0.20. But only from a private source. Very weird. Using a more recent version seems to work, so I'll check if that's an option.

[louwers]

Welp, same issue as the others now. joblib can't be found after installing scikit-learn from a private PyPI repo.

Poetry version 1.0.10
Python version 3.6.11