Poetry: Poetry ignores some transitive dependencies for the project

[ffix]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian 8.11, ArchLinux
• Poetry version: 0.12.17, 1.0.0b1
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/ffix/5ccc5e3f84ff188f3831392a03fc4a39

Issue

It looks like Poetry ignores some transitive dependencies for the project. For example, it ignores bz2file as a dependency for the smart-open package:

$ pip check
smart-open 1.8.4 requires bz2file, which is not installed.

But dependency specified in setup.py.
Full debug log is here: https://gist.github.com/ffix/77d153f8483dde2ded4dde3e35f64bfb

Another example is scikit-image:

$ pip check
scikit-image 0.13.1 requires matplotlib, which is not installed.
scikit-image 0.13.1 requires pywavelets, which is not installed.

[ffix]

And it looks like installing from git isn't affected:

$ poetry add --dry-run smart-open                   
Using version ^1.8.4 for smart-open

Updating dependencies
Resolving dependencies... (0.2s)


Package operations: 15 installs, 0 updates, 0 removals

  - Installing six (1.12.0)
  - Installing docutils (0.15.2)
  - Installing jmespath (0.9.4)
  - Installing python-dateutil (2.8.0)
  - Installing urllib3 (1.25.6)
  - Installing botocore (1.12.240)
  - Installing futures (3.3.0)
  - Installing certifi (2019.9.11)
  - Installing chardet (3.0.4)
  - Installing idna (2.8)
  - Installing s3transfer (0.2.1)
  - Installing boto (2.49.0)
  - Installing boto3 (1.9.240)
  - Installing requests (2.22.0)
  - Installing smart-open (1.8.4)

$ poetry add --dry-run 'git+https://github.com/RaRe-Technologies/[email protected]' 

Updating dependencies
Resolving dependencies... (1.8s)


Package operations: 16 installs, 0 updates, 0 removals

  - Installing six (1.12.0)
  - Installing docutils (0.15.2)
  - Installing jmespath (0.9.4)
  - Installing python-dateutil (2.8.0)
  - Installing urllib3 (1.25.6)
  - Installing botocore (1.12.240)
  - Installing futures (3.3.0)
  - Installing certifi (2019.9.11)
  - Installing chardet (3.0.4)
  - Installing idna (2.8)
  - Installing s3transfer (0.2.1)
  - Installing boto (2.49.0)
  - Installing boto3 (1.9.240)
  - Installing bz2file (0.98)
  - Installing requests (2.22.0)
  - Installing smart-open (1.8.4 1.8.4)

[kbaikov]

Noticed the same issue with:
matplotlib 2.2.4 requires backports-functools-lru-cache, which is not installed.
Even though the dependency is found in the wheel file metadata:

$ wheel2json matplotlib-2.2.4-cp27-cp27m-win_amd64.whl
{
    "abi": [
        "cp27m"
    ],
    "arch": [
        "win_amd64"
    ],
    "buildver": null,
    "derived": {
        "dependencies": [
            "backports.functools-lru-cache",
            "cycler",
            "kiwisolver",
            "numpy",
            "pyparsing",
            "python-dateutil",
            "pytz",
            "six"
        ],...

Info

Poetry
Version: 1.0.0b2
Python:  2.7.17

Virtualenv
Python:         2.7.17
Implementation: CPython
Path:           C:\Users\z0042n4z\src\poetry-issue\venv2
Valid:          True

System
Platform: win32
OS:       nt
Python:   C:\Python27

Steps to reproduce:

• Create a python 27 virtualenv 'python27 -m virtualenv venv`
• activate the venv
• create pyproject.toml poetry init
• install explicit version of matlab compatible with python 2.7. poetry add matplotlib=="2.2.4"
• import matplotlib:

$ python -c "import matplotlib.pyplot as plt"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\z0042n4z\src\poetry-issue\venv2\lib\site-packages\matplotlib\__init__.py", line 130, in <module>
    from matplotlib.rcsetup import defaultParams, validate_backend, cycler
  File "C:\Users\z0042n4z\src\poetry-issue\venv2\lib\site-packages\matplotlib\rcsetup.py", line 29, in <module>
    from matplotlib.fontconfig_pattern import parse_fontconfig_pattern
  File "C:\Users\z0042n4z\src\poetry-issue\venv2\lib\site-packages\matplotlib\fontconfig_pattern.py", line 28, in <module>
    from backports.functools_lru_cache import lru_cache
ImportError: No module named backports.functools_lru_cache

pyproject.toml

[tool.poetry]
name = "poetry-issue"
version = "0.1.0"
description = ""
authors = ["Konstantin Baikov"]

[tool.poetry.dependencies]
python = "^2.7"
matplotlib = "2.2.4"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

PS. pip resolves dependencies correctly

[neggert]

+1

[hyacz]

Noticed the same issue with:
pyinstaller 3.5 requires macholib, which is not installed.

OS: macOS 10.15.1 (19B88)
Poetry version: 0.12.17

pyproject.toml

[tool.poetry]
name = "poetry-demo"
version = "0.1.0"
description = ""
authors = ["Haohao Zhang <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]
pyinstaller = "^3.5"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

> poetry install
...

> poetry show --tree
pyinstaller 3.5 PyInstaller bundles a Python application and all its dependencies into a single package.
├── altgraph *
└── setuptools *

> poetry run pip check         
pyinstaller 3.5 requires macholib, which is not installed.

> poetry run pyinstaller --help               
Traceback (most recent call last):
  File "/Users/hyacz/Library/Caches/pypoetry/virtualenvs/poetry-demo-py3.7/bin/pyinstaller", line 10, in <module>
    sys.exit(run())
  File "/Users/hyacz/Library/Caches/pypoetry/virtualenvs/poetry-demo-py3.7/lib/python3.7/site-packages/PyInstaller/__main__.py", line 79, in run
    import PyInstaller.building.build_main
  File "/Users/hyacz/Library/Caches/pypoetry/virtualenvs/poetry-demo-py3.7/lib/python3.7/site-packages/PyInstaller/building/build_main.py", line 34, in <module>
    from ..depend import bindepend
  File "/Users/hyacz/Library/Caches/pypoetry/virtualenvs/poetry-demo-py3.7/lib/python3.7/site-packages/PyInstaller/depend/bindepend.py", line 28, in <module>
    from . import dylib, utils
  File "/Users/hyacz/Library/Caches/pypoetry/virtualenvs/poetry-demo-py3.7/lib/python3.7/site-packages/PyInstaller/depend/dylib.py", line 197, in <module>
    from macholib import util
ModuleNotFoundError: No module named 'macholib'

[kbaikov]

@sdispater any comments on the issue? Seems like a major functionality breakage.

[absassi]

@hyacz I just reported the PyInstaller issue (pyinstaller/pyinstaller#4609), as the actual cause is conditional code to define requirements in their setup.py, instead of using PEP 508 markers.

I don't think it is possible for Poetry to evaluate all possible branches in a setup.py that try to do "clever" things.