Poetry: Trying to do a build with`--no-binary:all:` fails

[gpiks]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [ ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS
• Poetry version: N/A
• Link of a Gist with the contents of your pyproject.toml file:

Issue

Attempting to install poetry with the --no-binary flag like so:
pip install --no-binary=:all: poetry fails with the following error:

(The verbose output of the command is too noisy so has been elided. However, this should be reproducible with any system.)

pip install --no-binary=:all: poetry
Looking in indexes: https://artprod.dev.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-dpkg/simple
Collecting poetry
  Downloading https://artprod.dev.bloomberg.com/artifactory/api/pypi/bloomberg-pypi-dpkg/packages/packages/eb/98/a598bb2663e0e43557a99a243ebd21b27b1fc07d2c146b2554aa45ace5dc/poetry-1.0.3.tar.gz (164 kB)
     |████████████████████████████████| 164 kB 722 kB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/gpikale1/bbg/PyInf/bmq/venv/bin/python3.7 /Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/tmph9dkx9b0
       cwd: /private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry
  Complete output (36 lines):
  Traceback (most recent call last):
    File "/Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-build-env-wyn0exvh/overlay/lib/python3.7/site-packages/intreehooks.py", line 53, in get_requires_for_build_wheel
      return self._backend.get_requires_for_build_sdist(config_settings)
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-build-env-wyn0exvh/overlay/lib/python3.7/site-packages/intreehooks.py", line 38, in _backend
      obj = self._module_from_dir(modname)
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-build-env-wyn0exvh/overlay/lib/python3.7/site-packages/intreehooks.py", line 25, in _module_from_dir
      mod = importlib.import_module(modname)
    File "/Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/importlib/__init__.py", line 127, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
    File "<frozen importlib._bootstrap>", line 983, in _find_and_load
    File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 726, in exec_module
    File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry/poetry/masonry/__init__.py", line 10, in <module>
      from .builder import Builder
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry/poetry/masonry/builder.py", line 1, in <module>
      from .builders.complete import CompleteBuilder
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry/poetry/masonry/builders/__init__.py", line 1, in <module>
      from .complete import CompleteBuilder
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry/poetry/masonry/builders/complete.py", line 6, in <module>
      from poetry.factory import Factory
    File "/private/var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/pip-install-lwxe6w0y/poetry/poetry/factory.py", line 10, in <module>
      from clikit.api.io.io import IO
  ModuleNotFoundError: No module named 'clikit'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/gpikale1/bbg/PyInf/bmq/venv/bin/python3.7 /Users/gpikale1/bbg/PyInf/bmq/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/ys/sr9drq4d49301cq_np_1nks40000gp/T/tmph9dkx9b0 Check the logs for full command output.

However, installing clikit with the --no-binary=:all: flag results in an error installing poetry implying a cyclic build dependency.

[jreese]

Is there any plan to resolve this? We also build all of our packages from source, and this breaks our ability to build anything that depends on or uses Poetry as its builder.

[PaulNewtech]

Same problem for us

[ryansobol]

Just ran into this problem today. My use case is attempting to install a Python application—that uses Poetry as its build tool—via Homebrew. pypa/pip#8404

[preston-hf]

Anyone found a workaround?

[abn]

@preston-hf replacing the build-system section with the following will work. However, this requires every poetry project in your dependency graph to do the same, otherwise when pip attempts to build from source it tries to also build poetry from source which will fail because pastel in poetry's dependency graph is awaiting a fix (https://github.com/sdispater/pastel/pull/8) that is exactly this issue. :)

[build-system]
requires = ["poetry-core>=1.0.0a9"]
build-backend = "poetry.core.masonry.api"

[ryansobol]

@abn Thanks for sharing this workaround! I took a look at the README for poetry-core. So the plan is to integrate it into poetry v1.1.0?

[abn]

@ryansobol yes. One of the main changes in 1.1.z is the decoupling of the PEP 517 build backend (poetry-core) and the poetry the tool.

[abn]

This should no longer be an issue. The current master branch works without any errors when using the --no-binary option. It is recommended that packages using poetry as a backend switch to the lighter and purpose built poetry-core instead.

Ouput from installing master branch.

$ pip install --no-binary=:all: --pre git+https://github.com/python-poetry/poetry.git@master

<.. snipped ..>

Successfully built poetry poetry-core keyring shellingham clikit crashtest virtualenv tomlkit cachy cleo SecretStorage jeepney pastel ptyprocess cryptography
Installing collected packages: poetry-core, jeepney, pycparser, cffi, six, cryptography, SecretStorage, keyring, shellingham, pastel, pylev, crashtest, clikit, pyparsing, packaging, appdirs, distlib, filelock, virtualenv, tomlkit, chardet, idna, urllib3, certifi, requests, cachy, pkginfo, requests-toolbelt, ptyprocess, pexpect, webencodings, html5lib, cleo, msgpack, lockfile, cachecontrol, poetry
    Running setup.py install for pycparser ... done
    Running setup.py install for cffi ... done
    Running setup.py install for six ... done
    Running setup.py install for pylev ... done
    Running setup.py install for pyparsing ... done
    Running setup.py install for packaging ... done
    Running setup.py install for appdirs ... done
    Running setup.py install for distlib ... done
    Running setup.py install for filelock ... done
    Running setup.py install for chardet ... done
    Running setup.py install for idna ... done
    Running setup.py install for urllib3 ... done
    Running setup.py install for certifi ... done
    Running setup.py install for requests ... done
    Running setup.py install for pkginfo ... done
    Running setup.py install for requests-toolbelt ... done
    Running setup.py install for pexpect ... done
    Running setup.py install for webencodings ... done
    Running setup.py install for html5lib ... done
    Running setup.py install for msgpack ... done
    Running setup.py install for lockfile ... done
    Running setup.py install for cachecontrol ... done
Successfully installed SecretStorage-3.1.2 appdirs-1.4.4 cachecontrol-0.12.6 cachy-0.3.0 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 cleo-0.8.1 clikit-0.6.2 crashtest-0.3.1 cryptography-3.1 distlib-0.3.1 filelock-3.0.12 html5lib-1.1 idna-2.10 jeepney-0.4.3 keyring-21.4.0 lockfile-0.12.2 msgpack-1.0.0 packaging-20.4 pastel-0.2.1 pexpect-4.8.0 pkginfo-1.5.0.1 poetry-1.1.0b2 poetry-core-1.0.0a9 ptyprocess-0.6.0 pycparser-2.20 pylev-1.3.0 pyparsing-3.0.0a2 requests-2.24.0 requests-toolbelt-0.8.0 shellingham-1.3.2 six-1.15.0 tomlkit-0.7.0 urllib3-1.25.10 virtualenv-20.0.31 webencodings-0.5.1

[dtrodrigues]

@abn - is there a timeline for releasing the next version of poetry to PyPI? I work on the Homebrew project and was thinking about submitting Issues/PRs to projects to have them use poetry-core instead so that source builds work and they can be updated/included in Homebrew, but a newer release of poetry would solve this problem "for free".

Also, the newest version of of poetry-core (1.0.0a9) has "a" in its name - presumably for alpha - and is marked as a pre-release on GitHub: https://github.com/python-poetry/poetry-core/releases/tag/1.0.0a9. Is it stable enough to be used in production?

[abn]

is there a timeline for releasing the next version of poetry to PyPI?

We are trying for end of this month. But with all of us being busy elsewhere that might slip again :(

so that source builds work and they can be updated/included in Homebrew

I am curious as to why this is required since pip already takes care of building from source when required and platform dependent packages already have bot osx and manylinux binaries in a lot of cases, for the rest the PEP 517 builds should in theory take care of it.

but a newer release of poetry would solve this problem "for free".

Yes it does. However, remember that using poetry itself as a build backend is rather inefficient because of the dependencies it will pull in. Moving to poetry-core is still recommended. But yes, in a a sense things would just start working even if they use poetry, but there is no guarentee that somethign in the dependency graph would introduce another breakage later on.

Also, the newest version of of poetry-core (1.0.0a9) has "a" in its name - presumably for alpha - and is marked as a pre-release on GitHub: https://github.com/python-poetry/poetry-core/releases/tag/1.0.0a9. Is it stable enough to be used in production?

Considerting the code is basically copy paste from 1.0.x, I think it is safe enough for now. But we do expect a new stable release to come out before poetry's next release.

[dtrodrigues]

Homebrew attempts to build from source in almost all cases in cases (https://docs.brew.sh/Acceptable-Formulae#we-dont-like-binary-formulae), so we need to compile the python dependencies from source vs depending on the pre-compiled wheels, and do so via (approximately) pip --no-binary :all:, which fails with the currently released poetry version: https://github.com/Homebrew/brew/blob/f04b0142102a872d255440aef75984c1b213c584/Library/Homebrew/language/python.rb#L293-L295

I definitely agree that it would be better if projects transitioned over to poetry-core, and I'm happy to suggest they do it, but I also don't have a great sense for how many projects that are directly/indirectly relying on poetry and how much work it would be on the Homebrew end and go through and individually request that projects update.