Poetry: Could not build wheels for XXX which use PEP 517

[albireox]

• [X] I am on the latest Poetry version.
• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: macOS 10.15
• Poetry version: 1.0.0b4

Issue

11 provides a workaround for custom build systems (e.g., Cython extensions) that works well with poetry. However, when you try to do pip install . on a project that has a custom build.py the build process fails with

ERROR: Could not build wheels for seiya which use PEP 517 and cannot be installed directly

This means, among other things, that one cannot use ReadTheDocs to document the code. This happens with my code but I have also tested it with pendulum.

Is there a workaround for this?

[albireox]

Doing some more Googling on this I came up with the solution described here. One can create a setup.py file such as

from distutils.core import setup

import numpy

from Cython.Build import cythonize
from setuptools.extension import Extension


global setup_kwargs


extensions = [
    Extension('seiya.cube.cubify',
              ['seiya/cube/cubify.pyx'],
              include_dirs=[numpy.get_include()])
]


setup_kwargs = {}
setup_kwargs.update({'ext_modules': cythonize(extensions, annotate=True)})


setup(**setup_kwargs)

and remove the need of using build.py. As far as I can see this works fine (even allows editable installs!). When you do poetry install it creates an UNKNOWN.egg-info directory with PKG-INFO in which all the fields are UNKNOWN. That does not seem to affect poetry but I'm not sure what will happen when you build a distribution (I have run poetry build and the results look ok, with the correct metadata, but I haven't tested it all the way by uploading it to PyPI and installing from a fresh environment).

IMHO poetry really needs to define a way to deal with extensions. There are many projects that depend on a custom build system and this kind of workarounds are getting old ...

[albireox]

@sdispater Any comments on this? I'm seeing problems with this workaround with a recent project that requires compilation of extensions and being able to install using pip install . and I cannot find a good solution.

[flokno]

+1 from me and probably related to https://github.com/python-poetry/poetry/issues/2214

I use numpy to compile a small fortran module. Installation works fine with poetry install, but with pip install ., pip complains that it does not find a setup.py. From what I can see this should be easy to fix since poetry install builds a temporary setup.py as far as I understand.

[albireox]

Thanks @abn for fixing this. However, I cannot completely make it work. If I add

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core>=1.0.0a6"]

to my pyproject.toml it fails with

  Building wheel for sdss-flicamera (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/bin/python3.8 /Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/tmpzqz9ab7n
       cwd: /Users/albireo/Documents/Code/sdss5/flicamera
  Complete output (24 lines):
  Traceback (most recent call last):
    File "setup.py", line 2, in <module>
      from setuptools import setup
  ModuleNotFoundError: No module named 'setuptools'
  Traceback (most recent call last):
    File "/Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 204, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/pip-build-env-p_gl0npf/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 57, in build_wheel
      return unicode(WheelBuilder.make_in(poetry, Path(wheel_directory)))
    File "/private/var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/pip-build-env-p_gl0npf/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 56, in make_in
      wb.build()
    File "/private/var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/pip-build-env-p_gl0npf/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 82, in build
      self._build(zip_file)
    File "/private/var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/pip-build-env-p_gl0npf/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 101, in _build
      self._run_build_command(setup)
    File "/private/var/folders/gp/vsl4fq4d47s9mpywmcbrsj7c0000gn/T/pip-build-env-p_gl0npf/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 129, in _run_build_command
      subprocess.check_call(
    File "/Users/albireo/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", line 364, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/Users/albireo/.pyenv/versions/3.8.2/envs/flicamera/bin/python3.8', 'setup.py', 'build', '-b', 'build']' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for sdss-flicamera
Failed to build sdss-flicamera
ERROR: Could not build wheels for sdss-flicamera which use PEP 517 and cannot be installed directly

However, if I do

[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core>=1.0.0a6", "setuptools>=46.0.0"]

Then it works. Is this expected and if so, should the build-system documentation be updated?

I'm running this on a virtual environment using pyenv with Python 3.8.2. Note that the virtual environment has setuptools installed but that doesn't seem to make a difference.

[abn]

@albireox what's the version of pip in your environment? Can you try updating it?

[albireox]

With the latest pip 20.1 the problem still happens (that's what I had been using originally).

[albireox]

Maybe related to https://github.com/python-poetry/poetry/issues/2276#issuecomment-610796041

[virtustate]

Same problem working through https://www.pythoncheatsheet.org/blog/python-projects-with-poetry-and-vscode-part-1/ installing pendulum. Conda installs pendulum fine. I use Conda as a pip fallback for things like scipy on Windows.

[abn]

@albireox the issue is the isolated environment created by pip when building th wheel. #2666 might help mitigate this issue and so will #2826 since that will mean we rely on virtualenv everytime to build packages.

If he issue persists, please create a new issue, so we can track it and discuss it correctly. (#3001 maybe?).

[albireox]

@abn I tried this again, without the setuptools requires, and now it seems to work fine. If I see the issue pop up again I'll open a dedicated issue.

[ag-hcoder]

Any solution yet?

[finswimmer]

@ag-hcoder: I believe the problem here is setuptools itself. See https://github.com/python-poetry/poetry/issues/3153#issuecomment-727174607