Cibuildwheel: Support for `pyproject.toml` only?

[heitorlessa]

Hello,

Firstly, let me thank you for the awesome work you've put onto this project!

I'm trying to use it for the first time in a project managed by poetry, where I have only pyproject.toml and no setup.py, and it fails with the following message in Github Actions:

cibuildwheel: Could not find setup.py at root of package

I setup a dummy project to reproduce it here: https://github.com/heitorlessa/test-actions

Question: Is that expected? If so, are there plans to support it?

Thank you!

[joerick]

I had a feeling this day would come! Yes this should in theory be as simple as checking for either a pyproject.toml or a setup.py in the root of the project. The fix should be fairly simple, but to test, it would be worth having an integration test that builds a poetry project. PR welcome!

[Czaki]

I have both (part of configuration in pyproject.toml), so existence of setup.py should be checked first.

[YannickJadoul]

I have both (part of configuration in pyproject.toml), so existence of setup.py should be checked first.

The order of checking doesn't matter, right? This is just a check to see if there is a Python project, and fail early. The actual execution of setup.py is done by pip wheel.

[heitorlessa]

Sure thing - Happy to do a PR, including the integ test

I don’t know code base yet but I suspect a quick search should yield some
checks for setup.py - Any pointers would be appreciated.

On Sun, 31 May 2020 at 12:15, Yannick Jadoul [e-mail]
wrote:

I have both (part of configuration in pyproject.toml), so existence of
setup.py should be checked first.

The order of checking doesn't matter, right? This is just a check to see
if there is a Python project, and fail early. The actual execution of
setup.py is done by pip wheel.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/joerick/cibuildwheel/issues/359#issuecomment-636456848,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZPQBEY66VGZO67BG23BV3RUI35DANCNFSM4NPCHGRA
.

[YannickJadoul]

Sure thing - Happy to do a PR, including the integ test I don’t know code base yet but I suspect a quick search should yield some checks for setup.py - Any pointers would be appreciated.

The check will be here: https://github.com/joerick/cibuildwheel/blob/ecbdc2e3d37e726e4cd0d0fd5ca1c3d53ebcedaf/cibuildwheel/__main__.py#L164

See also #358, as a PR with a pyproject.toml test. Maybe we could combine these two tests?

[Czaki]

If I good remember if project maintained with poerty it should be build using poetry build, not `pip wheel. But maybe I'm wrong.

[YannickJadoul]

If I good remember if project maintained with poerty it should be build using poetry build, not `pip wheel. But maybe I'm wrong.

That's more problematic. We wouldn't want to provide and check for all different kinds of builders (and handle how they produce wheels, etc).
But doesn't this then do the job to redirect pip wheel to Poetry? https://github.com/heitorlessa/test-actions/blob/9c56d934cc82045274981463d5a7f1f1055f6e82/pyproject.toml#L26 I'm not really familiar with PEP 517/518 either, though.

[heitorlessa]

If I good remember if project maintained with poerty it should be build using poetry build, not `pip wheel. But maybe I'm wrong.

Fortunately it doesn't, as pip already supports it.

Tested in a Docker env where Poetry isn't installed:

git clone https://github.com/heitorlessa/test-actions && cd test-actions
docker run -it -v "$PWD":/root python:3.8-slim /bin/bash
cd /root
pip wheel .

[heitorlessa]

Update - Had a look at the code, running integ locally would take quite a while by setting CI="true" env. Waiting for tests to complete in PR before I create own integ test

How should I run tests locally? Unit tests are working, but tests under tests/ folder fail at the first one

================================================================ test session starts ================================================================
platform darwin -- Python 3.7.4, pytest-5.4.2, py-1.8.1, pluggy-0.13.1 -- /Users/lessa/DEV/cibuildwheel/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/lessa/DEV/cibuildwheel, inifile: setup.cfg
collected 27 items                                                                                                                                  

test/test_0_basic.py::test FAILED                                                                                                             [  3%]

===================================================================== FAILURES ======================================================================
_______________________________________________________________________ test ________________________________________________________________________

tmp_path = PosixPath('/private/var/folders/h4/3psg4cys091411bp4827n1zm0hfv5w/T/pytest-of-lessa/pytest-5/test0')

    def test(tmp_path):
        project_dir = tmp_path / 'project'
        basic_project.generate(project_dir)

        # build the wheels
>       actual_wheels = utils.cibuildwheel_run(project_dir)

test/test_0_basic.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test/utils.py:71: in cibuildwheel_run
    cwd=project_path,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

popenargs = (['/Users/lessa/DEV/cibuildwheel/.venv/bin/python3', '-m', 'cibuildwheel', '--output-dir', '/var/folders/h4/3psg4cys091411bp4827n1zm0hfv5w/T/tmpykl570hr', '.'],)
kwargs = {'cwd': PosixPath('/private/var/folders/h4/3psg4cys091411bp4827n1zm0hfv5w/T/pytest-of-lessa/pytest-5/test0/project'), ...ocket_Render': '/private/tmp/com.apple.launchd.bTJkaN5sm4/Render', 'COLORFGBG': '15;0', 'COLORTERM': 'truecolor', ...}}
retcode = 2
cmd = ['/Users/lessa/DEV/cibuildwheel/.venv/bin/python3', '-m', 'cibuildwheel', '--output-dir', '/var/folders/h4/3psg4cys091411bp4827n1zm0hfv5w/T/tmpykl570hr', '.']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.

        The arguments are the same as for the call function.  Example:

        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['/Users/lessa/DEV/cibuildwheel/.venv/bin/python3', '-m', 'cibuildwheel', '--output-dir', '/var/folders/h4/3psg4cys091411bp4827n1zm0hfv5w/T/tmpykl570hr', '.']' returned non-zero exit status 2.

../../.pyenv/versions/3.7.4/lib/python3.7/subprocess.py:347: CalledProcessError

By looking at #358, the convention seems to be having integ tests under test.

---

Repro steps:

1. Fork project
2. Clone locally
3. Created virtual env python3 -m venv
4. Installed dev deps pip install -r requirements-dev.txt
5. Installed cibuildwheel as editble pip install -e .
6. Run tests ./bin/run_tests.py

[YannickJadoul]

The integration tests are not really meant to be run locally, since they expect a set-up CI environment (and determine where they are running by looking at some environment variables set on the CI services). You also don't really want to run cibuildwheel locally; see README:

cibuildwheel is not intended to run on your development machine. Because it uses system Python from Python.org it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like Travis CI, CircleCI, Azure Pipelines and AppVeyor are ideal.

The only exception is probably Linux, where docker images are used anyway. So if you have docker setup and don't mind downloading a bunch of images, you can run bun/dev_run_test (which basically just sets the platform, rather than having cibuildwheel check whether it's running on a CI service and what platform).

EDIT: I tend to just create a PR and force push over stupid mistakes. If you run the unit tests, the most obvious errors will already be caught.

[heitorlessa]

The integration tests are not really meant to be run locally, since they expect a set-up CI environment (and determine where they are running by looking at some environment variables set on the CI services). You also don't really want to run cibuildwheel locally; see README:

cibuildwheel is not intended to run on your development machine. Because it uses system Python from Python.org it will try to install packages globally - not what you expect from a build tool! Instead, isolated CI services like Travis CI, CircleCI, Azure Pipelines and AppVeyor are ideal.

The only exception is probably Linux, where docker images are used anyway. So if you have docker setup and don't mind downloading a bunch of images, you can run bun/dev_run_test (which basically just sets the platform, rather than having cibuildwheel check whether it's running on a CI service and what platform).

EDIT: I tend to just create a PR and force push over stupid mistakes. If you run the unit tests, the most obvious errors will already be caught.

I learned that by running out of space while trying to run these on Docker (Docker on Docker) :D

Just updated the PR to ensure Py2.7 is supported - I guess this project will drop support at some point given its EOL

[heitorlessa]

Resolving in favour of PR being approved, and due to poetry not supporting platform wheels