Cibuildwheel: Interaction with Travis stages

[MaxHalford]

Hey,

Sorry for the silly question, but I'm struggling with seeing how to build wheels for multiple platforms when using Travis stages. I'm aware of this example by @YannickJadoul, but I'm not well-versed enough in Travis to understand what's going on.

AFAIK, stages are the recommended way to use Travis. However, by looking at their documentation it seems that setting the matrix field is orthogonal to the stages field. The issue is that the "Travis CI" example in the README of cibuildwheel uses the matrix field.

On a sidenote, the .travis.yml file I'm working on is here. For the while it only builds wheels for manylinux i686 and manylinux x86_64 (even though MacOS is added to the matrix field). Ideally I would like to also build wheels for MacOS and Windows, if possible in a smooth way.

Hopefully I'm not the only one struggling with this!

[YannickJadoul]

@MaxHalford Stages are still in beta, though, or not anymore?

AFAIK, stages are the recommended way to use Travis.

You don't per se need them, as far as I understand. They're mainly meant as a level of grouping multiple single jobs within a full configuration. And the advantage is that if a job fails in one of the stages, later stages won't be built.

But yes, the integration with matrix is not always perfect. I recently had an issue where the global env was somehow not used in one of the stages...
Actually, in your case, the problem is that matrix: and jobs: include: are orthogonal. jobs: include: (or jobs: exclude:) tweak the jobs created by the combinatorial matrix expansion. (As you can see in the example you link to, I don't even use the matrix expansion, but just manually list all the different jobs that need to be run.)

I think the easiest you can do is to repeat this block three times:

  - stage: pypi
    sudo: required
    services:
      - docker
    env:
      - CIBW_BUILD="cp36-* cp37-*"
      - CIBW_BEFORE_BUILD="pip install cython"
    install:
      - pip install cibuildwheel==0.12.0 cython twine
    script:
      - cibuildwheel --output-dir dist
      - python setup.py sdist
      - ls dist
      - twine upload dist/* -u ${PYPI_USER} -p ${PYPI_PASSWORD}
    if: tag IS present

  - stage: pypi
    os: osx
    language: generic
    env:
      - CIBW_BUILD="cp36-* cp37-*"
      - CIBW_BEFORE_BUILD="pip install cython"
    install:
      - pip install cibuildwheel==0.12.0 cython twine
    script:
      - cibuildwheel --output-dir dist
      - python setup.py sdist
      - ls dist
      - twine upload dist/* -u ${PYPI_USER} -p ${PYPI_PASSWORD}
    if: tag IS present

  - stage: pypi
    os: windows
    language: shell
    env:
      - CIBW_BUILD="cp36-* cp37-*"
      - CIBW_BEFORE_BUILD="pip install cython"
    install:
      - pip install cibuildwheel==0.12.0 cython twine
    script:
      - cibuildwheel --output-dir dist
      - python setup.py sdist
      - ls dist
      - twine upload dist/* -u ${PYPI_USER} -p ${PYPI_PASSWORD}
    if: tag IS present

You can stop repeating yourself too much by doing something like this, though, using YAML's & and * to repeat values:

  - stage: pypi
    sudo: required
    services:
      - docker
    env: &pypi-env
      - CIBW_BUILD="cp36-* cp37-*"
      - CIBW_BEFORE_BUILD="pip install cython"
    install: &pypi-install
      - pip install cibuildwheel==0.12.0 cython twine
    script: &pypi-script
      - cibuildwheel --output-dir dist
      - python setup.py sdist
      - ls dist
      - twine upload dist/* -u ${PYPI_USER} -p ${PYPI_PASSWORD}
    if: tag IS present

  - stage: pypi
    os: osx
    language: generic
    env: *pypi-env
    install: *pypi-install
    script: *pypi-script
    if: tag IS present

  - stage: pypi
    os: windows
    language: shell
    env: *pypi-env
    install: *pypi-install
    script: *pypi-script
    if: tag IS present

[MaxHalford]

Ah okay I see. Good points. Also I didn't know about the & and * symbols in Travis, that's nice. Won't PyPI complain that the tag already exists when calling twine upload multiple times?

[YannickJadoul]

I think PyPI will only complain when you're trying to upload the same wheel twice.
And anyway, if the matrix/stages combination would have worked, you would have still had 3 separate jobs uploading to PyPI.

[YannickJadoul]

(Personally, I do upload to PyPI manually, after downloading the wheels from the GitHub release, though; I like that little bit of extra control)

[MaxHalford]

Got it. Thanks a lot for your precious help!

[MaxHalford]

@YannickJadoul I've had success with adding MacOS. I've also added a stage for Windows, but something seems wrong. I've specified that I want to build wheels for 3.6 and 3.7, and cibuildwheel seems to correctly understand this by looking at the printed configuration, but the 3.7 wheel if not being build. Here is the Travis log and here is the Travis file.

[YannickJadoul]

@MaxHalford I'll have another look tomorrow or the day afterwards, but something weird is going on, it seems, as demonstrated by the fact you get an error The command "cibuildwheel --output-dir dist --platform windows" exited with 1.

Somehow, the installation of Python 3.7 seems to fail, since you get a line showing assert os.path.exists(os.path.join(config_python_path, 'python.exe'))

[MaxHalford]

Indeed. On a sidenote, I had to manually set --platform windows, without that cibuildwheel failed to autodetect the platform.

[YannickJadoul]

@MaxHalford Sorry for the delay! Any updates?

Looking again, now, I think the problem was that you were building with cibuildwheel version 0.12.0, which did not support Windows on Travis CI yet. The README did already claim it did, but that's because the README on the master branch was still updated after releasing v0.12.0.
Anyhow, since 1.0.0 was released today, I would hope that solves your problems? :)

[MaxHalford]

Hey @YannickJadoul. Indeed the version number was the issue! Everything is now working. I have to say, cibuildwheel is truly an amazing tool. Keep it up!