Cibuildwheel: Test failure is not an error

[PerretB]

Except if I'm missing something, at the moment, the test feature is pretty useless as it seems that the result of the CIBW_TEST_COMMAND is not checked.

I tested building on Linux and Mac (on travis) with

CIBW_TEST_COMMAND="false"

In both cases the failure is silently ignored (https://travis-ci.org/PerretB/Higra/jobs/537406384 line 2503 and https://travis-ci.org/PerretB/Higra/jobs/537406384 line 1494). Note that on MacOS, the invalid exit value is recognized but nothing happens.

Expected behavior: cibuildwheel should stop with a non zero exit value if a test returns a non 0 value (and no wheels should be produced).

[YannickJadoul]

@PerretB As far as I can test, cibuildwheel does exit with value 1 when CIBW_TEST_COMMAND=false:

~/tmp-cibw$ cibuildwheel --platform linux

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 0.10.2
[...]
docker command: docker rm --force -v cibuildwheel-6de6d569-dbd0-4fce-9535-d47400ba9199
cibuildwheel-6de6d569-dbd0-4fce-9535-d47400ba9199
~/tmp-cibw$ echo $?
0
~/tmp-cibw$ CIBW_TEST_COMMAND=false cibuildwheel --platform linux

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 0.10.2
[...]
+ '[' '!' -z '' ']'
+ '[' '!' -z false ']'
+ pushd /root
+ PATH=/opt/python/cp27-cp27m/bin:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ sh -c false
docker command: docker rm --force -v cibuildwheel-e8ec35c4-228c-445f-852a-cbca90cc74af
cibuildwheel-e8ec35c4-228c-445f-852a-cbca90cc74af
~/tmp-cibw$ echo $?
1

I think the problem is that you are still executing pip install twine afterwards, in your .travis.yml and haven't run set -e. Somehow bash won't stop executing a script without set -e, and will only look at the exit code of the last command.

[PerretB]

Oh, what a stupid error on my side, I focused on cibuildwheel and forgot to carefully check my config files. Sorry for the annoyance.

[YannickJadoul]

No problem; it's pretty counter-intuitive, indeed :-)