Cirrus-ci-docs: Exception handling

[cjdb]

I was chatting with @fkorotkov on Twitter about handling exceptional situations.

To provide some context, I build programs using a C++ toolchain, and am using a package manager called Conan, and a build system called CMake. Conan is responsible for invoking all the CMake commands, including the testing framework (called CTest). When using CTest, it will maintain a series of logs in the directory Testing/Temporary, but only a pass/fail summary is output to screen.

CTest offers a flag for extracting failed tests' output, but I believe that exception handling is a powerful tool for a robust system, and that this would be a boon to Cirrus configurations. Ideally, what I'd like to be able to do is something akin to what's below. Note: this example might be asking for more than the proposed title.

script:
  configure:
    - mkdir build
    - cd build
    - conan install ..
    - conan build .. --configure
  build:
    try:
      - conan build .. --build
    on_failure:
      - echo "Compilation failed"
  test:
    try:
      - conan build .. --test
    on_failure:
      - cat Testing/Temporary/LastTest.log

[fkorotkov]

What do you think about having an ability to specify when a script should be executed:

foo_script:
  execute: on_success | on_failure | always

By default a script will be only executed on success of the previous one but you'll be able to add a script to collect some debug information:

debug_script:
  execute: on_failure
  commands:
    - cat Testing/Temporary/LastTest.log

BTW this is somewhat related to #48 where you'll be able to save folders/files when a task finishes/fails.

[hilbix]

I think this can already be handled by the scripts. Doing it yourself even looks more versatile and more easy to comprehend, at last for me. Example:

configure_script: mkdir build && cd build && conan install .. && conan build .. --configure
build_conan_script: conan build .. --build && touch CONAN_BUILD_OK || :
build_assets_script: make -C assets && touch ASSETS_BUILD_OK || :
build_fails_script: [ -e CONAN_BUILD_OK ] && [ -e ASSETS_BUILD_OK ] || { echo "Build failed"; false; }
test_conan_script: conan build .. --test || { cat Testing/Temporary/LastTest.log; false; }

But ..

I'd like to see failure tolerant scripts, such that above can become more easy to express:

configure_script: mkdir build && cd build && conan install .. && conan build .. --configure
build_conan_script_allow_failures: conan build .. --build && touch CONAN_BUILD_OK
build_assets_script_allow_failures: make -C assets && touch ASSETS_BUILD_OK
build_fails_script: [ -e CONAN_BUILD_OK ] && [ -e ASSETS_BUILD_OK ] || { echo "Build failed"; false; }
test_conan_script: conan build .. --test || { cat Testing/Temporary/LastTest.log; false; }

With a bit more Task-State-Environment-Magic this could even become simplified:

configure_script: mkdir build && cd build && conan install .. && conan build .. --configure
build_conan_script_allow_failures: conan build .. --build
build_assets_script_allow_failures: make -C assets
build_fails_script: [ 00 = "$CIRRUS_SCRIPT_build_conan$CIRRUS_SCRIPT_build_assets" ] || { echo "Build failed"; false; }
test_conan_script: conan build .. --test || { cat Testing/Temporary/LastTest.log; false; }

where CIRRUS_SCRIPT_<SCRIPT NAME> carries the result code N (as of exit N) of the script accordingly (which must be provided by the CIRRUS CI runtime, of course). For non-tolerant scripts this always is 0 of course (else the script which checks would not have been executed as the task were terminated. Note that you only can access ENV of scripts which ran before you in the same task, of course.)

Notes:

• script_allow_failures corresponds with the task setting allow_failures. Perhaps this can be aliased with script_tolerant or script_may_fail or similar which is more comprehensible ..

• If badges can be addressed on script level (#83), failure tolerant scripts allow to create badges for warnings like coding style violations etc. The badge becomes Red if the script fails, Green if it is OK and Grey if the script wasn't run. You won't want to break the task just because some comment isn't in sync with DOC style, right? Example:

test_task:
  prep_script: apt-get install vera++
  codestyle_script_allow_failures: make check-code
  build_script: make
  test_script: make test

The badge-URL would be something like:
https://api.cirrus-ci.com/github/<USER OR ORG>/<REPO>.svg?task=test&script=codestyle

(If the prep_script fails, the badge becomes Grey. I would not mind if any badge, which is for an unknown script, just is Gray.)

PS: This is only an idea/proposal how I would implement this in case I had the powers ;)

PPS: Some notes on repeated scripts:

main_task:
  test_script_allow_failures: make test1
  test_script_allow_failures: make test2
  test_script_allow_failures: make test3

This could either append the return codes to the environment CIRRUS_SCRIPT_test (giving something like 000) or just override them (giving only the last result). I'd vote for override, as this is more stable in case you move/edit/delete things. If you need to pass the code, then do it as:

main_task:
  test_script_allow_failures: make test1 && exit ${CIRRUS_SCRIPT_test:-0}
  test_script_allow_failures: make test2 && exit ${CIRRUS_SCRIPT_test:-0}
  test_script_allow_failures: make test3 && exit ${CIRRUS_SCRIPT_test:-0}
  final_script: if [ 0 = "$CIRRUS_SCRIPT_test" ] || { echo WTF; false; }

Which is far more easy than to handle concatenated codes. Think about adding a 4th test etc. Same for the badge. It just check against the last state, regardles which one did it. Example:

main_task:
  test_script_allow_failures: make test1 && exit ${CIRRUS_SCRIPT_test:-0}
  test_script_allow_failures: make test2 && exit ${CIRRUS_SCRIPT_test:-0}
  fail_script: false
  test_script_allow_failures: make test3 && exit ${CIRRUS_SCRIPT_test:-0}
  final_script: if [ 0 = "$CIRRUS_SCRIPT_test" ] || { echo WTF; false; }

The badge then returns the result from make test2, as make test3 never made it. Easy and convenient.

[fkorotkov]

As the work on this feature has just started, I put a little bit more thoughts into the syntax and will change it from:

exit_script: exit 1
debug_script:
  execute: on_failure
  commands:
    - cat "Handle failure!"

to

exit_script: exit 1
on_failure:
  debug_script: cat "Handle failure!"

I think this syntax is more concise. Please let me know what do you think.

[RDIL]

Well IMO we should be able to execute ONLY on failure.

[fkorotkov]

You might want to always upload tests results wether they are failing or not.

[cevich]

I think this is a super-useful feature. Can't wait to use it :smile:

[fkorotkov]

This feature is available now. Please check and comment on the documentation here: #194

[RDIL]

Now new docs will be deployed 🥇

[dotdoom]

This doesn't seem to work, at least not as advertised.
Here's a sample: https://github.com/dotdoom/onfailure/blob/08e57b06cfc356767d1967d5bf58bd7ecbad8f12/.cirrus.yml
Yet neither on_failure nor always get executed: https://cirrus-ci.com/build/5683167567020032

Am I doing it wrong?

[RDIL]

Hey @dotdoom!
You need to put script instructions inside the always and on failure blocks.

[dotdoom]

Ouch. Stupid me. Thanks!