Testthat: Exit code once a test fails

[lc0]

Hey guys,

Thank you for the awesome package! Although, I have a couple questions:

So I have

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"

Where I run my tests, a couple of these fail:

docker@0131698d01ec:/# R -e "setwd('/folder/path'); devtools::test()"

R version 3.3.1 (2016-06-21) -- "Bug in Your Hair"

....

> setwd('/folder/path'); devtools::test()
Loading package.here
Loading required package: testthat
Testing package.here

....

[1] "Testing message foo"
.[1] "Testing message foo"
1[1] "Testing message foo"
.[1] "Testing message foo"
...2..

Failed -------------------------------------------------------------------------
1. Failure: Testing message bar (@test-compute-test.R#46)
another.compute(...) not equal to `result`.
Component “disabled”: Mean relative difference: 0.147403


2. Failure: Testing message bar2 (@test-compute-test.R#129) -------------
awesome.function(...) not equal to `result`.
Component “somekpi”: Mean relative difference: 0.001006787


DONE ===========================================================================
>
>

Unfortunately the exit code, still 0

docker@0131698d01ec:/# echo $?
0

Any suggestions, how to make testthat return the exit code when any of the test fail?

As a side question: does testthat by any chance supports XUnit export? I do use Continuous Integration and would be quite convenient to export results of tests, so it could be used for other purposes.

Thanks for the help!

[hadley]

@jimhester do you have any smart way to handle this?

@lc0 see #481 for a XUnit reporter

[hadley]

Oh, now I remember: our expectation is that you're running the tests in a such a way that R quits when you get an error:

R -e "stop('error')"
echo $?

[jimhester]

@lc0 More specifically testthat::test_check() calls stop() when there is a test error (at the end of testthat:::run_tests()`), so will set the error code appropriately.

If you are trying to run tests in a CI environment like Jenkins you should probably be usingR CMD check, which will run your testthat tests and perform a number of other useful package checks and will return a non-zero error code if there are failures.

[zbjornson]

In case it's useful to others searching... I wanted to see more verbose output in CI than R CMD check permits, so I used:

R CMD check "${PKG_TARBALL}" --as-cran --no-tests
Rscript -e 'res=devtools::test(reporter="summary");df=as.data.frame(res);if(sum(df$failed) > 0 || any(df$error)) {q(status=1)}'

As far as I can tell, that has the added benefit of running all tests, including any in testthat::skip_on_cran blocks.

Part of the second line is based on all_passed.

[patr1ckm]

Bumping, it's a little surprising that devtools::test() doesn't send an error signal on failure. The workarounds are ok though.

[ilarischeinin]

If you want to make sure there are no ERRORs, WARNINGs, or NOTEs, this command is also something that could be used for CI:

Rscript -e 'stopifnot(all(sapply(devtools::check(), length) == 0L))'

Or if you just want to catch ERRORs:

Rscript -e 'stopifnot(length(devtools::check()$errors) == 0L)'

[hadley]

devtools::test() is designed for interactive usage, so it's not that surprising. There are many other ways to run tests as you can see in this thread.

[SMesser]

There's also a "stop_on_failure" flag in testthat::test_dir(). Set it to TRUE, and once the tests are complete, you'll get an error iff something failed. See https://cran.r-project.org/web/packages/testthat/testthat.pdf