Testthat: run all tests then stop on error?

[dchudz]

It seems like a function to run all the tests and then stop if there's an error would be pretty helpful. The reporter stop stops on error, and the default behavior runs all the test, but there's nothing I see that conveniently combines them.

It's only a few lines of code for a user:

results <- devtools::test(".")
results_df <- data.frame(results)
if (any(results_df$failed | results_df$error)) {
  stop(sprintf("Failed: %s, Error: %s", sum(results_df$failed), sum(results_df$error)))
}

But this seems common enough (for example, it's almost always what you want on a CI server) that a helper function or a reporter for it might be good.

[krlmlr]

You can combine the reporters: devtools::test(reporter = c("summary", "stop")).

[dchudz]

That doesn't seem to do what I described - I want information on the console about all of the tests, with an appropriate non-zero exit code if any of them failed.

[krlmlr]

There's also the FailReporter, which might better fit your needs: devtools::test(reporter = c("summary", "fail"))

[dchudz]

That looks to have the same issue? The tests stop running on the first failure.

To clarify my use case (which I expect to be pretty common), I'm running my unit tests on a CI server (Jenkins). Jenkins needs a non-zero exit code to properly fail for failing tests, but I want to see all of the failures, not just the first one.

I'm giving Jenkins as a specific example, but I expect this situation arises for other CI servers, etc.

My script above accomplishes the goal but is slightly brittle (e.g. would stop working if the name of the results columns changed in a new version of the package), but this seems common enough to warrant a built-in solution.

[krlmlr]

I think I understand your use case. On my system, reporter = c("summary", "fail") works as advertised: All tests are run, stop() is called in the end. Could you please submit a reprex to show what's not working?