Catch2: Catch Reports Exit Code 0 Even with Failed Tests

Created on 8 Mar 2018 · 7Comments · Source: catchorg/Catch2

Description

I am using Catch2 along with CMake to create a unit testing suite for my code. The testing framework itself works very well, but the problem is that catch returns an exit code of 0 even with failed tests (which makes no sense to me)

Steps to reproduce

Here is an example _without_ CMake that illustrates the problem.
I have a file main.cpp:

#define CATCH_CONFIG_MAIN
#include "Catch2.hpp"

int test()
{
    return 7;
}

TEST_CASE("should fail")
{
    REQUIRE(test() == 2);
}

Running this file gets me:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a.out is a Catch v2.2.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
should fail
-------------------------------------------------------------------------------
main.cpp:9
...............................................................................

main.cpp:11: FAILED:
  REQUIRE( test() == 2 )
with expansion:
  7 == 2

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

And then:

echo $?

Results in the output 0.

Why isn't the exit code non-zero? I am using CTest, so if there is a failing test, then CTest doesn't report it. This makes the framework particularly useless if failing tests are not reported because otherwise, tests will always pass. How should I fix this so that the correct exit code is shown?