Catch2: Feature Request: optionally only redirect to Catch::cout/cerr if test failed

[GPMueller]

In my opinion it would be a useful option to redirect both std::cerr and std::cout into a temporary buffer, which is only printed to console after a test failed.
I was not able to discern if this could be done by writing a new Reporter.

In a code I am developing I have a lot of Log messages to help with debugging etc., which I usually do not want to show up for tests which passed. On failing tests, however, the output would often be very useful.

[horenmar]

Have you tried using INFO messages? They are scoped, and only printed when a test fails (or when you ask for everything being printed using -s.

See the logging documentation for details.

[GPMueller]

Sorry, I should have been more precise: the logging messages I was referring to are part of the code I am testing with Catch and stream into std::cout.

Indeed, the behaviour I would like to have for std::cout and std::cerr corresponds to INFO messages.

[horenmar]

This makes sense, but I will need to think about good way to expose this to users. Right now, it is up to the reporters to specify whether they want to take care of the stdout/stderr messages by themselves, or to let them print straight to stdout, which means that they aren't prepared to handle suddenly receiving them, if user specified so on the command line.

However, you can make that happen for your right now, by creating your own reporter (or just modifying an existing one), which specifies that it wants std::cout and std::err redirected. To do so, reporter::getPreferences().shouldRedirectStdOut has to be true, the easiest way of doing so is setting it in constructor, after bases are constructed, the way XmlReporter does it:

        XmlReporter( ReporterConfig const& _config )
        :   StreamingReporterBase( _config ),
            m_xml(_config.stream()),
            m_sectionDepth( 0 )
        {
            m_reporterPrefs.shouldRedirectStdOut = true;
        }

and then handling the redirected output in reporter::testCaseEnded.

[henryiii]

I was also looking for this, and was about to post an issue. I'm experimenting with a transition to Catch from GTest, but almost all my test cases print a large amount of information (part of the API, not the test cases), and would prefer to have that only shown on an error. That's the way CTest works with "CTEST_OUTPUT_ON_FAILURE". I'm combining multiple executables (catch works nicely as a single executable, and my library has a long initialization time), and that makes the testing output all or nothing.

[philsquared]

We can probably add a command line option to force it to capture or not and check that in the reporter constructors.

[henryiii]

reporter::testCaseEnded

I don't see a way to output what I need here. If I have two sections in a test case, the stdout gets stuck on after the assertion prints and after both sections have run (using this method). I need something like a "print out stdout if an assertion fails, then empty buffer" that runs after sections or assertions.

Currently working around this by manually capturing stdout in my test fixture, then adding it INFO, which does exactly the right thing. If there was a way to add the output to the INFO message spot, that would work ideally, I think.

[henryiii]

Would this be any easier to implement now that CATCH_CONFIG_EXPERIMENTAL_REDIRECT is available? A not-so-hacky way to capture output and then display it only on a failure, in the right place, the way GoogleTest works, would be still great through the INFO workaround I mentioned does work for the moment. :)

[pwinston]

Our problem is test code calls into production code which logs to stdout. We can't change the production code to use Catch2 INFO macros because it's production code, it's not test code.

Dumping the stdout when a test fails would be nice, but having a flag to dump all stdout as it goes would be nice too. I can see 3 different behaviors all being useful to someone:

1. Never show stdout, it's just spam and we never want to see it.
2. Show stdout on failure. We don't care except when debugging a failure.
3. Show stdout fully for all tests, including successful ones.

I can see 3 being something you leave off 98% of the time. But as someone working on the tests you toggle it on now and then, to see if the test does what you think it does based on the logging. For example do we create a new FooManager for each test? Yes we do the log shows that.

[niosHD]

For reference what others are doing, the pytest test runner, for example, by default captures STDOUT and STDERR and prints them only when the test failed. However, the capturing can be disabled by specifying the -s flag on the command line. (see https://docs.pytest.org/en/latest/capture.html)

[philsquared]

FTR this was the way Catch originally worked (always capture stdout/ stderr). But there were some issues as a consequence of that and so the current model (redirect is by reporter) was adopted.
Unfortunately I don't recall, now, what those issues where - and in particular if the still apply or can be worked around another way. Will have to dig into some issues....