Catch2: !shouldfail still prints assertion failures

[eloff]

Since the test is expected to fail, should it still print the error? In this case the failure is a success - it should be quiet I think.

/home/eloff/projects/teradb/bin/testteradb

blockmallocfail

/home/eloff/projects/teradb/test/blockmalloc.cc:128
...............................................................................

/home/eloff/projects/teradb/test/blockmalloc.cc:131: FAILED:
REQUIRE( 1 == 0 )

test cases: 5 | 4 passed | 1 failed as expected
assertions: 365 | 364 passed | 1 failed as expected

[philsquared]

Sorry, this is a bit late. In case you're still watching, @eloff, my response is that this is a bit of a philosophical question.
The key question is, why are you marking it as [!shouldfail], rather than testing the inverse condition, or checking that exceptions are thrown - or whatever other way would normally be used?
In my case where I've used this it's because I'm documenting that a requirement is known to not be currently fulfilled, or there is a known issue, but I want to keep track of it: and be notified if it accidentally starts working.
It's a rare case but sometimes useful.
In that case I do want to see the actual failure (to be reminded that it needs fixing) - I just don't want it to fail my run.

Perhaps you have a different use case. It would be useful to hear it (assuming it's still relevant to you?).

[eloff]

I went back and looked up that test file, I think I just assumed !shouldfail is a way of documenting/separating tests for failure conditions from success conditions. I don't see any reason why I couldn't just check inverse conditions or exceptions as you say.

Maybe the only thing to do here is update the docs for !shouldfail to make the intended use more clear and mention that a passing !shoudlfail test will still produce output.

[philsquared]

Point well taken. Thanks for taking the time and trouble to respond!

[philsquared]

I've updated with:

• [!shouldfail] - like [!mayfail] but fails the test if it passes. This can be useful if you want to be notified of accidental, or third-party, fixes.

So I'm going to close this one now.

[alanconway]

I realise this is closed but I have an alternate proposition.

I have tests that I wrote for a know bug that isn't fixed. I don't want to invent the tests because they're testing for the correct behaviour which someday we will implement. I would like to know if the problem gets fixed accidentally.

However I don't want a ream of scary-looking output from my test suite as long as it is failing as expected. The return code is correct, so it won't upset the CI bots, but developers who see all that output are liabable to waste time investigating (I did it myself even though I wrote the tests and marked them [!shouldfail]!!) The only solution I have so far is to skip them, which makes !shouldfail redundant.

To my thinking a [!shouldfail] test should automatically invent the output of it's CHECKs. I.e. CHECK(i == 42) would be silent if i != 42, and would say "CHECK(i == 42) passed unexpectedly!" or somesuch if it did pass. Thus it would be silent if it fails as expected and noisy in all the right places if it passes unexpectedly. Simply removing the [!shouldfail] would then convert it into a correct positive test.

By the way - excellent test framework. I'm in the process of replacing a homespun one - my tests are much simplified and more readable. This "inverted" test is the only feature I miss, and I can certainly work around it.

[MeAmAnUsername]

@alanconway I have the same use case as you (test prints lots of failures that I really don't want to see).

Inverting the condition wouldn't work if there are multiple CHECKs as only one of them needs to fail for the test to fail.

I don't know if it would be useful to print succeeding CHECKs if the test passes, as there may be a lot of CHECKs that are still supposed to pass. It's probably better to just say the test passed unexpectedly and letting the developer figure out why. Another approach would be to give developers more granularity so that they can specify exactly which CHECK should fail and printing when that CHECK doesn't fail.

[alanconway]

On Sat, May 11, 2019 at 12:05 AM MeAmAnUsername [e-mail]
wrote:

@alanconway https://github.com/alanconway I have the same use case as
you (test prints lots of failures that I really don't want to see).

Inverting the condition wouldn't work if there are multiple CHECKs as only
one of them needs to fail for the test to fail.

I don't know if it would be useful to print succeeding CHECKs if the test
passes, as there may be a lot of CHECKs that are still supposed to pass.
It's probably better to just say the test passed unexpectedly and letting
the developer figure out why. Another approach would be to give developers
more granularity so that they can specify exactly which CHECK should fail
and printing when that CHECK doesn't fail.

Good point, I think you're right. So [!shouldfail] would suppress all
failure messages unless the whole test passes, in which case it fails with
"passed unexpectedly". At that point you are alerted that, against all
probability and common sense, a piece of software has fixed itself. You can
remove [!shouldfail] and treat it like a regular test.