Catch2: Feature Request: Implementing custom messages for REQUIRE/CHECK macros.

[haozhangphd]

Description

This is a feature Boost.Test has (BOOST_\). It would be very nice if we can define custom error message if any specific test fails. A use-case is as follows:

int minutes = 100;
REQUIRE_MESSAGE( 0 <= minutes <= 59 , "Minutes " << minutes << " outside allowed range.");

Output:

Example.cpp:X: FAILED:
Minutes 100 outside allowed range.

[onqtam]

Here is how u can extend Catch to have these macros:

#define CHECK_MESSAGE(cond, msg) do { INFO(msg); CHECK(cond); } while((void)0, 0)
#define REQUIRE_MESSAGE(cond, msg) do { INFO(msg); REQUIRE(cond); } while((void)0, 0)

[haozhangphd]

Thank you for you reply. I know there are ways of rolling my own macros, but it is still better if this feature is built-in.

Also if I run my test command with -s option, the message shows up, even if the message only talks about how a specific test case fails.

[horenmar]

You might want to check out matchers. They are bit of an overkill for simple expressions like in the example, but those should be self-documenting anyway. For more complex tests they let you introduce any amount of logic needed for proper testing and stringification.

[haozhangphd]

Yes I sometimes use matchers to fulfill similar function. However like you said this is an overkill and not as convenient as having a built-in macro. Using a CHECK_MESSAGE and a REQUIRE_MESSAGE macro surely require a lot less lines of codes than implementing custom matcher classes.

Another drawback with the matcher classes is that the value under test must preceded the string output, I don't have the flexibility of putting the values in the middle of the string output, whereas this can be easily done with a CHECK_MESSAGE or REQUIRE_MESSAGE macro.

[philsquared]

The trouble with specific examples is they may not convey the general idea of why you want something.
So what I say next is on the basis that it may prompt you to give another example that forces us to generalise - in the true spirit of TDD :-)

For your example I would write it something more like:

TEST_CASE("minutes") {
    int minutes = 100;
    SECTION( "Minutes must be within range" ) {
        REQUIRE( minutes >= 0 );
        REQUIRE( minutes <= 59 );
    }
}

The output, on failure, is then:

minutes
  Minutes must be within range
-------------------------------------------------------------------------------

[..]

  REQUIRE( minutes <= 59 )
with expansion:
  100 <= 59

Which would seem to have all the information you need. Granted it's not organised exactly how you would like - but I think it's usually good enough. We don't want to spend our timing optimising for output formatting on the failing case (which we should, hopefully, rarely see) - so I've tried to design things so that you get rich enough feedback with minimum effort.

So, now, you can either disagree that this is enough in this case (and if you have thoughts there I'm still interested to hear them - whether or not we end up changing anything) - or you might concede on this example and think of another one that better teases out what you want.

Your move :-)

[haozhangphd]

Thank you for your reply. Yes I think my previous example was indeed too trivial and does not justify an extra macro. However custom messages are really helpful sometimes when we need to peek into the internal state of the program to see why a particular test fails. A use case for this new macro is as follows:

TEST_CASE("Widget") {
    for (int ctorParam1 = 0; ctorParam1 < 10; ++ctorParam1) {
        for (int ctorParam2 = 0; ctorParam2 < 10; ++ctorParam2) {
            Widget w(ctorParam1, ctorParam2);
            w.doSomething();
            REQUIRE_MESSAGE(w.valueX() > 0, "The X value of Widget is invalid for when ctorParam1 = "
                                            << ctorParam1 << " and ctorParam2 = " << ctorParam2
                                            << ". The internal state of Widget is " << w.someInternalParameter() << ".");
        }
    }
}

There are alternative ways of getting the internal state of the program when a test case fails, but I think using a custom message for the REQUIRE/CHECK macros is by far the most convenient way.

[philsquared]

Ok, but what's wrong with?:

TEST_CASE("Widget") {
    for (int ctorParam1 = 0; ctorParam1 < 10; ++ctorParam1) {
        for (int ctorParam2 = 0; ctorParam2 < 10; ++ctorParam2) {
            CAPTURE( ctorParam1 );
            CAPTURE( ctorParam2 );
            Widget w(ctorParam1, ctorParam2);
            w.doSomething();
            INFO( "The internal state of Widget is " << w.someInternalParameter() );
            REQUIRE(w.valueX() > 0);
        }
    }
}

[haozhangphd]

Yes there are ways of capturing the internal state of the program already even without this extra macro. I just think it is more convenient to combine the custom message and the assertion macros into a single macro. Also the users have more freedom if we can customize the fail message.

[philsquared]

You can format INFO to look exactly like your REQUIRE_MESSAGE output - it's just a bit more fiddly (yet still less typing than REQUIRE_MESSAGE - less characters in INFO() than _MESSAGE :-) )
The only real difference is that the message goes before the assertion.
But it's more flexible because you can have several of them. In my example I put the two captures earlier. I didn't have to, but because I could it means that if the constructor of Widget throws, or the call to doSomething(), then those values will be logged - even though we haven't got to the assertion yet. I used CAPTURE because that automatically logs the name the variable and it's value - saving some redundancy - but does mean you may have to type more - I could have just put it all in an INFO)

So, in summary, using INFO is:

a) less typing
b) more flexible
c) allows more state to be captured at more times

with the only "downside" being that it appears before the assertion instead of after it.

That sounds like a good set of trade-offs to me :-)

So is it just an aesthetic thing, or do you still think REQUIRE_MESSAGE is more convenient?

[haozhangphd]

Thank you for your detailed explanation. Now thinking about it more, the difference is just an aesthetic thing. The extra macro does not provide as much convenience as I previously thought.