Googletest: Is interleaving `EXPECT_CALL()`s and calls to the mock function *really* undefined behavior?

[ElectricRCAircraftGuy]

There seems to be a lot of confusion on this topic and the documentation really needs clarity. Please update the documentation to either validate this statement, or to correct it if it's wrong. There are several statements here, so I'll label the official Google one as "Wording 1 (official)" and the alternate possible wordings which others assert and which contradict this to be "Wording 2" and "Wording 3". Which of these 3 wordings is actually correct? I have personally seen tests which appear to run just fine which definitely go against the official "Wording 1", but I currently believe they are simply "undefined behavior which happens to work". What's the truth? Can you update the documentation please to provide examples and make this not a debated topic anymore?

Google documentation directly states:

Wording 1 (official)

"Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions" (https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests)

The part I especially want you to focus on is this part:

In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions

Yet, I have personally seen the following appear to work just fine. Here, testMethod() calls the mocked myMockObj.myMethod(). Notice that here we are definitely interweaving calls to EXPECT_CALL() and the mocked method, which is exactly what the Google documentation above says is _undefined behavior._ This works just fine though! Is it undefined behavior that happens to work fine? Or, is it really not undefined behavior?

TEST(FooTest, testCaseName)
{
    MyMock myMockObj;
    ...
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(true));
    testMethod();
    ASSERT_THAT(...);

    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(false));
    testMethod();
    ASSERT_THAT(...);
}

The code snippet above comes from @Marko Popovic's Stack Overflow answer here. In his answer, he asserts that doing the above code snippet is fine, and NOT undefined behavior, and thus implies that the Google documentation is wrong. His assertion seems to be the following (these are my own words, summarizing what he demonstrates):

Wording 2

Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. Furthermore, once you do an EXPECT_CALL() for a given mocked method, and then call that method, if you do another EXPECT_CALL() afterwards to then change the expectations set on future calls to that method, you must call the method again, or else the 2nd EXPECT_CALL() is now undefined behavior.

A colleague of mine has another theory, and I summarize his understanding this way:

Wording 3

Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. Therefore, interleaving EXPECT_CALL()s and calls to the mocked functions is just fine so long as you call at least one EXPECT_CALL() prior to any call to the mocked method being tested by this EXPECT_CALL(), and follow all other rules of EXPECT_CALL()s.

I have personally written a great deal about the rules of multiple EXPECT_CALL()s here, but have assumed the Google documentation to be correct (which maybe it isn't), when it says:

In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions (https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests)

So, is my documentation correct? In particular this part?

---

[QUOTE OF MY OWN WRITING START]

Question 3: Can I call EXPECT_CALL to set some expectations on a mock method, call the mock method, then call EXPECT_CALL on the method again to change the expectations, then call the mock method again?

This question wasn't even explicitly asked by the OP, but the only reason I found this page is because I was searching for this answer for many hours and couldn't find it. My Google search was "[gmock multiple expect_call][10]." Therefore, others asking this question will also fall on this page and need a conclusive answer.

A: NO, you can NOT do this! Although it may seem to work in testing, according to Google, it produces undefined behavior. See general rule #2 above!

"Important note: gMock requires expectations to be set before the mock functions are called, otherwise the behavior is undefined. In particular, you mustn't interleave EXPECT_CALL()s and calls to the mock functions" (https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests)

Therefore, this is NOT ALLOWED!

[QUOTE OF MY OWN WRITING END]

I've added my own answer to Stack Overflow here too: https://stackoverflow.com/questions/40089204/interleaving-expect-calls-and-calls-to-the-mock-functions/61467271#61467271. I pose questions in my "answer" which led me to open this issue.

[sbenzaquen]

On Mon, Apr 27, 2020 at 3:32 PM Gabriel Staples [e-mail]
wrote:

There seems to be a lot of confusion on this topic and the documentation
really needs clarity. Please update the documentation to either validate
this statement, or to correct it if it's wrong. There are several
statements here, so I'll label the official Google one as "Wording 1
(official)" and the alternate possible wordings which others assert and
which contradict this to be "Wording 2" and "Wording 3". Which of these 3
wordings is actually correct? I have personally seen tests which appear
to run just fine which definitely go against the official "Wording 1".

Google documentation directly states:
Wording 1 (official)

"Important note: gMock requires expectations to be set before the
mock functions are called, otherwise the behavior is undefined. In
particular, you mustn't interleave EXPECT_CALL()s and calls to the mock
functions" (
https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests
)

Yet, I have personally seen the following appear to work just fine.
Here, testMethod() calls the mocked myMockObj.myMethod(). Notice that
here we are definitely interweaving calls to EXPECT_CALL() and the
mocked method, which is exactly what the Google documentation above
says is undefined behavior. This works just fine though! Is it
undefined behavior that happens to work fine? Or, is it really not
undefined behavior?

It is undefined.
"Appears to work" is one of the possible things that undefined behavior can
do and should never be used as a proof that undefined behavior didn't
happen.

A very real and very common form of undefined behavior you can observe here
is race conditions in multithreaded code.
If you invoke EXPECT_CALL after the object is already in use you
open yourself to language level and application level race conditions, both
which can cause all sorts of bad things to happen.
Even in single threaded code you can end up with reentrancy bugs that
give you similar problems.

TEST(FooTest, testCaseName)
{
MyMock myMockObj;
...
EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(true));
testMethod();
ASSERT_THAT(...);

EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(false));
testMethod();
ASSERT_THAT(...);

}

The code snippet above comes from @Marko Popovic's Stack Overflow answer
here https://stackoverflow.com/a/40104286/4561887. In his answer, he
asserts that doing the above code snippet is fine, and NOT undefined
behavior, and thus implies that the Google documentation is wrong. His
assertion seems to be the following (these are my own words, summarizing
what he demonstrates):

Snippet from that page:
"This will be fine since there is no call to myMethod without set
expectations"
That has nothing to do with the problem. Is it not UB to call a method that
doesn't have expectations set.
The whole point is that the framework can diagnose problems like that when
using a StrictMock, for example, or ignore when using a NiceMock.

Wording 2

Important note: gMock requires expectations to be set before the mock
functions are called, otherwise the behavior is undefined. Furthermore,
once you do an EXPECT_CALL() for a given mocked method, and then call
that method, if you do another EXPECT_CALL() afterwards to then change
the expectations set on future calls to that method, you must call the
method again, or else the 2nd EXPECT_CALL() is now undefined behavior.

A colleague of mine has another theory, and I summarize his understanding
this way:
Wording 3

Important note: gMock requires expectations to be set before the mock
functions are called, otherwise the behavior is undefined. Therefore,
interleaving EXPECT_CALL()s and calls to the mocked functions is just
fine so long as you call at least one EXPECT_CALL() prior to any call to
the mocked method being tested by this EXPECT_CALL(), and follow all
other rules of EXPECT_CALL()s.

I have personally written a great deal about the rules of multiple
EXPECT_CALL()s here
https://stackoverflow.com/questions/44034633/google-mock-can-i-call-expect-call-multiple-times-on-same-mock-object/60905880#60905880,
but have assumed the Google documentation to be correct (which maybe it
isn't), when it says:

In particular, you mustn't interleave EXPECT_CALL()s and calls to the
mock functions (
https://github.com/google/googletest/blob/master/googlemock/docs/for_dummies.md#using-mocks-in-tests
)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/google/googletest/issues/2828, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADLRCPIXJFFSMTCTH4KW4DTROXMUZANCNFSM4MSGMDJQ
.

[asoffer]

As per Sam's comment, yes this is UB and something we do not intend to define.

[ElectricRCAircraftGuy]

Thank you, Google contributors & collaborators, for responding, can you confirm or deny the following too please?

Perhaps this is NOT undefined behavior though!? I added Mock::VerifyAndClearExpectations(&myMockObj).

TEST(FooTest, testCaseName)
{
    MyMock myMockObj;
    ...
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(true));
    testMethod();
    EXPECT_EQ(...);

    Mock::VerifyAndClearExpectations(&myMockObj); // <== NOTICE THIS ADDED LINE!
    EXPECT_CALL(myMockObj, myMethod(_)).WillOnce(Return(false));
    testMethod();
    EXPECT_EQ(...);
}

[mliszcz]

Hi GoogleTest maintainers. Would it be possible to get your feedback on the above question from @ElectricRCAircraftGuy?

One more use-case would be bringing system under test into desired state in the test fixture constructor. This cannot be worked around by simply reordering the statements. For now I used placement new and recreated the mocks but I don't like this solution.

struct MyTest : public Test
{
    MyServiceMock myService;
    MySystem mySystem;
    MyTest() :
        myService(),
        mySystem(&myService)
    {
        mySystem.goToStateX(); // this performs mock function calls on myService
        // Would calling VeryfyAndClearExpectations(&myService) here make it well-defined?
    }
};

TEST_F(MyTest, DoYInStateX)
{
    EXPECT_CALL(myService, ...);
    mySystem.doY();
}