Spring-boot: Offer a way for MockitoTestExecutionListener to enable strict stubbing

[jschmied]

Running tests with SpringRunner does not report unused stubs like MockitoJUnitRunner does.

In MockitoTestExecutionListener.initMocks it should use

Mockito.mockitoSession()
  .initMocks(this)
  .strictness(Strictness.STRICT_STUBS)
  .startMocking();

to enable strict mocking.

[bclozel]

This feature is still being debated by the community in mockito/mockito#769, and I'm not sure we're in a position to force that choice right now on the Spring Boot community. Looking at the comments in that issue, it seems that some developers had issues while applying this to their projects; arguable, some of those might be linked to their setup or even bad practices.

If we do consider that change, we should do that in a proper milestone and document that change properly. I'm marking this issue for team attention and we'll discuss this.

In the meantime @jschmied, don't hesitate to point us to official Mockito documentation/best practices about this feature.

[jschmied]

The discussion you are refering ist ourdated. Since Mockito 2.+, strict stubbing is used by default when initializing our mocks using either of:

MockitoJUnitRunner
MockitoJUnit.rule()

Spring ist using Mockito.initMocks() instead of Mockito.mockitoSession().initMocks(this).strictness(Strictness.STRICT_STUBS).startMocking(); what is causing this problem.

See: https://www.baeldung.com/mockito-unnecessary-stubbing-exception

So by default (We are at Mockito 2.23 in Spring Boot 2.1) all plain JUnit test use strict stubbing by default but all Sprint tests do not.

This behaviour is inconstent and anoying.

Of course, enabling it would possibly force users to clean up their spring tests. All plain JUnit test had to be cleaned up with switch to spring boot 2.x already.

It would be nice to be able it at least by some property or annotation (or in my oppinon, better switch it on and give user the possibility to disable it easily)

[wilkinsona]

As far as I can tell, the issue to which @bclozel linked remains accurate. MockitoJUnit.rule() calls JUnitRule(Plugins.getMockitoLogger(), Strictness.WARN) in Mockito 2.28 and continues to do so in Mockito 3.1.

[jschmied]

I have seen nobody using MockitoJUnit.rule(), anybody uses MockitoJUnitRunner.

Fact is: if you have two JUnit test, one with

@RunWith(MockitoJUnitRunner.class)

one with

@RunWith(SpringRunner.class)
@SpringBootTest()

the first one will fail when there are unnecessary stubs (default is STRICT), the second one will pass.

You can test easily.

[wilkinsona]

On further inspection, as far as I can tell, Mockito does not use STRICT_STUBS by default in any situation. As with the rule, the runner's default behaviour is to report problems to the console. This is the equivalent of Strictness.WARN. Strictness.STRICT_STUBS is "planned as default for Mockito v4".

I think we could safely switch to Strictness.WARN. It will result in some additional console output containing hints on how to improve the tests' usage of Mockito but, unlike STRICT_STUBS, it will not cause test failures. I think there's even an argument to be made to do this in a Spring Boot maintenance release so that we're aligned with Mockito's default behaviour. We would then switch to Strictness.STRICT_STUBS by default when we upgrade to a version of Mockito that does so.

[jschmied]

This would be nice. I usually run JUnit from eclipse first and unused stubs make the tests failing in the IDE. I did not test this as part of maven build on Jenkins. So you might be right.

[wilkinsona]

@jschmied Could you share some example tests with us please? It would be useful to see exactly what you are seeing and to confirm that Mockito's behaviour matches its documentation.

[jschmied]

Sure.
This one fails with UnnecessaryStubbingException:

@RunWith(MockitoJUnitRunner.class)
public class TestJUnit {
    @Test
    public void test1() {
        List<Object> list = mock(List.class);
        when(list.get(anyInt())).thenReturn(new Object());
    }
}

This one passes:

@RunWith(SpringRunner.class)
@SpringBootTest
public class TestSpring {

    @Test
    public void test1() {
        List<Object> list = mock(List.class);
        when(list.get(anyInt())).thenReturn(new Object());
    }

}

Only the annotations on top differ

[wilkinsona]

Thank you, @jschmied. I can see now that I was wrong before. Sorry. Mockito does fail by default when using MockitoJUnitRunner. It does not fail by default when using MockitoRule but does output some hints to the console.

The concept of strictness differs between the runner and the rule. There are three variants of the runner:

• Silent
• Strict
• StrictStubs

There are three variants of Strictness used by the rule:

• LENIENT
• WARN
• STRICT_STUBS

The Silent runner is equivalent to the LENIENT rule. The StrictStubs runner is equivalent to the STRICT_STUBS rule. The Strict runner and the WARN rule are not equivalent. The former fails when there is unnecessary stubbing while the latter only outputs hints to the console.

An equivalent of the Strict runner doesn't appear to be available to us as there's no equivalent Strictness. STRICT_STUBS would be too much as it's stricter than Mockito is by default even when using the runner. Using WARN by default in Boot feels reasonable to me. It matches the default of the rule and is no more strict than the default of the runner.

[philwebb]

I think there's currently too much risk in changing our defaults at this point. The MockitoTestExecutionListener.initMocks method currently just delegates to MockitoAnnotations.initMocks (from the org.mockito package).

As soon as the Mockto team decide that MockitoAnnotations.initMocks should switch to a stricter form, we'll pick it up by virtue of an upgrade.

The only problem that we see with our current approach is that we don't offer a good way to change the defaults for an individual test. I think we should investigate offering a way to switch the strictness for an individual test.

[BloodShura]

+1. It would be nice if we could change the strictness used in a test. Even better if you could support @MockitoSettings(strictness = STRICT_STUBS).

[mminot-yseop]

Indeed, it took me a while to understand why my unit and integration tests were behaving differently, and noticing that @MockitoSettings(strictness = STRICT_STUBS) didn’t help with @SpringBootTest was quite a letdown. For now I’ll have to resort to something like…

@AfterEach
void afterEach() {
    // Only stubbed (or otherwise expected) calls allowed.
    ignoreStubs(/* explicit list of mockbeans here */);
    verifyNoMoreInteractions(/* the same list */);
}

This is not really ideal (and does not provide the usual nice helpful errors for unnecessary stubbings).

[iekdosha]

A workaround to manually set the strictness would help.