Spring-cloud-contract: Request mappings missing if a test context is reused in a test setup with a common test base class

[luddwichr]

Describe the bug

We use WireMock to stub responses for HTTP requests to external systems.
To reduce boilerplate code in our test classes, we introduced a test base class that handles general test setup for Spring Boot and WireMock.
The test classes may run in different test contexts (e.g. differing active profiles) that are shared across the test classes (no DirtiesContext used).

This setup worked fine so far. However, some tests fail when switching from version 2.2.2.RELEASE (via HOXTON.SR4) to version 2.2.3.RELEASE ( via HOXTON.SR5). It turned out that all tests of test classes fail, which run in a text context that was already used by a previous test class. The failure reason is that no request mapping was found, although the respective stubs were set up in the tests.

Potential reasons for this new behavior are changes that were made to WireMockTestExecutionListener and WireMockConfiguration in this commit. Unfortunatelly, I could not yet exactly figure out what the root cause is.

Sample

see my demo project on GitHub.

The tests in this project use a TestRestTemplate to send HTTP requests. A WireMock stub is used to simulate the response. To "simulate" different test contexts, the test classes are annotated with different active profiles. There are two different setups. In the root package, the test classes contain the entire test setup. In the subpackage inheritance, a base class is used to reduce code duplication / simplify the test setup.

If all tests in package com.example.demo are run, the tests pass. If all tests in package com.example.demo.inheritance are run, the test ZTest fails ("no request mapping was found").

[jordanms]

I have the same problem: https://stackoverflow.com/questions/64063372/multiple-application-contexts-in-spring-junit-test-with-wiremock

[marcingrzejszczak]

@luddwichr I took your sample, bumped contract to 2.2.6.BUILD-SNAPSHOT, bumped Boot to 2.3.5.RELEASE and all of the tests are passing. What should I do to make them fail?

[jordanms]

@luddwichr I took your sample, bumped contract to 2.2.6.BUILD-SNAPSHOT, bumped Boot to 2.3.5.RELEASE and all of the tests are passing. What should I do to make them fail?

I tested with Boot 2.3.5.RELEASE and contract 2.2.5.RELEASE and test Z fails. My test execution order is: B, A, X, Y, Z, C.

@marcingrzejszczak Which repository you use to get the snapshot version?

[marcingrzejszczak]

Add this section

<repositories>
                <repository>
                    <id>spring-snapshots</id>
                    <name>Spring Snapshots</name>
                    <url>https://repo.spring.io/snapshot</url>
                    <snapshots>
                        <enabled>true</enabled>
                    </snapshots>
                    <releases>
                        <enabled>false</enabled>
                    </releases>
                </repository>
                <repository>
                    <id>spring-milestones</id>
                    <name>Spring Milestones</name>
                    <url>https://repo.spring.io/milestone</url>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </repository>
                <repository>
                    <id>spring-releases</id>
                    <name>Spring Releases</name>
                    <url>https://repo.spring.io/release</url>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </repository>
            </repositories>
            <pluginRepositories>
                <pluginRepository>
                    <id>spring-snapshots</id>
                    <name>Spring Snapshots</name>
                    <url>https://repo.spring.io/snapshot</url>
                    <snapshots>
                        <enabled>true</enabled>
                    </snapshots>
                    <releases>
                        <enabled>false</enabled>
                    </releases>
                </pluginRepository>
                <pluginRepository>
                    <id>spring-milestones</id>
                    <name>Spring Milestones</name>
                    <url>https://repo.spring.io/milestone</url>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </pluginRepository>
                <pluginRepository>
                    <id>spring-releases</id>
                    <name>Spring Releases</name>
                    <url>https://repo.spring.io/release</url>
                    <snapshots>
                        <enabled>false</enabled>
                    </snapshots>
                </pluginRepository>
            </pluginRepositories>

[jordanms]

@marcingrzejszczak it fails.

[marcingrzejszczak]

It doesn't for me. We'd need to understand what's the difference between your code and mine. Can you show the execution of your tests (preferably with trace log level for Spring Cloud Contract) ?

[luddwichr]

@marcingrzejszczak First of all, thanks a lot that you are taking care of this issue! Much appreciated!

I ran the tests again, always with same order as mentioned by @jordanms - B, A, X, Y, Z, C. The tests fail (i.e. test Z) with any combination of Spring Boot 2.3.5, 2.4 and Wiremock 2.2.3 RELEASE, 2.2.5 RELEASE and 2.2.6 BUILD-SNAPSHOT (using the pom fragment of your previous comment).

The tests only run successfully for 2.2.2 RELEASE. What is your test execution order?

Unfortunatelly, at the time I could not find a fast and easy way to make the test execution order deterministic :/. Are you aware of any means to achieve that?

[marcingrzejszczak]

Hmm I have a different order - B, A, C, Y, X, Z

[marcingrzejszczak]

Hey @luddwichr can you run those tests with logs in trace mode and attach it as a file? That could be handy

[marcingrzejszczak]

I've managed to replicate this locally by running ./mvnw verify -Dsurefire.runOrder=random, now I can work on it

[luddwichr]

@marcingrzejszczak Looks like you already resolved the problem - cool! I guess you don't need the logs anymore? I'ld delete them from the comments to make the issue a bit more compact...