Quarkus: KubernetesMockServer tests sensitive to runOrder/reuseForks

[jglick]

Describe the bug

The KubernetesMockServer test resource seems to carry over mocks between @QuarkusTest classes in the same JVM. Since Surefire defaults to reuseForks=true and runOrder=filesystem, this means that in a project with multiple test suites using the mock server, even if individual tests (mvn test -Dtest=SomeTest) reliably pass, the whole suite (mvn test) may fail with cryptic errors.

Expected behavior

Each @Test should get an independent mock context.

Actual behavior

Sometimes expect() calls seem to carry over between test classes, resulting in random failures.

To Reproduce

Do not currently have a public demo project, but could probably put one together. Noticed after introducing a new @QuarkusTest which caused an existing one to start failing in a CI environment, but not locally. Added

<runOrder>alphabetical</runOrder>

and tests passed (locally); with

<runOrder>reversealphabetical</runOrder>

they failed as on CI. Adding

<reuseForks>false</reuseForks>

seems to have fixed the issue.

Configuration

N/A

Screenshots

N/A

Environment (please complete the following information):

• Java etc.: quay.io/quarkus/centos-quarkus-maven:19.2.0
• Quarkus version or git rev: 0.21.2

Additional context

If the test framework itself cannot easily be fixed, at least the Maven archetype should preconfigure

<forkCount>1</forkCount>
<reuseForks>false</reuseForks>
<runOrder>alphabetical</runOrder>

for both maven-surefire-plugin and maven-failsafe-plugin. (forkCount is advisable due to #2269: Quarkus does not support running tests in parallel.)

[geoand]

@iocanel how can one reset the Mock server? I searched a little but I didn't come up with anything useful

[jglick]

Even with reuseForks=false, test methods within a class seem to reuse the mock server, which is pretty awkward: to create a genuinely independent test you need to create a fresh class. (Or maybe a nested class?) Of course there is a bit of overhead to start the app separately for each test class, but I think this is worth the reduced chance of wasting time tracking down interference between tests.

[geoand]

I agree 100%.

@iocanel do we have a way to reset the mock server? If so, then I can patch the Quarkus parts to call whatever is needed to reset it between tests

[iocanel]

I don't remember by heart.
Let me have a look.

[iocanel]

I am looking at the kubernetes-tests of the kubernetes-client itself and they use something like:

 @Rule
 KubernetesServer server = new KubernetesServer();

Which seems to do the trick!
Now, I am no expret in junit rules but from peeking the code it seems that internally it creates a new mock server and mock client per test case.

Does that work for you?

[geoand]

@iocanel I see. I'll have to see how this can be handled.

Thanks for checking!

[geoand]

@jglick If you would like to take a stab at it, I won't stop you :).
Just saying because I don't know when I will be able to get around to this one...

[gastaldi]

AFAIK KubernetesMockServer needs to be implemented as a JUnit 5 extension : https://stackoverflow.com/questions/51012335/how-to-replace-rule-annotation-in-junit-5