Junit5: Introduce @TestClassOrder for ordering test classes in JUnit Jupiter

[jflefebvre06]

Like @TestMethodOrder, could you please introduce a @TestClassOrder annotation to define the execution order of test classes?

[marcphilipp]

Where would you declare that? On a package?

[marcphilipp]

Tentatively slated for 5.6 M1 for _team discussion_.

[sormuras]

Where would you declare that? On a package?

And a @TestPackageOrder on a module!

@TestPackageOrder(...)
open /*test*/ module foo {}

[pietrygamat]

Where would you declare that? On a package?

It would be a welcome addition on a root class holding multiple @Nested classes. (Or do I miss something, and there already is a way to control execution order of @Nested classes?).

[vlsi]

I've a use case for the ordering: run faster tests first.

Of course, in an ideal world, each test method executes under 1 millisecond.

In practice that does not hold, and Apache Calcite has multiple tests that take 1-5-15 seconds (see https://github.com/apache/calcite/blob/52bc246d5e07510689635f407f4d9dc494b0af4d/core/src/test/java/org/apache/calcite/test/CalciteSuite.java#L197 ).

It would be nice if there was a way to tag method and/or class and/or package with something like @TypicallyTakes("10 seconds"), so JUnit would arrange test class/method execution in such a way so it executes faster tests first (so user gets immediate feedback faster) or slower tests first (to reduce overall test time by allowing more parallelism).

[yuraku]

hey 👋. Is there any update on this issue? any plans for implementing it? thanks

[marcphilipp]

@yuraku I'm afraid this issue slipped through the cracks because I forgot to actually assign it to a milestone for _team discussion_. I've now rectified this mistake, and it's now in the backlog for the team to discuss. 😉

[marcphilipp]

We briefly discussed this issue in yesterday's team call. It's not yet specific enough to decide whether work on it should start. It should definitely work similar to @TestMethodOrder but where would you define the annotation? Should we support it both on packages and modules? Should there be a way to define a default orderer as a configuration parameter? Which implementations would we include by default (OrderAnnotation and Random?)?

[yuraku]

@marcphilipp thanks for getting back. As for me it should support both packages and modules. I think it might be defined (used) as a class annotation to specify the order place. Another option that might help is to extend functionality of @SelectClasses, @IncludeTags to accept the items in the order, that needs to be executed (ie. signup (1), signing (2), someotherstuff (3)).

[Biriuck]

Another option that might help is to extend functionality of @SelectClasses, @IncludeTags to accept the items in the order, that needs to be executed (ie. signup (1), signing (2), someotherstuff (3)).

@_SelectClasses_ is already taking the the test classes in the proper order. The only thing it will need is to be able to run suite classes with _useJUnitPlatform_() and _includeTags_ (like 'Fast','NotSoFast', 'Slow') . Sometimes you need to not only run the test classes in a specific order but also to filter in/out some of the @Test methods

[vlsi]

I wonder how JUnit's class/method ordering plays with/against Gradle's
ordering.

For instance, Gradle knows which tests failed during the previous run, and
it sends the failing tests first. It would be sad if JUnit shuffles that.

>

[Biriuck]

I have a workaround for this....
In Gradle I have:

test{
       //....systemProperty.......
       ignoreFailures=true
       filter{
               includeTestMatching "com.mycompany.auto.tests.suites.MyOrderedSuite"
       }

       useJUnit()  // !!! NOT useJUnitPlatform()
}

In JUnit I have:

import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.suite.api.ExcludeTags;
import org.junit.platform.suite.api.IncludeTags;
import org.junit.platform.suite.api.SelectClasses;
import org.junit.runner.RunWith;

import com........TestA;
import com.........TestB;

@RunWith(JUnitPlatform.class)
//@SelectClasses({ TestA.class, TestB.class })
@SelectClasses({ TestB.class, TestA.class })
@IncludeTags({ "TAG1", "TAG2" })
@ExcludeTags({ "TAG3" })
public class MyOrderedSuite{}

This is not only executing the TestB class test methods before the ones from TestA class, but is also running only the test methods tagged with @TAG1 and @TAG2, excluding the ones tagged with @TAG3

[ledoyen]

I share the same need than @vlsi of having faster tests run first.

It would be great if we could define our own stragegy based on annotations, lets say @SlowTest to mark tests than will run in last

[vlsi]

It is "funny" that build systems (e.g. Gradle) have their own notion of the desired test order: https://github.com/gradle/gradle/pull/12012

The build system knows better what was the actual test outcome and test duration during the previous runs.

So it would really be sad if JUnit reorders tests requrested by the build system :-/

[sormuras]

So it would really be sad if JUnit reorders tests requrested by the build system :-/

We users control build tools, test ordering, ... everything.

Build tools should adhere to choices users make in the first place. Then they can optimize on that basis as much as they can. Just imagine, I want my test order to be shuffled on every run. No build tool should interfere here.

[vlsi]

Exactly, so there should be:
a) An interface from the build tool to control test execution order. Or something more higher-level like optimize-for-overall-time (~schedule slowest tests first for maximum parallelism) vs optimize-for-first-failure (~scheduler fastest tests first) vs optimize-for-bugs-found (~randomize order)
b) Of course, the implementation of #a should be represented somehow in the JUnit APIs. However, #a means that the setting passed by the build system must override whatever is written in @TestClassOrder or @TestMethodOrder.

[JiangHongTiao]

Hi guys, yes for us it would be very helpful to have this annotation. Unfortunately we have a old code with some static non-final properties mixed with system properties. So sometimes it depends on how VM loads classes and how some of initialization blocks are executed. Sad, but it's like that.
It would be very helpful for us to shuffle test classes and also be able to reproduce this shuffle in a dev environment so we can debug it and fix it.

From our perspective what we really need is:

• To have an annotation (package/module) where we can specify the order of classes (@TestClassOrder)
• Able to specify a seed for random shuffle (as for methods) - CLI params override code params
• Respecting tags are not necessary for us, it's rather nice to have.

@vlsi I think optimize-for-overall-time might be the next step. And for that there can be another annotation introduced, similarly like @Order there can be @TypicalDuration or something like that. Then you may add another MethodOrderer

I'm not sure what do you mean by optimize-for-first-failure, since JUnit is only executing tests, it doesn't store any execution statistics if I'm not mistaken. Do you want to introduce also those statistics storage? If so, then it can store also execution time stats, but it sees for me like a bigger feature.

[stuartwdouglas]

I have a similar use case where I want to run all tests with a similar config one after the other, to avoid having to setup/tear down the environment each time.

The annotation based approach would not really work for me, but it would be great if extensions could control this somehow.

[asaikali]

I have multi module project ,module A, B, C ... etc. I want to be able to run some code just before all the tests in module A run and just after all the test in module A finish. Would be nice to have a per module lifecycle callbacks. Marking a class as a per module Lifecycle callback would be wonderful.

[asaikali]

@ledoyen i am using gradle for this project. So I don't want count on the maven or gradle I would like to be able to right click on a src/test/java run all tests from the IDE and still have pre-all test and post-all tests callback run once.

I am using SpringBoot and for my integration tests I want to be able to inject dependencies implementing org.junit.platform.launcher.TestExecutionListener means the code is executing without any knowledge of spring test context. Implementing an interface from the Launcher seems way too low level. What if there was an annotation @BeforeTestPlanExecution and @AfterTestPlanExecution that would do make things much easier and have these callbacks be the same as other callbacks such as @BeforeEach ... etc.

[nishantvas]

Apart from being able to manage order of test execution on a whole, the most immediate use case would be

It would be a welcome addition on a root class holding multiple @Nested classes. (Or do I miss something, and there already is a way to control execution order of @Nested classes?).

Unless there is already a way of controlling execution order of nested classes

[rcsnogm]

Can it include duplicates?
For example we have some Junit 4 test suites that do something like this:

@Suite.SuiteClasses({        
    //Run tests starting with initial state
    InitialStateTest.class,
    Feature1Test.class,
    Feature2Test.class,

    //Begin second state
    SecondStateTest.class,
    Feature1Test.class,
    Feature2Test.class,

    //Begin third state
    ThirdStateTest.class,
    Feature1Test.class,
    Feature2Test.class
})

Which runs the test classes in this order:

InitialStateTest.class,
Feature1Test.class,
Feature2Test.class,
SecondStateTest.class,
Feature1Test.class,
Feature2Test.class,
ThirdStateTest.class,
Feature1Test.class,
Feature2Test.class

Note how the FeatureXTest classes are repeated. I tried using @SelectClasses like so:

@SelectClasses({
    //Run tests starting with initial state
    InitialStateTest.class,
    Feature1Test.class,
    Feature2Test.class,

    //Begin second state
    SecondStateTest.class,
    Feature1Test.class,
    Feature2Test.class,

    //Begin third state
    ThirdStateTest.class,
    Feature1Test.class,
    Feature2Test.class
})

Which runs as follows without repeating the FeatureXTests:

InitialStateTest.class,
Feature1Test.class,
Feature2Test.class,
SecondStateTest.class,
ThirdStateTest.class

Maybe there is another way to do this in JUnit 5?

[marcphilipp]

Why would you like to run the same test classes multiple times?

[rcsnogm]

@marcphilipp To model a legacy JUnit 4 test suites that have some extensive state changes. As illustrated in the example. They are used for performance/integration tests. They are not unit tests. We would rather migrate them vs. re-write. This is the only thing preventing a complete migration to JUnit 5. Love to ditch the Junit 4 dependencies and use of vintage engine after bug #2295 adds support for suites without using @RunWith(JUnitPlatform.class). Assuming I understand the intent of the bug.

[wimdeblauwe]

Would like this as well. For me the ordering would be based on the Spring Boot test slices used. So first unit tests without any Spring context stuff, then @JsonTest, then @DataJpaTest, then @WebMvcTest and end with @SpringBootTest.

[ledoyen]

From what I understand there are many use cases, that can be covered by the equivalent of the org.junit.jupiter.api.MethodOrderer but defined at the root level.

The addition of an optional configuration parameter (something like junit.jupiter.execution.class.order.config.custom.class) and a org.junit.jupiter.api.ClassOrderer extension point should cover most of it.

If the core team wants to keep a way of adding built-in features here (such as marking a class with TestClassOrder as suggested in the first place), an optional configuration parameter junit.jupiter.execution.class.order.strategy with values being default or custom (in which case the previous parameter become mandatory) can be added.
This way more built-in strategies can be added in the future.

Let me know if this is something you would be interrested in, I can work on it.

[marcphilipp]

Team decision: We feel that this issue has gained enough community interest and are therefore willing to consider a PR along the lines of the proposal summarized in https://github.com/junit-team/junit5/issues/1948#issuecomment-737809799.

@ledoyen Feel free to start working on your PR if you're so inclined.

[sbrannen]

Closing this issue since the new feature has been introduced in conjunction with the PR: https://github.com/junit-team/junit5/pull/2488#issuecomment-769255868