Junit5: Provide mechanism to order the sequence of tests

[elygre]

Overview

Although it is typically discouraged, having a way to order tests is useful to some people. The JUnit 4 @FixMethodOrder annotation specifically and deliberately makes it impossible to implement a simple ordering scheme, forcing users who want/need this to resort to method names like test245_ensureReasonableTestName().

Proposal

• The minimal "requirement" would be to allow users to write their own MethodSorter for @FixMethodOrder
• A reasonable implementation would be to allow @FixMethodOrder(ORDERED), and then @Order(integer) to specify individual tests. Tests could be run from smallest number to highest, with non-annotated methods having a default of 0. Methods with the same order could be run in parallel; different order tiers could not.

Related Issues

• #48
• #607
• #884
• #1707

Deliverables

• [x] Introduce a mechanism for ordering _testable_ methods within JUnit Jupiter.
• [x] Validate collection of methods ordered by an extension; log warning or error if methods were added or removed.
• [x] Introduce built-in support for alphanumeric ordering.
• [x] Introduce built-in support for @Order based ordering.
• [x] Introduce built-in support for random ordering.
• [x] Introduce _Configuration Parameter_ for setting the seed used in random ordering.
• [x] Test ordering support in top-level classes.
• [x] Test ordering of @Test, @TestFactory and @TestTemplate methods.
• [x] Test ordering support in @Nested classes.
• [x] Test ordering support in conjunction with parallel test execution.
• [x] Extract junit.jupiter.execution.order.random.seed constant and document it.
• [x] Write Javadoc for all new types.
• [x] Update @API declarations for types with changed visibility, etc.
• [x] Document new features in User Guide.
• [x] Document new features in Release Notes.

Out of Scope

• ❌ Consider introducing a mechanism for ordering tests with the JUnit Platform -- potentially at a later date in conjunction with a separate dedicated issue.

[sbrannen]

Hi @elygre,

Test execution order is something we are certainly considering for the core feature set of JUnit 5.

In fact, introducing an annotation such as @Order is something we've already discussed.

We'll chime in here once we begin work on _ordering_.

Cheers,

Sam

[kcooney]

@elygre ine thing we considered was something like https://github.com/junit-team/junit/pull/1130

[ttddyy]

In addition to test method, I believe ordering should be applicable to @BeforeAll/@AfterAll/@BeforeEach/@AfterEach as well.

Also, it would be nice if behavior are clearly defined(or documented) when multiple @BeforeAll/@AfterAll/@BeforeEach/@AfterEach are presented. (i.e. within same class, via inheritance, or via interface).
Currently, it is HierarchyDown and HierarchyUp, but not defined for cases multiple annotations are presented within the same class.(I think it is JDK's returuning order for now.)

Thanks,

[dweiss]

I like the changes and the new API proposal, although (sigh) I'm not entirely convinced resignation from @RunWith is a good idea. My point is that with JUnit4 you had a set of contracts which you could replicate in your custom runner. With this new API tweaking certain things seems impossible if there's no support for them in the "default" engine... at least that's my impression from looking at the code/ specs so far.

Take method ordering. The way I implemented it for randomized testing is by adding an annotation that provides a class which receives a list of tests to be executed:

https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/annotations/TestCaseOrdering.java#L32

Would this kind of flexibility be possible with JUnit5?

[jlink]

Well, you can always build and register your own test engine. If it
subclasses from the junit5 engine, anything should be possible. But that's
a mere guess. We haven't really thought about one test engine replacing
another...

2015-11-23 22:39 GMT+01:00 Dawid Weiss [email protected]:

I like the changes and the new API proposal, although (sigh) I'm not
entirely convinced resignation from {{RunWith}} is a good idea. My point is
that with JUnit4 you had a set of contracts which you could replicate in
your custom runner. With this new API tweaking certain things seems
impossible if there's no support for them in the "default" engine... at
least that's my impression from looking at the code/ specs so far.

Take method ordering. The way I implemented it for randomized testing is
by adding an annotation that provides a class which receives a list of
tests to be executed:

https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/annotations/TestCaseOrdering.java#L32

Would this kind of flexibility be possible with JUnit5?

—
Reply to this email directly or view it on GitHub
https://github.com/junit-team/junit-lambda/issues/13#issuecomment-159073860
.

[dweiss]

That's my point. What I liked about JUnit4 was the fact that contracts (ok, not clearly formalized, but still there) were somewhat separate from the "execution". I hoped for JUnit5 to follow this paradigm. Specify annotations, interfaces and the execution model, leaving actual implementation out (but providing a default reference).

I am not saying the route taken is wrong. I just express my expectations and desires. :)

[jlink]

I still don't see how extending an existing test engine is qualitatively
different from extending junit4's default runner - and this is what many of
the more powerful runners actually do. Maybe you can clarify?

JUnit5TestEngine is not designed with "subclassability" in mind but that
could be changed...

2015-11-23 23:17 GMT+01:00 Dawid Weiss [email protected]:

That's my point. What I liked about JUnit4 was the fact that contracts
(ok, not clearly formalized, but still there) were somewhat separate from
the "execution". I hoped for JUnit5 to follow this paradigm. Specify
annotations, interfaces and the execution model, leaving actual
implementation out (but providing a default reference).

I am not saying the route taken is wrong. I just express my expectations
and desires. :)

—
Reply to this email directly or view it on GitHub
https://github.com/junit-team/junit-lambda/issues/13#issuecomment-159085026
.

[dweiss]

Let me explain the underlying need a bit. I am a committer to Lucene and Solr. We use a lot of pseudorandomness in tests. Interchangeable components can get swapped, parameters can get picked at random, strings get picked from a random space, etc. Simplifying a lot, every test is still reproducible because the source of randomness is controlled by the framework. There are also tons of other infrastructure elements: parameter factory methods (parameterized tests), more realistic test and suite timeouts that detect and warn about stray threads left behind, different seed test reiterations, test listeners (they can attach to the runner, so they receive suite started/ finished events), custom test method providers... Lots of it. If you're interested in more details and the philosophy behind it there are some lectures I presented in the past about the topic [1, 2].

The reasons we chose JUnit as the underlying framework were basically twofold:

• virtually everybody knows (more or less) how to use it and is more or less familiar with the default contracts (the test class is loaded once, test methods are invoked on a new instance of the class, before/after hooks wrap around the test or class, etc.),
• there is strong tooling support in every IDE, CI system, reporting tools, you name it.

But the two above don't really mean the test runner is the same as in regular JUnit. It is similar, but has multiple differences that go well beyond the default implementation. Most of these are so low-level that they couldn't be implemented by extending base JUnit4 runner because it encapsulates or hides certain test preprocessing points (I've tried). Or it combines functionality scattered across different runners (for example parameterized tests). Finally, having our own runner makes the implementation less tightly bound to JUnit's version -- the details of the base runner's implementation may change, but all we use from JUnit is basically annotations and the (unfortunate) Description object.

Perhaps an example would highlight it better. Take this class as a simple example:

@RunWith(RandomizedRunner.class)
public class TestFoo {
  final int a;

  public TestFoo(int a) {
    this.a = a;
  }

  @Test
  @Repeat(iterations = 3, useConstantSeed = false)
  public void checkMe() {
    Assert.assertTrue(Math.abs(a + randomIntBetween(0, 3)) >= 0);
  }

  @ParametersFactory(argumentFormatting = "a = 0x%08x")
  public static Iterable<Object[]> parameters() {
    return Arrays.asList(
        $(1),
        $(0x7fffffff - 2));
  }
}

The output of running this is Eclipse is 6 tests, with a pseudo-nested suite above.

Note the tests need to have unique Descriptions (otherwise Eclipse gets terribly confused) and this description is generated at the runner's level (the runner also expands the hierarchy so that test repetitions for different seeds and arguments are unique and placed under the method's node).

Finally and again -- I'm not saying anything in JUnit is badly implemented. We just need very specific stuff that goes beyond the defaults (and would occasionally require stuff that is hidden from subclasses for the right reasons). So if there is a way to extract a "meta test API" level and abstract it from the actual runner implementation it would be beneficial.

[1] https://berlinbuzzwords.de/sites/berlinbuzzwords.de/files/media/documents/dawidweiss-randomizedtesting-pub.pdf
[2] https://berlinbuzzwords.de/session/randomize-your-tests-and-it-will-blow-your-socks

[marcphilipp]

Closed in favor of #48.

[devrandom]

There is a specific need to have a pseudo-random test order, since that allows finding unintended test interactions, and allows repeating the test in that order using the logged seed.

48 does not cover that use case.

[marcphilipp]

@devrandom Please open a new issue for the pseudo-random order use case.

[devrandom]

@marcphilipp - I see actually that junit4 has a deterministic test order (based on String.hashCode) by default. I couldn't quickly find the default ordering in junit5. Can you tell me where it is defined?

[sbrannen]

I couldn't quickly find the default ordering in junit5. Can you tell me where it is defined?

You couldn't find it... because it's simply not defined.

The ordering of execution of tests within a test class in JUnit Jupiter is based solely on the order in which the methods are returned from the JVM via reflection, which since Java 7 is undefined. In other words, there is no guaranteed order.

[sbrannen]

Reopening in order to address concerns raised in #884 and #48.

[sbrannen]

Introduced _Deliverables_ section.

[Andrei94]

How about introducing the following: add an annotation @ScenarioBuilder for a method and make that method return Collection<String> which would hold the order of the tests.
In order to create the setup and teardown of the scenario, I believe the @BeforeClass and @AfterClass would work just fine. Also, the @BeforeEach and @AfterEach could work just the way they are.

```
public class MyScenario {
@ScenarioBuilder
public Collection buildScenario() {
return new ArrayList() {{add("test1"); add("test2"); add("test3");}};
}

@BeforeClass
public void setUpScenario() {
}

@BeforeEach
public void setUpTest() {
}

@Test
public void test1() {
}

@Test
public void test2() {
}

@Test
public void test3() {
}

@AfterEach
public void tearDownTest() {
}

@AfterClass
public void tearDownScenario() {
}

}

````
This way, the list of tests to be run could also be filtered, like with categories in JUnit4, if one so desires.

Any thoughts about this?

[RupprechJo]

This feature is absolutely essentially for migration existing tests to JUnit5. Yes, I know all the arguments like "Unittests should have no order". And I agree totally for real unit tests. But JUnit is a de facto standard for running every kind of test, also integration tests. And those tests can have strong arguments for ordering. The most obvious is performance. If the setup of an integrationtest runs several minutes, like setting up a database, you don't want to do this for every method in that test. And then you have to order the test methods to get predictable results. Sometimes the world is not so nice, as it should be.

In our group of companies we have in total about 10.000 test methods in hundreds of test classes. To be able to migrate them, we need ordering ot tests, or we simply cannot migrate. It would also be OK, if this is possible by an extension, that we can implement.

[mlevison]

Suggestion - create an official extension/runner for integration tests. All
things that are tied to integration tests should be bundled/packaged there.

Reasoning: Since I first found JUnit in 2001 (a best guess), I've been
struggling to help people focus on writing simple clean Unit Test (or
MicroTests if you prefer). I get concerned when features get added to JUnit
(and NUnit) that weaken that focus. All to often I show up at clients and
find them using convoluted setup/teardown when a real Unit Test was hiding
underneath. I see some value in Integration tests but would prefer the
infrastructure be kept separate so people have to make an explicit decision
to use them.

Cheers
Mark

On Thu, Aug 10, 2017 at 2:48 PM, Johannes Rupprecht <
[email protected]> wrote:

This feature is absolutely essentially for migration existing tests to
JUnit5. Yes, I know all the arguments like "Unittests should have no
order". And I agree totally for real unit tests. But JUnit is a de facto
standard for running every kind of test, also integration tests. And those
tests can have strong arguments for ordering. The most obvious is
performance. If the setup of an integrationtest runs several minutes, like
setting up a database, you don't want to do this for every method in that
test. And then you have to order the test methods to get predictable
results. Sometimes the world is not so nice, as it should be.

In our group of companies we have in total about 10.000 test methods in
hundreds of test classes. To be able to migrate them, we need ordering ot
tests, or we simply cannot migrate. It would also be OK, if this is
possible by an extension, that we can implement.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/junit-team/junit5/issues/13#issuecomment-321640236,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAImZqMujOmdrpuKhwO9_DfpNMtjdduxks5sW1CTgaJpZM4GkiVa
.

--

[image: headshot-square-300x300]
http://www.flickr.com/photos/36331075@N00/9674877046/ Mark Levison | 1
[phone] | Twitter https://twitter.com/mlevison | LinkedIn
http://ca.linkedin.com/in/marklevison | Facebook
https://www.facebook.com/agilepainrelief
Certified ScrumMaster Training: Vancouver
http://agilepainrelief.com/courses/vancouver | Edmonton
http://agilepainrelief.com/courses/edmonton | Ottawa
http://agilepainrelief.com/courses/ottawa | Montreal
http://agilepainrelief.com/courses/montreal | Toronto
http://agilepainrelief.com/courses/toronto
Certified Product Owner & Private Training also available ~ Our Training
Schedule http://agilepainrelief.com/courses/certified-scrum-agile-training
Agile Pain Relief Consulting http://agilepainrelief.com/ | Notes from a
Tool User http://agilepainrelief.com/notesfromatooluser
Proud Sponsor of Agile Tour Gatineau Ottawa http://goagiletour.ca/ and Agile
Coach Camp Canada http://agilecoachcampcanada.wordpress.com/

[marcphilipp]

This issue won't make it into 5.0, but we're certainly considering it for 5.1.

[RupprechJo]

5.1 would be fine. Did you already plan, when to release the 5.1?

@mlevison: I understand your point, but the reality is like I described it.

[marcphilipp]

The current focus is to ship 5.0. After that's done the team will revisit all issues currently assigned to 5.1 Backlog and plan which will go into the next release. Having said that, there might pop up some urgent issues after we release 5.0. In general, we intend to release by far more often than it was the case with JUnit 4, a.k.a. Vintage. Thus, I would expect 5.1 to be released 1-2 months after 5.0.

[kcooney]

If the setup of an integration test runs several minutes, like setting up a database, you don't want to do this for every method in that test

@RupprechJo I suggest doing this in a @BeforeClass method (@BeforeAll in JUnit 5) or in a @ClassRule. Then in your @After/@AfterEach method you can clear out the database (some people do this by starting a transaction in @Before and rolling it back in @After). Note in JUnit 5 you can now make your @BeforeAll methods non-static if you are willing to share a single instance of your test class among all of the test methods.

The advantage of using @BeforeClass ,etc, is that developers can run/debug a single test method in their IDE or build tool and the database would still be created before the test method starts.

If you do this and you still have unpredictable results then that likely means that running one test method affects the next one. This can be a difficult situation to be in, since it makes it hard to add tests for disable tests.

[VladRassokhin]

From CI perspective test classes reordering during tests invocation is extremely useful to run previously failed tests (classes) first to get feedback earlier. Maybe it's already could be done somehow during discovering?

[IgnatBeresnev]

This issue won't make it into 5.0, but we're certainly considering it for 5.1.

@marcphilipp, since it didn't make it into 5.1, how about 5.2? Any plans/updates?

[sbrannen]

It's in the 5.2 Backlog. So, yes, we are tentatively planning this for 5.2.

[yuranos]

Sounds good, but for the "tentatively" part. Scenarios are very important for all sorts of tests higher up the testing pyramid. This feature can also be very useful for Consumer Driven Contracts, for example. Running consumer-driven scenarios against your API will usually imply retaining a specific state.

[sbrannen]

@yuranos, I agree with you wholeheartedly.

It's not that we are "tentatively" planning on doing it. On the contrary, we are definitely planning on doing it. The only "tentative" part is in which release it will be included.

Having said that, it's now _tentatively_ planned for 5.3. 😉

[dasAnderl]

are you kidding me ? you cant set the execution order of tests? this was opened 3 years ago!
this is a very fundamental feature no ?. is there any prospect when this will come?

[jbduncan]

@dasAnderl I think you'd be surprised just how far one can go without needing one's tests to run in a certain order!

In my experience, most tests are actually very independent of one another, even when they're in the same test class, so it's usually not that important that they run in a specific order. :wink:

But having said this, I do understand the frustration of seeing a feature request that's apparently key to a project's success but not being implemented despite being open for _n_ years. So try to hold onto that frustration; the JUnit 5 team is quite small, so there's only so much they can do when there's hundreds of issues to deal with; they will eventually get around to this issue, but it will take time. :)

[dasAnderl]

We do big test automatization projects for german car companies. although im aware that in most cases workarounds can be found for the issue, this does not work for us. we need to be able to execute tests (teststeps) in a testclass in a defined order. this feature is an absolute must for any testing framework. for us this probably means we have to go back to junit4 :/

[RupprechJo]

As I posted a year ago, I totally agree to @dasAnderl. We also cannot migrate our 10.000+ tests to junit 5 due to that open ticket.

[sbrannen]

Hi everybody,

Yes, we hear your cries and concerns, and we will implement support for ordering test methods within a given test class.

This is still tentatively slated for the 5.3 backlog; however, realistically this will be addressed in 5.4 later this year.

We apologize sincerely for the delay.

[dasAnderl]

Its great to hear that JUnit will implement this feature. however i migrated our new testing project to TestNG. Ordering is one reason (although possible in JUnit 4), but motivating reason was dependsOn in TestNG. This goes much further as just ordering, it allows for skipping tests on failed preceding tests or unmet assumptions in previous test. i could not find a way to do this in JUnit 4 without using a custom runner. see this TestNG example:

package seat.mod.testing.testhelper;

import org.assertj.core.api.Assertions;
import org.assertj.core.api.Assumptions;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestNgTest {

    @DataProvider
    public Object[][] getData() {
        return new Object[][]{{5, "five"}, {6, "six"}};
    }
    @Test(dataProvider="getData")
    public void parametrizedTest(int p1, String p2) {
        System.out.println("Instance DataProvider Example: Data(" + p1 + ", " + p2 + ")");
    }

    @Test
    public void C_failedAssumption() {
        Assumptions.assumeThat(false).isTrue();
    }

    @Test( dependsOnMethods = {"C_failedAssumption"})
    public void B_shouldNeverVeExecutedBecauseOfFailedAssumption() {
        Assertions.assertThat(false).isTrue();
    }

    @Test( dependsOnMethods = {"C_failedAssumption", "B_shouldNeverVeExecutedBecauseOfFailedAssumption"})
    public void E_shouldNeverVeExecutedBecauseOfFailedAssumption() {
        Assertions.assertThat(false).isTrue();
    }

    @Test
    public void D() {

    }

    @Test( priority = -6)
    public void Z_shouldRunFirst() {
    }
}

[plorenz]

@dasAnderl Be warned that the TestNG ordering implementation has its own problems. It builds a graph of dependencies from method to method (not just within a class, but across ALL of your tests). So all tests of a given priority will depend all tests of a lower priority across your whole project. The implementation in 6.14.3 is so inefficient that we couldn't upgrade to it. We have ~10k tests and they didn't finish running after a day and a half, with all the time spent in dependency calculation and graph updating. It looks like they've addressed some of this in 7.0 SNAPSHOT, but 7.0 is not yet. We've been investigating moving our testng to junit for this reason.

[sbrannen]

@dasAnderl, please note that support for "dependencies" between test methods will be addressed in #48 (support for "Scenario Tests").

[sbrannen]

@plorenz, thanks for sharing your experience with TestNG. I was aware of the fact that TestNG builds up a dependency graph across the test suite, but I was not aware of such performance issues. So that's good to know about. In any case, in #48 we do not plan to build up a dependency graph across anything more than a single test class for JUnit Jupiter.

[smoyer64]

@sbrannen I'm assuming from previous discussions that when you say "a single test class" it will include super-classes and interfaces. For my use cases, this is sufficient and suites take care of the rest. What are your thoughts with @Nested?

[sbrannen]

Yes, by "single test class", I mean all methods _visible_ in a given concrete test class.

[sbrannen]

What are your thoughts with @Nested?

Methods are not inherited within a @Nested class structure, simply because it is in fact not a conventional "class hierarchy". So the rules for ordering would be identical, applying to testable methods visible within a single concrete @Nested class.

[aalmiray]

@sbrannen just read a potential fix has been punted to 5.4. I wish it could be earlier as I need this feature for functional testing. Is there anything we could do during a Hackergarten meeting?

[sbrannen]

Hi @aalmiray,

Since we just released 5.3 RC1 last night, we won't be introducing any new features in 5.3 between now and 5.3 GA.

So, unfortunately, ordering support will have to come in 5.4.

Though... we could potentially try to get that out in an early milestone not too long after 5.3 GA.

Is there anything we could do during a Hackergarten meeting?

Potentially. 😉

[aalmiray]

@sbrannen Understood. Tonight is HG Bern, HG Basel is set for Aug 28th, unfortunately can't make it to HG Luzern on Sep 6th 🙁

[sbrannen]

@bechte, I see you "self-assigned this on Sep 12, 2017", but it doesn't appear you've worked on it (feature branch somewhere?).

So I'm unassigning you for the time being. 😉

[sbrannen]

_in progress_

[sormuras]

_5.4 gonna be feature-rich_

[sbrannen]

In case anybody is interested....

The first passing test using the new @Order annotation can be seen in commit https://github.com/junit-team/junit5/commit/415c8144d46af0ed9e598b38ff197d0dda984d53. 😎

[RupprechJo]

Has every method to be annotated with @Order or can I define an ordering globally for that file like @FixMethodOrder was in JUnit4?

[sbrannen]

@Order is just one variation of "ordering" that we will support out of the box.

The current work is just a "spike" with a hard-coded, single implementation.

The end result will be a new SPI that allows for any type of ordering/sorting.

For something like @FixMethodOrder from JUnit 4, that will likely end up looking something like @MethodOrdering(ALPHANUMERIC).

[RupprechJo]

Great news!

[sbrannen]

The proposed @MethodOrdering annotation and MethodOrderer SPI can now be seen in commit https://github.com/junit-team/junit5/commit/af447c92f3bb55c87c0be0e34138a4038de6ffea.

[sbrannen]

Alphanumeric ordering is also now in place... https://github.com/junit-team/junit5/commit/3eafd28ec0f7f443c52f880f3d2e80d85ae7b4e0

[alpar-t]

Great work, thank you ! Are we also getting random order ?

[sbrannen]

Yes, I'll implement "random order" as well. 😉

[thekalinga]

@sbrannen I am noob in this area, but have a question regarding specifying order as a dependency

i.e executing based on dependency DAG

i.e if methodA says depends on methodB, can this scenario also be supported?

@Test
@DependsOn("methodB")
void methodA() {}

@Test
void methodB() {}

The runtime should execute methodB first before methodA

[sbrannen]

i.e if methodA says depends on methodB, can this scenario also be supported?

Yes, one can implement the new SPI to support use cases like that.

Your custom MethodOrderer can _sort_ the list of method descriptors any you want it to.

The methods will then be executed exactly in that order.

[sbrannen]

Random ordering is now in place... https://github.com/junit-team/junit5/commit/429e6dfecb04a2b93841691864a5c0cdc637c8a2

Simple as 🥧!

[checketts]

How does the ordering affect nested classes? (Sorry if it was addressed already, I just didn't see it in the tests).

[tcfurrer]

Also what about execution order of top-level classes? This issue isn't providing anything new for that, right? Is anything planned? Would it have consistent/similar syntax? For example could there be @DependsOn support at the class level later?

By the way, great work here! I can't wait to install Junit 5.4. We really appreciate all the good work you guys are doing.

[alpar-t]

@sbrannen any chance we could make the seed of the random order configurable with some system property ? This is important to make the failures reproducible. https://github.com/randomizedtesting/randomizedtesting implements this.

[sbrannen]

How does the ordering affect nested classes? (Sorry if it was addressed already, I just didn't see it in the tests).

Method ordering will be supported in @Nested test classes as well.

I'll add some tests for that.

[sbrannen]

Also what about execution order of top-level classes? This issue isn't providing anything new for that, right? Is anything planned? Would it have consistent/similar syntax? For example could there be @DependsOn support at the class level later?

No, we don't have any plans to support ordering of top-level test classes.

By the way, great work here! I can't wait to install Junit 5.4. We really appreciate all the good work you guys are doing.

Thanks!

[sbrannen]

@sbrannen any chance we could make the seed of the random order configurable with some system property ? This is important to make the failures reproducible. https://github.com/randomizedtesting/randomizedtesting implements this.

Yes, I'll add support for that.

[sbrannen]

Update : _Deliverables_ have been expanded and updated.

[sbrannen]

How does the ordering affect nested classes? (Sorry if it was addressed already, I just didn't see it in the tests).

Verified support for @Nested test classes in commit https://github.com/junit-team/junit5/commit/2517ca6fbf1540ac2c6afb04ad2d6552bdde1b48.

[alpar-t]

Thanks for your updates. Are there any technical difficulties in ordering the test classes or is it just not something in scope right now ? Should I open a separate issue for it ? We are looking to pseudo randomize test class order as well, in addition to the method order.

[sbrannen]

Thanks for your updates.

Sure thing!

Are there any technical difficulties in ordering the test classes or is it just not something in scope right now ? Should I open a separate issue for it ? We are looking to pseudo randomize test class order as well, in addition to the method order.

To be honest, that's something we never intended to support, and we still don't plan to support it.

However, if you feel strongly about it, feel free to open a separate issue. If there is considerable interest in the community, we will potentially consider it.

[kcooney]

If the seed of the random order is configurable with a system property, we should consider updating JUnit 4.13's Ordering class to support the same system property.

[marcphilipp]

@kcooney Good point! Could you please create an issue for that over at the junit4 repo?

[sbrannen]

@marcphilipp and @kcooney,

For JUnit Jupiter we consistently use the junit.jupiter prefix for all _Configuration Parameters_.

Thus, my plan is to call the _Configuration Parameter_ (which can optionally be set via a JVM system property) something similar to junit.jupiter.method.order.random.seed.

So if we _really_ want to use the same property name for JUnit 4.13 and Jupiter, I suppose we'd have to name it something like junit.method.order.random.seed (omitting the jupiter part).

Thoughts?

[smoyer64]

@sbrannen Not much of JUnit 5 is backwards-compatible anyway - naming just this one configuration parameter to match the name in JUnit 4 seems like it would instill a (small) false sense of security. I'd far rather see the naming be consistent _within_ JUnit 5. (just my two cents).

[marcphilipp]

Agreed, I think it's far more important that both support setting it via a system property than it being the same property.

[sbrannen]

@smoyer64 and @marcphilipp, thanks for chiming in.

I agree that we should remain consistent in Jupiter regarding naming conventions for configuration parameters.

I will therefore tentatively go with junit.jupiter.execution.order.random.seed, leaving out method since the same seed may later be used for test class ordering or similar use cases.

[sbrannen]

Update: extensions for method ordering will be able to specify the default ExecutionMode for concurrent test execution. See https://github.com/junit-team/junit5/commit/9d692ac80d0f7a6093c5a3789c77d291667c5ae4

[sbrannen]

Update: _validation_ has been added in commit c07b521e0db566fa8e88372724258986f3f06c93.

[sbrannen]

Update: support for parallel test execution has been tested in commit 8d1e74997f34775e7f7fee5b2cfca9f37159bbb0.

[sbrannen]

Update: _Configuration Parameter_ support for setting the seed used in random ordering has been introduced in commit 7939b4cd634c2787b61384fb496f6bf63057ec04.

[sbrannen]

Update: Javadoc pushed in commit b1a4726dfd98d4be1fb323cbbbcd47da9e4aef83.

[sbrannen]

Drum roll........

Support for ordering the sequence of test methods has been merged into master in commit 5300e650c2662586ecb9f409223ec6bed9d5eae6. 😎

[aalmiray]

Yay! Thanks again

[sbrannen]

¡de nada! / You're welcome! 😉

[anatoliy-balakirev]

Hi. @sbrannen thanks for adding this to Junit!
We are currently using Spring Cloud Contract in conjunction with Junit 5 and would like to start using Wiremock's scenarios there. However, it looks like SCC team is waiting for this functionality to move out of EXPERIMENTAL status, so I wonder if you have an idea on when it might happen? Is there some way to track this? What is the process in general?

[sbrannen]

Hi. @sbrannen thanks for adding this to Junit!

You're very welcome.

We are currently using Spring Cloud Contract in conjunction with Junit 5 and would like to start using Wiremock's scenarios there. However, it looks like SCC team is waiting for this functionality to move out of EXPERIMENTAL status,

Do you have a link to an issue for that?

so I wonder if you have an idea on when it might happen? Is there some way to track this? What is the process in general?

In general, we flag new APIs as experimental for their initial release and then wait a few minor releases to see if the community raises any issues with the API or implementation.

This particular feature was released in JUnit Jupiter 5.4, and 5.5 is currently being worked on. There have been a few improvements to the implementation since 5.4.0 but no changes to the API. So if that continues to be the case, we might switch the API status from experimental to stable in 5.6 (even potentially for 5.5 GA -- to be discussed).

[anatoliy-balakirev]

Ok, thanks for info!

Do you have a link to an issue for that?

Here is the link: https://github.com/spring-cloud/spring-cloud-contract/issues/887

[sbrannen]

Thanks for the link.

[Fernal73]

How does this work with @Execution(CONCURRENT)? It doesn't.

[sbrannen]

@Fernal73, as stated in the Javadoc for MethodOrderer, the default execution mode for ordered methods is SAME_THREAD, but it...

Can be overridden via an explicit @Execution declaration on the test class or in concrete implementations of the MethodOrderer API.

If you are experiencing behavior contrary to that, please open a new ticket providing further details.

Thanks

[Fernal73]

I've experienced out of order invocations when setting the Execution mode as concurrent.
Is this expected behaviour given that the feature is experimental and not fully mature?

How do you decide that the ordering does not interfere with concurrency? I'd expect ordered methods to be executed in a single thread wrapped in a synchronizer that orders the methods as per their values so that shared state in the test class is not adversely impacted causing failed tests.

If all my methods are ordered, shouldn't the methods be executed in a single thread irrespective of ExecutionMode?

I have since updated my tests to not have a shared state; @Order no longer matters and has been removed.

[sbrannen]

I've experienced out of order invocations when setting the Execution mode as concurrent.

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Defaults to SAME_THREAD, since ordered methods are typically sorted in a fashion that would conflict with concurrent execution.

---

How do you decide that the ordering does not interfere with concurrency?

The MethodOrderer implementations provided by JUnit Jupiter do not, since none of them overrides getDefaultExecutionMode().

I'd expect ordered methods to be wrapped in a synchronizer should the execution mode be set to concurrent.

This issue was closed more than 1.5 years ago.

If you wish to make suggestions for improvements or changes to the current behavior or feature set, please open a new issue.

[Fernal73]

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Is this to be expected even for @Order annotated methods? Other test methods, of course.

This ought to be documented better. If @Order methods go out of sync with ExecutionMode.CONCURRENT, then they ought to be permitted only with SINGLE_THREAD mode.

See previous response for edits.

[Fernal73]

This issue was closed more than 1.5 years ago.

If you wish to make suggestions for improvements or changes to the current behavior or feature set, please open a new issue.

I'm inclined to go with the workaround that @Order annotated methods not be mixed with methods to be executed concurrently viz, have a separate Test class for each type.

[sbrannen]

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Is this to be expected even for @order annotated methods?

Yes

This ought to be documented better. If @order methods go out of sync with ExecutionMode.CONCURRENT, then they ought to be permitted only with SINGLE_THREAD mode.

Again, this closed issue is not the place to make recommendations.

If you feel strongly about those points, please open one or more new issues to address them.