Junit5: Opt-in support for parallel execution

[marcphilipp]

Related to #90.

[ptahchiev]

+1 for this one. TestNG has had this feature for a long time and the maven surefire plugin supports parallel tests out of the box:

http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

[radai-rosenblatt]

+1. would definitely help in shortening build times

[leonard84]

Will this be part of the platform, or the engine, or both?

While thinking about better support for parallel execution in Spock, I identified some problems that should be handled by the testing framework.

• Parallel (integration) testing in conjunction with global state (system properties/db/...) can leak into other tests
  
  
  
  • e.g., Spock has the @RestoreSystemProperties extension, which will save the SystemProperties before the execution and restore it after the test. This works fine for single threaded execution, but in concurrent test execution this will lead to race conditions.
  
  
  • The bad solution would be to disable concurrent execution
  
  

My idea is that each test (class/method) could declare its use of global resources, e.g. @UseResource(value="SystemProperties", mode=ReadWrite). The engine would keep a list of ReentrantReadWriteLock for each of the resources and acquire/release those locks accordingly. Care would have to be taken to avoid deadlocks, by ordering the lock acquisition accordingly.

This way the framework could offer the maximum concurrency while still offering providing tools to isolate problematic tests.

There would be a list of predefined resources, such as Global (all tests would implicitly acquire a ReadLock for this) or SystemProperties, but users can define their own to match their application.

Thoughts? Of course all names are open for change.

[marcphilipp]

I think an engine needs to decide on which level tests can be executed in parallel. However, there are quite a few similarities between different testing frameworks where the platform could provide support, e.g. via a base implementation that multiple test engines can use.

I really like your reentrant lock idea! Is there already a TestEngine implementation for Spock? If so, do you use the HierarchicalTestEngine abstraction? I wrote a ForkJoinPool based prototype for it some time ago.

[leonard84]

@marcphilipp no Spock doesn't have it's own TestEngine yet. As better support for parallelism would be a major part in it, I'd wanted to check in on this first before running in the wrong direction/doing redundant work. Switching to a own TestEngine will be done with Spock 2.0

[wlsc]

Any news on that topic?

[marcphilipp]

Not yet, I‘m afraid.

[christophsturm]

@marcphilipp can you share your ForkJoinPool based prototype?

[marcphilipp]

Sure! I've actually re-implemented it based on the current master now:
https://github.com/junit-team/junit5/compare/experiments%2Fparallel-execution

[christophsturm]

@marcphilipp whats the best way to try this out?
btw, I was thinking of collecting the duration of each test and then sort by that time descending. that would ensure that slow test run early and on different cores, and you don't have a slow test running at the end when all other cores are idle.

[sormuras]

Use JitPack with the experimental branch. Will compile the dependency coordinates when I get home. Which build tool do you prefer?

[christophsturm]

ah cool i did not know that the experimental branch is published. I'm using gradle but the parallel test running is most interesting to me from inside IDEA.

[mannodermaus]

Great to see a concrete milestone attached to this !

Here are the Jitpack coordinates of the experiments-parallel-execution branch.

[sormuras]

Slightly different coordinates here: https://jitpack.io/#junit-team/junit5/experiments~parallel-execution-SNAPSHOT

Both should work. Caveat: some results collecting listeners are not thread safe, yet.

[christophsturm]

this is my first time using jitpack but it does not seem to work
```> Could not resolve all files for configuration ':compileClasspath'.

Could not find com.github.junit-team:junit5:experiments%7Eparallel-execution-SNAPSHOT.
Searched in the following locations:
https://repo1.maven.org/maven2/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/maven-metadata.xml
https://repo1.maven.org/maven2/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/junit5-experiments%7Eparallel-execution-SNAPSHOT.pom
https://repo1.maven.org/maven2/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/junit5-experiments%7Eparallel-execution-SNAPSHOT.jar
https://jitpack.io/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/maven-metadata.xml
https://jitpack.io/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/junit5-experiments%7Eparallel-execution-SNAPSHOT.pom
https://jitpack.io/com/github/junit-team/junit5/experiments%7Eparallel-execution-SNAPSHOT/junit5-experiments%7Eparallel-execution-SNAPSHOT.jar
```

[christophsturm]

i got it running, the problem was that the ~ character was always escaped to %7e. manually creating the url worked for me

[sormuras]

Nice. Note to myself: avoid / in branch names.

How did the fork-join-pool work out for you?

[christophsturm]

it seems to work. I will report back after some more tests!

[christophsturm]

I think there's a problem with @BeforeEach. if you take this test:

public class TestTest {
    @BeforeEach
    public void be() {
        System.out.printf("before");
    }

    @Test
    public void test1() {
        System.out.printf("t1");
    }
    @Test
    public void test2() {
        System.out.printf("t2");
    }
}

it outputs beforebeforet2t1

[christophsturm]

ok never mind, I notice that this output is totally valid.
Still its strange because one of my tests that seems to depend on side effects, runs fine multi threaded in junit4 but it fails with junit5. anyway, not a bug in junit5.

[christophsturm]

one question, why are you using a ForkJoinPool? should't a normal fixedthreadexecutorservice act exactly the same in this usecase, but with less overhead?

[marcphilipp]

We rely on its work-stealing properties: When a container executes it needs to be able to fork execution for each child and resume once they‘re done, recursively. Moreover, we‘ll probably add resource locks which ForkJoinPool should support nicely.

[christophsturm]

is the HierarchicalTestExecutorService pluggable? i would like to experiment with some performance optimisations and not have to rebuild junit. also somehow unrelated, but is it possible to configure a callback that runs before and after my whole suite? In my junit4 suite i use a CPUTimer class that measures cpu load. (trying to get my test suite to use all cores on my threadripper system)

[christophsturm]

i have now ported my whole test suite to junit5 and it's running well with the parallel support. I have only one small issue left, but thats jitpack related, the gradle plugin is adding junit dependencies with the original groupId and not the jitpack groupId, which is to be expected, but it results in using the snapshot version instead of the experimental version:

+--- org.junit.platform:junit-platform-launcher:1.1.0-SNAPSHOT FAILED
\--- org.junit.platform:junit-platform-console:1.1.0-SNAPSHOT FAILED

this will just go away when the branch is merged to master and i can use a snapshot version. is this going to happen soon?

[marcphilipp]

Thanks for testing this early prototype! It will still take a while for us to discuss and polish this. I'll meet with @leonard84 early January to discuss additional requirements.

[rosolko]

@marcphilipp Hi.
Can you please provide info about thread count limitation.
I'm using gradle.
In the past I can limit count by maxParallelForks property for test task.
Do we have similar ability in current case(experimental implementation)?

[marcphilipp]

@rosolko Not yet. Currently it creates a default fork-join pool which will create a thread pool with a size that depends on the number of cores on your system. Adding an option to set the number of threads should be very simple, though.

[marcphilipp]

I've just merged master into the feature branch:
https://github.com/junit-team/junit5/compare/experiments%2Fparallel-execution

[christophsturm]

thanks. why don't you merge it to master? it's not enabled per default anyway.

[leonard84]

@christophsturm because it is very experimental and not nearly finished or polished at all

[k3nda]

Hi, what is the progress? When you are going to release parallel execution of tests?

[rosolko]

@k3nda From 4.6 version gradle support parallel execution via forks.

[sormuras]

@k3nda There's an active feature branch which might be integrated as an experimental feature in 5.2.0-SNAPSHOT ... soon.
If you want to test-drive it now, compile and install it locally, or use jitpack. The coordinates are something like:

    String group = "com.github.junit-team.junit5";
    String version = "experiments~parallel-execution-SNAPSHOT";

    ...resolve(group, "junit-jupiter-api", version);
    ...resolve(group, "junit-jupiter-engine", version);
    ...resolve(group, "junit-platform-console", version);
    ...resolve(group, "junit-platform-commons", version);
    ...resolve(group, "junit-platform-engine", version);
    ...resolve(group, "junit-platform-launcher", version);

https://jitpack.io/#junit-team/junit5/experiments~parallel-execution-SNAPSHOT mind the ~, not %7E!

[sormuras]

Added a proof-of-concept branch at https://github.com/junit-team/junit5-samples/compare/parallel-execution ... which will never be merged. But it demonstrates that the Jupiter-internal parallel execution mode works.

[k3nda]

@rosolko I am using maven
@sormuras ok, I will try asap. And is it possible execute it with _maven-surefire-plugin_?

[sormuras]

[...] And is it possible execute it with maven-surefire-plugin?

See the Maven instructions shown at jitpack URL above. If it works with Surefire? No idea. Keen on trying it out?

[sormuras]

@marcphilipp The https://github.com/junit-team/junit5/tree/experiments/parallel-execution branch is still based on 5.1-M2 ... it lacks the final org.junit.jupiter.api.condition package and other fixes. Will @leonard84 or you rebase it to master in the near future?

[marcphilipp]

I'll meet with @leonard84 this weekend. I'll try to merge master into the feature branch before.

[marcphilipp]

I've just updated the feature branch. Feel free to try it out using Jitpack.

[dlanaghen]

Parallel testing is a big deal to us. I have experimented with your version and my first simple tests look real promising. Is there a ball park for when this might get merged into Junit5 and when an official release timeframe might be? If this is going to be real, I might forge ahead and begin really exercising it.

[marcphilipp]

@dlanaghen Thanks for experimenting with it and letting us know!

We plan to include it as an experimental feature in 5.3 M1 which is currently scheduled for June 24. If no regressions or major issues are found, you can expect a RC and the GA release within two weeks from that date.

[dlanaghen]

@marcphilipp this is great news. With better parameterized testing features and your new multi-threaded test runner, you will have filled a couple of big holes in JUnit. Thanks for your work.

[koretzki]

@marcphilipp, great to hear, this (no parallelism) was a road blocker for us to adopt fully JUnit 5.

[markthegrea]

Where is the documentation on how to use this?

[sormuras]

@markthegrea It is still on the _snapshot_ version of the User-Guide, because it is integrated in version 5.3-RC1 of the Jupiter Engine: https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution

When 5.3 is released, the _current_ User-Guide will show it, too.