Junit5: Publish a module that combines the Jupiter API, Params, and Engine in one artifact

[philwebb]

Overview

It's fairly common to require junit-jupiter-api, junit-jupiter-params and junit-jupiter-engine when writing tests. Currently this means that any build file need three distinct imports, and possible a version property declaration to ensure version compatibility.

The currently Maven and Gradle samples themselves show such a setup.

It would really help the getting started experience if a single junit-jupiter dependencies was available that pulled in the others transitively. The would change a Maven build from this:

<project>
    ...
    <properties>
        <junit.jupiter.version>5.3.1</junit.jupiter.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit.jupiter.version}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    ...
</project>

to this

<project>
    ...
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.3.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
    ...
</project>

Deliverables

• [x] A new POM file.

[sormuras]

junit-jupiter-engine is a test runtime dependency and should not be in <scope>test</scope> at all. The Gradle build you linked uses testCompile twice, and testRuntime once. It's a short-coming of Maven not to distinguish between those two scopes and also a lack of "magic" in Surefire's JUnit Platform Provider that it doesn't auto-resolve the matching engine for the used API. This junit-platform-maven-plugin does the trick.

junit-jupiter-engine has a transitive dependency on junit-jupiter-api. Thus, if you only want to save some lines of XML, leave out junit-jupiter-api here -- unnecessarily making all engine types visible to the test authors. And hiding the API artifact from beginners.

junit-jupiter-params is an optional and still experimental API. Let the test author be in control what is needed to write tests.

A POM file that merges two artifacts, API and Params, seems like an overkill to me.

Analog to JUnit 4's junit:junit artifact, the minimal POM file of project that has no explicit _main_ dependencies and uses JUnit Jupiter as its test framework, only contains:

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.3.1</version>
    <scope>test</scope>
  </dependency>
</dependencies>

If you want to use @ParameterizedTest in addition to the basic Jupiter API, the minimal POM file reads:

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.3.1</version>
    <scope>test</scope>
  </dependency>
</dependencies>

junit-jupiter-api is pulled in transitively.

Both minimal POM file snippets require the underlying build tool auto-resolves the junit-jupiter-engine.

[snicoll]

A POM file that merges two artifacts, API and Params, seems like an overkill to me.

I can certainly disagree with that. I'd prefer we'd not discuss build system shortcomings and whatever was done here to make it easier in one vs. the other. IMO, all that matters is the users community at large and a somewhat consistency with JUnit 4.

It feels to me the modularity and architecture of JUnit Jupiter puts the onus on the user for the most basic task of configuring a project. Yes, one part is the API and only required to write tests while the other is a mandatory component that is only required at runtime to run your tests. Does a regular user care?

If you do, then the work that was done as part of JUnit Jupiter is great because it gives you the flexibility to define the right semantic in your build. If you don't, or if you're happy with the status quo that JUnit 4 provides, why should we insist here on telling users they have to do it this way?

I'd welcome some adaptations to make the transition to Junit Jupiter as smooth as possible, even if that means making some compromise in what you feel is the right way to configure a project.

[sormuras]

Does a regular use care _[about the split of API and runtime]_?

I guess not. That's why a regular user should only depend on junit-jupiter-api. A single dependency delivering a bunch of features. If a regular+1 user needs parameterized test, junit-jupiter-params is only a single dependency away -- which even replaces the former one to junit-jupiter-api.

This is not complex.

[...] if you're happy with the status quo that JUnit 4 provides [...]

If you're happy with X, stay with X. X includes JUnit 3, 4, TestNG, and even psvm test programs or any other framework out there that helps you writing tests. I like all of them.

[...] why should we insist here on telling users they have to do it this way?

We shouldn't do this at all. We may provide options. Present each option in its minimal, understandable, easy-to-learn (maybe hard-to-master) way.

JUnit 5 modularity is its unique feature. Actually, there is no JUnit 5 in the sense there was a single JUnit 4 artifact -- why pretend it exists? IMO, having too much transition magic/helpers only obscures reality and hinders users to really learn about the new choice. With junit-platform-runner, junit-jupiter-migrationsupport and junit-vintage-engine there are already enough transition helpers out there.

The best helper is: https://junit.org/junit5/docs/current/user-guide ;-)

[smoyer64]

I'm not sure it's even fair to compare the JUnit 5 Parameterized tests to those built into JUnit 4 - the functionality really isn't in the same league. In 99% of my past projects, I ended up importing both JUnit 4 _and_ JUnitParams. I do feel the pain of transitioning to JUnit 5 when trying to explain how to configure Surefire and which engine it will pick. This, IMHO, is improving and part of that pain has been staying on the bleeding edge. When the tool-chains all include one of twenty compatible (with my test class) engine version, I think this should go away.

[philwebb]

junit-jupiter-engine is a test runtime dependency and should not be in test at all. The Gradle build you linked uses testCompile twice, and testRuntime once.

I did notice that, but I wondered if the distinction made much difference to most users. Is the additional dependency worth it?

Surefire's JUnit Platform Provider that it doesn't auto-resolve the matching engine for the used API. This junit-platform-maven-plugin does the trick.

Is the junit-platform-maven-plugin the recommended way to configure Maven? The reference documentation doesn't mention it, so I'd not seen it before. Or is this something that's ultimately going to be part of the maven-surefire-plugin?

I'm asking because at some point we're planning to upgrade Spring Boot's testing support to use JUnit 5 and I'd like to make sure we use the recommended configuration.

junit-jupiter-engine has a transitive dependency on junit-jupiter-api. Thus, if you only want to save some lines of XML, leave out junit-jupiter-api here

Thanks, that makes things clearer. So depending on junit-jupiter-engine makes the build file smaller, but the user might accidentally import a class that they shouldn't. Using the junit-jupiter-api is technically better, but somewhat academic for Maven and Eclipse users since those project don't offer the classpath isolation needed to prevent accidental imports anyway. Using the junit-platform-maven-plugin means no direct junit-jupiter-engine dependency is needed at all, and it's figured out when the tests run?

junit-jupiter-params is an optional and still experimental API. Let the test author be in control what is needed to write tests.

Understood, perhaps at some point it will get promoted into the core module. Until then, two dependencies makes sense.

BTW, I think some additional documentation or links early in the reference guide might really help with the getting started experience. A couple of "quick start" examples for Maven and Gradle in the installation section would probably help 90% of your users get going quickly. I didn't find the "Running Tests" section until a bit later so I was mainly going on the samples as my guide for how to configure the build system.

[philwebb]

In hindsight it feels like a single dependency isn't worth it. I misunderstood the role of the engine jar and it's clear that there's a desire for junit-jupiter-params to remain separate for now. If anyone from the JUnit team wants to chime in on https://github.com/spring-projects/spring-boot/issues/14736 with recommendations for the Spring Boot configuration, please do so.

[smoyer64]

@philwebb There are also now JUnit 5 pages for both the Surefire and Failsafe plugins (e.g. https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html).

[sormuras]

@philwebb

Is the junit-platform-maven-plugin the recommended way to configure Maven?

At the moment it is just an option, when running on Java 11 and above. It is my playground to investigate "Testing In The Modular World", especially when it comes to _white box_ testing on the module-path.

With those specific requirements, it won't be the recommended way to configure Maven. Perhaps, some day in near future, when 90% of all Java projects are on Java 11+. ;-)

Or is this something that's ultimately going to be part of the maven-surefire-plugin?

Although I'm actively helping out to keep Surefire feature-wise up to date, I'm not sure if all features of the junit-platform-maven-plugin are portable.

Using the junit-platform-maven-plugin means no direct junit-jupiter-engine dependency is needed at all, and it's figured out when the tests run?

True. This feature is planned to be included in maven-surefire-plugin 2.22.2...

BTW, I think some additional documentation or links early in the reference guide might really help [...]

Yeah. A link to the section below and a link to the junit5-samples would improve the situation.

[sbrannen]

BTW, I think some additional documentation or links early in the reference guide might really help with the getting started experience. A couple of "quick start" examples for Maven and Gradle in the installation section would probably help 90% of your users get going quickly.

I think that's a good idea, @philwebb!

We already have links to the junit5-samples repository, but we could certainly rework the initial content of the User Guide.

I didn't find the "Running Tests" section until a bit later so I was mainly going on the samples as my guide for how to configure the build system.

When you say "samples" in this context, do you mean the examples inlined within the User Guide or the aforementioned junit5-samples repository?

[sbrannen]

@philwebb, I've gone ahead and reworked some of the User Guide content in light of your suggestion.

Is commit c80f2e9a867c4c964c647bb02930d7a0007d6ded along the lines of what you had in mind?

FYI: you'll be able to see it in all of its glory here in about 15 minutes (or whenever the CI server finally publishes the snapshot).

[sormuras]

Team Decision: Publish an experimental org.junit.jupiter:junit-jupiter:${version} pom-only artifact starting with 5.4.0-M1.

[snicoll]

Thanks for the feedback. Can it be a jar please (even if it's empty?). Adding an artifact of type pom means that we have to specify the type and that's a bit unusual.

(FTR, Spring Boot starters are addressing a similar feature and they are jars for that reason).

[sormuras]

Can it be a jar please (even if it's empty?). Adding an artifact of type pom means that we have to specify the type and that's a bit unusual.

Mh, Gradle seems to "understand" that and resolves transitive dependencies. Maven needs that extra line... so yes, considering to publish almost empty jar files. They might contain compiled module descriptors (module-info.class) in a not so distant future, actually.

[sormuras]

The "implementation" part is done in #1691 -- updating the initial documentation and release notes is next.

[Tibor17]

@philwebb
@sormuras
@snicoll
This could be a typical BOM as org/junit/junit-jupiter-bom/<version>/pom.xml.
So the junit-jupiter-bom normally has dependencies section and dependencyManagement with packaging=pom. Whenever you put a dependency on it in user's POM, all three are available as one. If you put it to the user's dependencyManagement, the user can pickup some of them. It is one consistent BOM and all listed artifacts are related and obvious which should be logically combined in user's project.

[Tibor17]

Example in user's POM:

<dependencies>
  <dependency>
    <groupId>org.junit</groupId>
    <artifactId>bom-junit-jupiter</artifactId>
    <version>5.4.0</version>
    <type>pom</type>
  </dependency>
</dependencies>

[sormuras]

We do already offer a junit-bom here: https://search.maven.org/search?q=a:junit-bom

We decided to offer an "empty jar" because of @snicoll request:

Thanks for the feedback. Can it be a jar please (even if it's empty?). Adding an artifact of type pom means that we have to specify the type and that's a bit unusual.
(FTR, Spring Boot starters are addressing a similar feature and they are jars for that reason).

[Tibor17]

@sormuras
We are talking about two different things juit-bom is not junit-jupiter-bom of course.
And regarding jar file, it's your decision, but it is not Maven standard. If jar file then classes are in; otherwise packaging=pom since it is dependencies aggregator. You are mixing two different approaches.

[jbduncan]

We are talking about two different things juit-bom is not junit-jupiter-bom of course.

Hmm... junit-bom seems to apply to JUnit 5 rather than earlier JUnit versions? At least, that's what reading https://search.maven.org/artifact/org.junit/junit-bom/5.3.2/pom seems to imply to me. Or have I misunderstood something?

And regarding jar file, it's your decision, but it is not Maven standard. If jar file then classes are in; otherwise packaging=pom since it is dependencies aggregator. You are mixing two different approaches.

Well, I suppose one argument in favour of having a JAR as well as a BOM is that it allows more Gradle users to use this aggregation feature (since importing BOMs in Gradle isn't a built-in feature in earlier versions of Gradle, IIRC), and a second argument would be that it gets rid of the verbosity of importing all of junit-jupiter-api, junit-jupiter-engine and junit-jupiter-params manually.

[snicoll]

If jar file then classes are in; otherwise packaging=pom since it is dependencies aggregator. You are mixing two different approaches.

I am not sure you can call that a Maven standard, right? I understand what you mean but the jar approach is a better fit for this use case (IMO) due to the verbosity of Maven (having to add the type) and some plugins being confused having a pom packaging type as a dependency.

[eggilbert]

Hello -
I tried using this new junit-jupiter aggregate artifact, and I ran into some curious behavior that I'm not sure is intentional.

If I have only this dependency:

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

Then my dependency tree looks like this, which seems sensible:

[INFO] \- org.junit.jupiter:junit-jupiter:jar:5.4.0-M1:test
[INFO]    +- org.apiguardian:apiguardian-api:jar:1.0.0:test
[INFO]    +- org.junit.jupiter:junit-jupiter-api:jar:5.4.0-M1:test
[INFO]    |  +- org.opentest4j:opentest4j:jar:1.1.1:test
[INFO]    |  \- org.junit.platform:junit-platform-commons:jar:1.4.0-M1:test
[INFO]    +- org.junit.jupiter:junit-jupiter-params:jar:5.4.0-M1:test
[INFO]    \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0-M1:test
[INFO]       \- org.junit.platform:junit-platform-engine:jar:1.4.0-M1:test

But if I use the junit-bom to declare the version of all related artifacts:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.junit</groupId>
            <artifactId>junit-bom</artifactId>
            <version>5.4.0-M1</version>
            <scope>import</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Then my dependency tree looks like the following, where transitive dependencies are suddenly compile scope instead of the desired test scope:

[INFO] \- org.junit.jupiter:junit-jupiter:jar:5.4.0-M1:test
[INFO]    +- org.apiguardian:apiguardian-api:jar:1.0.0:compile
[INFO]    +- org.junit.jupiter:junit-jupiter-api:jar:5.4.0-M1:compile
[INFO]    |  +- org.opentest4j:opentest4j:jar:1.1.1:compile
[INFO]    |  \- org.junit.platform:junit-platform-commons:jar:1.4.0-M1:compile
[INFO]    +- org.junit.jupiter:junit-jupiter-params:jar:5.4.0-M1:compile
[INFO]    \- org.junit.jupiter:junit-jupiter-engine:jar:5.4.0-M1:compile
[INFO]       \- org.junit.platform:junit-platform-engine:jar:1.4.0-M1:compile

Is it expected that the scope of transitive dependencies is different when the junit-bom is involved? We were looking to do what was described in the description of the issue here, where the 3 test scoped dependencies could be included with one declaration _(and remain test scoped)_.

[eggilbert]

Upon further examination, it appears that this behavior is happening pulling in anything using junit-bom:5.4.0-M1, not just the new junit-jupiter artifact.

It looks like everything in the 5.4.0-M1 bom has an explicit <scope>compile</scope> on it, where as there was no specified scope in the 5.3.2 bom. Maybe that's the culprit?

[sormuras]

@eggilbert Do you mind opening a new issue for the side-effect between BOM and the junit-jupiter aggregator artifact?

[sbrannen]

It looks like everything in the 5.4.0-M1 bom has an explicit <scope>compile</scope> on it, where as there was no specified scope in the 5.3.2 bom. Maybe that's the culprit?

Yes, I believe that is in fact the culprit. The BOM should not define any _scope_ for managed dependencies.

@eggilbert, would you mind opening a new issue for that?

@sormuras, my requested issue would supersede your request -- right?

[sbrannen]

@eggilbert, would you mind opening a new issue for that?

On second thought, I'll go ahead and create raise a bug report against 5.4 M2 so that we don't forget about it.

[sbrannen]

FYI: I opened #1712.

[eggilbert]

@sbrannen thanks!