Spring-boot: Investigate JUnit 5

[philwebb]

JUnit 5 looks like it might be a very awkward upgrade for both our users and Spring Boot itself. We'll need to come up with a plan going forward for how people should upgrade.

Of particular concern:

• You can't just run JUnit 4 tests with JUnit 5 😢
• There are new @Test annotations and expectedException has been dropped
• There is no @Rule annotation (this will break a lot of our tests and we'll need to offer alternatives to the rules we offer)

We might be able to offer JUnit 5 alongside JUnit 4 for a while (possibly a long while) so that people can use both.

I imagine probably going to frustrate a lot of users given that mostly their own test code has very little direct contact with the JUnit API. Mostly people use @Test, @Before/@After, perhaps a few @Rule annotations and some Assert calls.

Related issue https://github.com/junit-team/junit5/issues/169

[philwebb]

/cc @sbrannen

[marcphilipp]

Regarding the first bullet point: You can run JUnit 4 tests with the new JUnit Platform. You just have to add junit-vintage-engine to the classpath.

Please note that JUnit 5 is not final yet. We've merely shipped M1. As such, we are still working on providing migration patterns from JUnit 4 to JUnit 5. So let's keep in touch! :)

[philwebb]

Thanks @marcphilipp!

[sdeleuze]

JUnit 5 support would be very useful as part of the general Spring Boot + Kotlin improvements discussed in #5537.

Using JUnit 4 in Kotlin is almost ok except for one major pain point: @BeforeClass and @AfterClass have to be static methods, and Kotlin does have that. There is an ugly workaround using companion object but this is not something we should promote IMO. Even in Java this limitation (that TestNG does not have) can be annoying, even there is less impact than in Kotlin.

The good news is that JUnit 5 M4 is likely to support non-static @BeforeAll and @AfterAll methods (the related issue is https://github.com/junit-team/junit5/issues/419). If that happens, it would make JUnit 5 a very nice fit for testing Spring Boot + Kotlin applications.

cc @sbrannen

[jchesterpivotal]

Speaking only for myself, the @DisplayName and @Nested annotations introduced by Jupiter could allow more BDD-ish style testing with JUnit. I'd be in favour of some kind of support showing up in the 2.x series.

[driverpt]

@philwebb , i believe that it's possible to run JUnit4 Tests with JUnit5, from what i understand, you just need to add the junit-vintage-engine artifact

Link

I think that we should integrate JUnit5 as standard in spring-boot-starter-test and add a spring-boot-starter-test-junit4-compat artifact to support Legacy JUnit4 stuff. And for those who are more conservative, just have a spring-boot-starter-test-junit4 just with JUnit 4

[pvorb]

JUnit 5 has been released!

Would it make sense to work on support for JUnit 5 for Spring Boot 2?

[wilkinsona]

Thanks to Spring Framework's test framework, we don't really need to do anything in Boot to support JUnit 5. We already have a sample showing how to use the two together.

[snicoll]

And we have dependency management for it now (see #10240). I suspect we'll need to refine this a bit more though.

[pvorb]

I didn't know it's already usable. So much the better!

[sdeleuze]

Nice, I have been able to update MiXiT Kotlin + Boot 2.0 application to use Junit 5 and it works as expected, including the per class lifecycle mode: https://github.com/mixitconf/mixit/commit/05c68f4694c0d83aca0de0e4e610f4bfde7d04da.

[sbrannen]

@wilkinsona,

Thanks to Spring Framework's test framework, we don't really need to do anything in Boot to support JUnit 5.

Did you already address that issue (can't recall which one right now) regarding support for "test annotations" so that it includes test annotations from JUnit Jupiter?

[sbrannen]

Ahhh... found the issue: #6898 which is still _open_.

[sbrannen]

@philwebb & @wilkinsona,

FYI: _just for fun_... I migrated your OutputCapture rule for JUnit 4 to a JUnit Jupiter extension and _published_ it in my demo project.

• @CaptureSystemOutput
• SystemOutputTests

If you're interested in having as a part of Spring Boot, I could submit a PR. 😉

[sdeleuze]

@sbrannen while playing with JUnit 5 + Java 9 and trying to pass JVM arguments to Junit 5 in order to add some modules, I faced https://github.com/junit-team/junit5/issues/650 and found that various capabilities are waiting for https://github.com/gradle/gradle/issues/1037 to be fixed. As you know I am a big Junit 5 fan, but I am wondering if the builtin Gradle Junit 5 support is required before switching to it by default in Boot or not. Any thoughts?

[philwebb]

@sdeleuze My gut instinct is that we not should switch to it by default in 2.0.

(I accidentally said switch in 2.0 but I meant to say we should not switch) I've edited this comment.

[wilkinsona]

@philwebb Should we move this to an earlier milestone?

[sbrannen]

@sdeleuze

while playing with JUnit 5 + Java 9 and trying to pass JVM arguments to Junit 5 in order to add some modules, I faced junit-team/junit5#650 and found that various capabilities are waiting for gradle/gradle#1037 to be fixed.

That's correct. We've been _waiting_ for a long time now, and we _hope_ the Gradle team picks up the task (no pun intended) by the end of 2017.

As you know I am a big Junit 5 fan, but I am wondering if the builtin Gradle Junit 5 support is required before switching to it by default in Boot or not. Any thoughts?

I'd say the built-in support from the JUnit 5 team is "good enough" for most builds, but it is obviously far from feature complete. There are, however, custom workarounds for some missing features.

The status of the built-in Maven Surefire plugin is about the same. The Maven Surefire team has taken over the code, but they have yet to publish anything.

So, in the end, it's honestly a tough call: the build support _works_ for Maven and Gradle, but it's currently _suboptimal_.

[sbrannen]

With regard to testing using Java 9 modules, that's a whole different can of worms.

The world is still deciding/debating how best to do that.

FWIW, @sormuras has put in a lot of work for prototyping JUnit 5 + Java 9. Plus, JUnit 5.1 will include support for scanning the module-path in addition to the current support for scanning the class-path (i.e., to find test classes).

[sdeleuze]

@sbrannen Ok thanks, that's true that Java 9 is a whole story by itself.

@philwebb As proposed by @wilkinsona I think you should not wait 2.0.0 for the switch in order to have user feedbacks.

[philwebb]

@sdeleuze Oh wow. That was a total typo. I meant to say we should not switch :/

[wilkinsona]

Phew!

[philwebb]

Yeah, sorry about that :)

[sbrannen]

@philwebb

@sdeleuze My gut instinct is that we not should switch to it by default in 2.0.

OK. I can fully understand that rationale due to the current state of flux in the ecosystem.

But an option to choose JUnit Jupiter instead of JUnit 4 for start.spring.io would at least be quite nice! 😉

[snicoll]

But an option to choose JUnit Jupiter instead of JUnit 4 for start.spring.io would at least be quite nice!

Sorry but we have no intention of doing that. We don't want to overwhelm users with too many options and, IMO, the right call for this is to go with whatever the default in Spring Boot is.

[philwebb]

Yup I agree with @snicoll, at least for 2.0 we should offer an easy way to try JUnit Jupiter (which I think we now have) but not add it to start.spring.io. Sorry @sbrannen :(

[sbrannen]

No worries, and no reason to be _sorry_. I only said it would be "quite nice", not a must. 😉

[davidkarlsen]

Also note https://issues.apache.org/jira/browse/SUREFIRE-1330

[wilkinsona]

We discussed this today in the context of possibly switching to JUnit 5 by default in Spring Boot 2.1. We concluded that the ecosystem isn't ready yet due to the additional build tool configuration that's still required. SUREFIRE-1330 will help with that but it has no ETA.

[marcphilipp]

@wilkinsona Thanks for the update! I'm sure you're aware that Gradle now provides native support.

According to https://github.com/junit-team/junit5/issues/809#issuecomment-370908079, starting with 3.0.0-M2 Surefire will not require additional configuration anymore.

@Tibor17 Are there any updates on the Surefire timeline?

[sbrannen]

@wilkinsona, understood.

Thanks for sharing the team decision.

[bclozel]

SUREFIRE-1330 has been fixed in Surefire 2.22.0. Spring Boot has upgraded in #13500, targeting 2.1.
See also #12470.

[philwebb]

Spring Boot 2.2 will switch to JUnit 5 by default. This issue has now been superseded by the following issues:

• #13739
• #14736
• #14738
• #16942
• #16957
• #16959
• #17029