Junit5: JUnit 5 Test Suites don't run in Intellij Idea

[ericjturley]

Background

Using these annotations is the existing means to make a test suite in JUnit 5:

@RunWith(JUnitPlatform.class)
@IncludeEngines("junit-jupiter")
@SelectPackages("...")

The Problem

But on running the suite in Idea, no tests run, and it declares: Empty test suite.

Related Resources

476 explains the reasons, but only offers to log a warning. (It also contains additional links to SO and an Idea bug for context)

Acceptance Criteria

Resolution of this issue requires making some kind of recursive package test suite work in Intellij Idea.

[marcphilipp]

Just for clarification: The JUnitPlatform runner is a means to execute tests via the JUnit Platform using JUnit Vintage's JUnitCore. It is intended to bridge the gap for IDEs and build tools that don't support the new JUnit Platform.

However, it looks like IntelliJ IDEA now always uses the JUnit Platform to execute the Vintage suite class.

@akozlova Can you give us some details on how IDEA decides whether it should use the Platform or plain Vintage to execute a test?

[akozlova]

I believe that running single class with @RunWith annotation, IDEA should treat the class as JUnit 4. Given that classpath would contain jupiter classes, I need to check if it's really the case.

[akozlova]

I assumed wrong. Suppose user ask to run all tests in a package. IDEA finds junit 5 in the classpath and just compose a request like DiscoverySelectors.selectPackage(packageName). I expected that everything else would be done by junit, is it wrong? What should be changed here?
Please note that IDEA still uses junit 5.0.0-M2 to discover and launch tests.

[marcphilipp]

What do you do when you have a project that only uses JUnit 4? Will running a test use the Launcher API? Or will it use JUnit 4's mechanism (JUnitCore or Runner/RunNotifier) directly?

[akozlova]

If there is no junit 5 dependency then IDEA detects all classes to run by IDE means and then start tests with JUnitCore. If there is junit 5 in the classpath, then I rely completely on junit mechanism. Is it ok?

[marcphilipp]

IMHO that makes sense. However, in the case of @RunWith(JUnitPlatform.class) that doesn't work. 🤔

@junit-team/junit-lambda What do you guys think?

[sbrannen]

The main issue here is that the JUnit Platform does not provide a means for declaring a test suite decoupled from all testing frameworks (or TestEngine implementations) running on the platform.

So, without solving that issue, we will not resolve this GitHub issue (at least not in a general purpose fashion).

Keep in mind that the JUnitPlatform _Runner_ was only ever intended as a temporary work-around until IDEs and build tools supported launching the JUnit Platform natively. So, since IntelliJ IDEA (and beta support in Eclipse 4.7) already support the platform natively, the JUnitPlatform runner should no longer be used.

If people want a declarative means for configuring a suite on the platform (e.g., like the annotations supported by the JUnitPlatform runner), that should be addressed as a separate first-class feature of the JUnit Platform.

[signed]

This IDEA behaviour surprised me as well, even with the log messages printed.
Adding the following test to the junit5-maven-consumer sample project

package com.example.project;

import org.junit.jupiter.api.Test;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
public class PlatformRunningOnVintageTest {

    @Test
    void name() throws Exception {
        System.out.println("Hello world");
    }
}

and running in IDEA @RunWith(JUnitPlatform.class) is ignored.
The test is executed with the jupiter engine instead of junit 4 and the JUnitPlatform.class Runner what I expected.
And for some reason the warning is printed twice.

... com.intellij.rt.execution.application.AppMain com.intellij.rt.execution.junit.JUnitStarter -ideVersion5 -junit5 com.example.project.PlatformRunningOnVintageTest
Jan 11, 2017 10:12:34 PM org.junit.platform.launcher.core.ServiceLoaderTestEngineRegistry loadTestEngines
INFO: Discovered TestEngines with IDs: [junit-jupiter, junit-vintage]
Jan 11, 2017 10:12:34 PM org.junit.vintage.engine.discovery.DefensiveAllDefaultPossibilitiesBuilder$DefensiveAnnotatedBuilder buildRunner
WARNING: Ignoring test class using JUnitPlatform runner: class com.example.project.PlatformRunningOnVintageTest
Jan 11, 2017 10:12:34 PM org.junit.vintage.engine.discovery.DefensiveAllDefaultPossibilitiesBuilder$DefensiveAnnotatedBuilder buildRunner
WARNING: Ignoring test class using JUnitPlatform runner: class com.example.project.PlatformRunningOnVintageTest
Hello world

[noopur2507]

Eclipse has the same issue. See bug 512772.

[smoyer64]

@sbrannen

I've now created #681 and #687 which together allow the creation of a test suite without resorting to creating a "wrapper engine".

[marcphilipp]

JUnitPlatform now works in IDEA, Eclipse will hopefully follow soon (see bug linked above). Thus, I'm closing this issue.

[noopur2507]

What's the expected behavior/resolution here from Eclipse? Based on my understanding, Eclipse will use the Launcher API to launch a test when JUnit 5 libraries are present on the build path. As discussed earlier, junit-vintage will ignore the @RunWith(JUnitPlatform.class) class. Also, junit-jupiter ignores this class. So, how should the class be run?

[marcphilipp]

Can you just run classes annotated with @RunWith(JUnitPlatform.class) with JUnit 4?

[sbrannen]

Can you just run classes annotated with @RunWith(JUnitPlatform.class) with JUnit 4?

Yes, that works.

The catch is that you have to manually switch to the "JUnit 4 Runner" in "Run Configurations" which is a nuisance, and many people don't know about it.

[sbrannen]

Hi @noopur2507,

This is a rather special case.

In fact, it's admittedly ugly, but from my perspective there's only one viable solution in terms of User Experience in Eclipse.

If I were in charge of Eclipse's JUnit support, I would add an if-block to check if the class is annotated with @RunWith(JUnitPlatform.class). If that's the case, I'd have Eclipse run that test class using the "JUnit 4 Runner" _by default_ (even if JUnit Platform libraries are on the classpath).

Does that sound reasonable to you?

[noopur2507]

Thanks for your replies, Marc and Sam. The suggestion sounds fine. I will take it forward in the corresponding Eclipse bug.

[sbrannen]

You're welcome.

Looking forward to the improvement in Eclipse!

And... thanks for the feedback.

[pascencio]

JUnitPlatform now works in IDEA, Eclipse will hopefully follow soon (see bug linked above). Thus, I'm closing this issue.

Sorry @marcphilipp which link? I have the same bug, but with the 2019.3 version.

Thanks in the advance!

[sbrannen]

JUnitPlatform now works in IDEA, Eclipse will hopefully follow soon (see bug linked above). Thus, I'm closing this issue.

Sorry @marcphilipp which link? I have the same bug, but with the 2019.3 version.

The link was for Eclipse, not IDEA. In any case, the link is in https://github.com/junit-team/junit5/issues/569#issuecomment-283616144.