Junit5: error when junit4 not in classpath

[JaynLau]

Steps to reproduce

if junit4 not in classpath, JUnit4VersionCheck.checkSupported() reports the error below when a junit5 test case exists in the code.

java.lang.NoClassDefFoundError: junit/runner/Version
        at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
        at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
        at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
        at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:164)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Context

• Used versions (Jupiter/Vintage/Platform): since 5.4
• Build Tool/IDE: maven, IntelliJ IDEA

Deliverables

• [x] check junit.runner.Version exists first in JUnit4VersionCheck.checkSupported()

[marcphilipp]

Why is junit-vintage-engine on the classpath but not JUnit 4?

[JaynLau]

The project uses a common parent POM. The parent POM makes JUnit 4 optional. It reports warnings but running tests successfully when the version of JUnit is below 5.6. With JUnit 5.6.0-M1, the tests are broken with java.lang.NoClassDefFoundError: junit/runner/Version

[JaynLau]

This error may also be reported if the project excludes JUnit 4 when it depends on spring-boot-starter-test.
spring-boot-starter-test depends on junit-vintage-engine by default.

[sormuras]

Exclude junit-vintage-engine then. Or don't depend on spring-boot-starter-test.

[JaynLau]

@sormuras
Nice way!
But for most people, to be honest, they don't know the relationships between junit-vintage-engine and JUnit 4. And there are many dependencies in a maven project, some people don't even care about them, except an error reported.
I think a better way is to make it more fault tolerant, easier to use, don't care about the the relationships between junit-vintage-engine and JUnit 4, whether junit-vintage-engine and JUnit 4 both on classpath or not.

[marcphilipp]

We can improve the error message but it's a user error to use the junit-vintage-engine and not have JUnit 4 on the classpath.

[sormuras]

[...] And there are many dependencies in a maven project, some people don't even care about them [...]

They'll (re-)learn to care. All dependencies are your (read: their) responsibility. Transitively.

We can improve the error message but it's a user error to use the junit-vintage-engine and not have JUnit 4 on the classpath.

+1

[krusche]

We also experience this error in IntelliJ when executing tests in IntelliJ.
We use Gradle to specify the dependencies. The following configuration leads to an exception within the IDE so that no tests are executed:

    testImplementation "org.junit.jupiter:junit-jupiter-api:5.6.0"
    testImplementation "org.junit.jupiter:junit-jupiter-params:5.6.0"
    testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.6.0"

The error message is

org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:189)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:168)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)

After a couple of hours investigating the error, we found this issue and downgraded to 5.5.2

    testImplementation "org.junit.jupiter:junit-jupiter-api:5.5.2"
    testImplementation "org.junit.jupiter:junit-jupiter-params:5.5.2"
    testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:5.5.2"

Now, the tests are executed and can be debugged within IntelliJ. The error still occurs though

Jan 27, 2020 6:07:53 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
java.lang.NoClassDefFoundError: junit/runner/Version
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:164)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
    at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.lang.ClassNotFoundException: junit.runner.Version
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 9 more

When executing the tests using the command line with gradle, both version work correctly, so the problem only occurs in IntelliJ

[marcphilipp]

@krusche You seem to be adding the junit-vintage-engine to the test runtime classpath somehow. Are you using Spring Boot by any chance?

@JaynLau Instead of excluding junit:junit when using the spring-boot-starter-test you should exclude junit-vintage-engine which is what start.spring.io does.

Team decision: Improve the error message to state that JUnit 4 is not on the classpath.

[krusche]

Sorry for the late answer, I somehow did not get a notification for your reply.
Yes, we are using Spring Boot. I just saw that the error also occurs when executing the tests on the command line. However the tests are still executed. We are now on 5.6.1 (which is stored in the variable junit_version)

It does not make a difference whether I use

testImplementation "org.junit.jupiter:junit-jupiter-api:${junit_version}"
testImplementation "org.junit.jupiter:junit-jupiter-params:${junit_version}"
testRuntimeOnly "org.junit.jupiter:junit-jupiter-engine:${junit_version}"

or

testImplementation "org.junit.jupiter:junit-jupiter-api:${junit_version}"
testImplementation "org.junit.jupiter:junit-jupiter-params:${junit_version}"
testImplementation "org.junit.jupiter:junit-jupiter-engine:${junit_version}"

What confuses me a bit, we do not use any Junit4 features, so why does it need to be on the class path anyway?

[krusche]

Just found this post on StackOverflow: https://stackoverflow.com/questions/59900637/error-testengine-with-id-junit-vintage-failed-to-discover-tests-with-spring
I will try this out

[krusche]

Yes, it helps to exclude org.junit.vintage:junit-vintage-engine in the spring-boot-starter-test dependency.

For others with the same problem, I used the following to exclude it in Gradle.

testImplementation ("org.springframework.boot:spring-boot-starter-test:${spring_boot_version}") {
        exclude module: "junit"
        exclude group: "org.junit.vintage", module: "junit-vintage-engine"
        exclude group: "com.vaadin.external.google", module: "android-json"
}