Junit5: Unsupported or missing version of JUnit 4 should fail the build

Created on 3 Jul 2019 · 15Comments · Source: junit-team/junit5

I'm using junit and vintage from the bom 5.5.0 in a Maven based project.
https://github.com/nielsbasjes/yauaa

In this project I have created a udf for Apache Pig.
In the pig dependency for this udf I also get junit 4.11.

The consequence of this older version of junit is that I get this perfectly fine stack trace during my build:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Jul 02, 2019 11:42:33 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)

and later on

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

The bug I'm reporting is that I think this is should be a fatal error and not just a warning.

None of the tests I have ran and the ONLY reason I noticed this is because I saw a sudden drop in the code coverage of the tests.

So if this problem happens then please fail the build.

Vintage diagnostics

Source

nielsbasjes

Most helpful comment

This was fixed in #1909.

@nielsbasjes Would you care to verify using the latest snapshot?

marcphilipp on 23 Sep 2019

👍2

All 15 comments

Tentatively slated for 5.6 M1 for _team discussion_

sbrannen on 3 Jul 2019

Related to #242.

marcphilipp on 3 Jul 2019

Team Decision: We agree that this should fail during test execution. 🙂

marcphilipp on 5 Jul 2019

👍2

I updated the title to point out that a missing JUnit 4 artifact should also fail the build for use cases where someone excluded JUnit 4 from the classpath but left junit-vintage-engine in the classpath.

The logic in JUnit4VersionCheck should already handle this use case as well, but I wanted to ensure we cover both use cases explicitly: _missing_ and _unsupported_ JUnit 4 artifacts.

sbrannen on 23 Sep 2019

This was fixed in #1909.

@nielsbasjes Would you care to verify using the latest snapshot?

marcphilipp on 23 Sep 2019

👍2

The problem has not been fixed in junit-vintage-engine-5.6.0-20190924.094617-81.jar.

I get

WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.

then

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

and

[INFO] BUILD SUCCESS

To facilitate you guys repeating what I have I created a testing branch on my project
https://github.com/nielsbasjes/yauaa/tree/ReproduceJunit5Issue1944

I have the sonatype snapshot repo enabled using my ~/.m2/settings.xml

To limit the time needed to reproduce this (some of my tests take a while) do:

mvn clean install -DskipTests=true

verify I actually use the right version in the pom.xml

fgrep junit5.version ~/.m2/repository/nl/basjes/parse/useragent/yauaa-parent/5.13-SNAPSHOT/yauaa-parent-5.13-SNAPSHOT.pom 
    <junit5.version>5.6.0-SNAPSHOT</junit5.version>
    <version>${junit5.version}</version>

then

cd udfs/pig/

verify the actual dependency versions for this module

$ mvn dependency:list | fgrep junit
[INFO]    org.junit.jupiter:junit-jupiter-api:jar:5.6.0-SNAPSHOT:test
[INFO]    org.junit.platform:junit-platform-commons:jar:1.6.0-SNAPSHOT:test
[INFO]    org.junit.jupiter:junit-jupiter-engine:jar:5.6.0-SNAPSHOT:test
[INFO]    org.junit.platform:junit-platform-engine:jar:1.6.0-SNAPSHOT:test
[INFO]    org.junit.vintage:junit-vintage-engine:jar:5.6.0-SNAPSHOT:test
[INFO]    junit:junit:jar:4.11:provided

then

mvn clean test

I see this (which says BUILD SUCCESS):

$ mvn clean test
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building Yauaa : UDF : Apache Pig 5.13-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ yauaa-pig ---
[INFO] Deleting /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target
[INFO] 
[INFO] --- git-commit-id-plugin:3.0.0:revision (get-the-git-infos) @ yauaa-pig ---
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-maven) @ yauaa-pig ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.4:prepare-agent (pre-unit-test) @ yauaa-pig ---
[INFO] jacoco.surefireArgLine set to -javaagent:/home/nbasjes/.m2/repository/org/jacoco/org.jacoco.agent/0.8.4/org.jacoco.agent-0.8.4-runtime.jar=destfile=/home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/jacoco.exec
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ yauaa-pig ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ yauaa-pig ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ yauaa-pig ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ yauaa-pig ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ yauaa-pig ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Sep 24, 2019 12:24:27 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:168)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:155)
    at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:120)
    at org.apache.maven.surefire.junitplatform.TestPlanScannerFilter.accept(TestPlanScannerFilter.java:56)
    at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:102)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.scanClasspath(JUnitPlatformProvider.java:143)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Sep 24, 2019 12:24:27 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:168)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:155)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-checkstyle-plugin:3.1.0:check (checkstyle-check) @ yauaa-pig ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.4:report (post-unit-test) @ yauaa-pig ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.658 s
[INFO] Finished at: 2019-09-24T12:24:28+02:00
[INFO] Final Memory: 48M/599M
[INFO] ------------------------------------------------------------------------

nielsbasjes on 24 Sep 2019

@nielsbasjes Thanks for testing! Must be Surefire swallowing the error. I'll take a closer look myself.

marcphilipp on 24 Sep 2019

Just by looking at the messages it seems the JUnitException is classified as a WARNING.

And a warning should not fail the build...

nielsbasjes on 24 Sep 2019

@nielsbasjes We still log this as a warning when Surefire calls discover(). However, if Surefire called execute() with the same TestPlan, it would throw an exception. I'll add an integration test.

marcphilipp on 24 Sep 2019

@nielsbasjes The missing piece was that Surefire bundles an old version of junit-platform-launcher. If you add the snapshot version to the test scoped dependencies, it fails.

I've created #2028 to ensure Surefire and other build tools and IDEs upgrade their bundled Launcher once 5.6 is released.

marcphilipp on 25 Sep 2019

@marcphilipp Thanks. I've check it and when I make my maven look like this

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>5.6.0-SNAPSHOT</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.vintage</groupId>
      <artifactId>junit-vintage-engine</artifactId>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

it correctly fails with

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Sep 29, 2019 4:12:16 PM org.junit.platform.launcher.core.DefaultLauncher discoverEngineRoot
SEVERE: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:166)
    at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:122)
    at org.apache.maven.surefire.junitplatform.TestPlanScannerFilter.accept(TestPlanScannerFilter.java:56)
    at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:102)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.scanClasspath(JUnitPlatformProvider.java:143)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Sep 29, 2019 4:12:16 PM org.junit.platform.launcher.core.DefaultLauncher discoverEngineRoot
SEVERE: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
    at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:166)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:130)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] Running junit-vintage
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.006 s <<< FAILURE! - in junit-vintage
[ERROR] junit-vintage  Time elapsed: 0.003 s  <<< ERROR!
org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
    at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   VintageTestEngine » JUnit TestEngine with ID 'junit-vintage' failed to discove...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.972 s
[INFO] Finished at: 2019-09-29T16:12:17+02:00
[INFO] Final Memory: 24M/396M
[INFO] ------------------------------------------------------------------------

Thanks for fixing this.

nielsbasjes on 29 Sep 2019

@nielsbasjes Thanks for double-checking! 👍

marcphilipp on 29 Sep 2019

I had this problem when using junit 4.8.2 and then changing it to 4.13 worked (see doc). I am just commenting it here so as to validate my understanding that vintage engine requires junit 4.13 and not older to be able to detect tests.

Please correct me if I am wrong.