Junit5: JUnit 5 Test Suites don't run with surefire 3.0.0-M3 under pure Maven

[1vanan]

Overview

I'm trying to launch test suite with surefire plugin under Maven version 3.0.0-M3. But if I do it with @RunWith(JUnitPlatform.class) annotation in pure Maven, situation No tests were executed! occurs. I figured out that this problem appears with 2.22.x version surefire, on the earlier versions all is fine. Therefore, dynamic tests with @TestFactory work fine with newer versions and do not work on 2.21.x and earlier.

Problem reproduction

I faced this problem during migration to JUnit 5 a part of open-source project Apache Ignite [1]. Also, for clarity, I tried to reproduce it on a small project [2]. I tried to run tests in two ways: with @SelectClasses annotation and dynamically, with @TestFactory annotation (commented) [3]. The first approach works fine with 2.21.x version surefire and earlier and the second one is vice versa. For me, it is necessary to run the test suite MainSuiteTest under pure maven with the newest version that supports Java 11. I use command mvn -Dtest=MainSuiteTest test and get the next result:

LOG

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.472 s
[INFO] Finished at: 2019-02-15T18:04:51+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project quickstart: No tests were executed!  (Set -DfailIfNoTests=false to ignore this error.)

From IDE both approaches work fine and it does not depend on the surefire version.

Context

• Surefire plugin: 3.0.0-M3
• JUpiter: 5.3.2
• Vintage: 5.3.2
• Platform launcher: 1.3.2
• Platform runner: 1.3.2
• IDE: Intellij IDEA 2018.1.7 (Ultimate)
• Maven: 3.6.0
• Java: 1.8.0_152

Links

[1] https://github.com/apache/ignite/pull/5888/files#diff-04dc5d029560773ac79faaf3da0fbb22R134
[2] https://github.com/1vanan/quickstart/tree/Junit5
[3]https://github.com/1vanan/quickstart/blob/Junit5/src/test/java/com/quickstart/application/testSuites/MainSuiteTest.java

[marcphilipp]

I know it's a little confusing, but the suite class annotated with @RunWith(JUnitPlatform.class) is actually a JUnit 4 test class and as such needs to be executed with Surefire's JUnit 4 provider. Surefire 2.22.x is the first version with autodetection of the JUnit platform, i.e. as soon as junit-platform-engine is on the classpath it will attempt to execute all test classes using the JUnit Platform. However, the JUnit Platform cannot execute suites annotated with @RunWith(JUnitPlatform.class). There will eventually be support for suites in the Platform (cf. #744). In the meantime, you can force Surefire to use its JUnit 4 provider. In your example, however, I would recommend to not use any suite at all and just let Maven and your IDE discover test classes automatically.

[1vanan]

@marcphilipp Thank you for the answer!
Now with dependency

      <dependency>
        <groupId>org.apache.maven.surefire</groupId>
        <artifactId>surefire-junit47</artifactId>
        <version>3.0.0-M3</version>
      </dependency>

it works.

[1vanan]

I chose this way as it is necessary to use suites in Ignite because on TeamCity all tests are aggregated to suits and run under them [1].

[1] https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&branch_IgniteTests24Java8=pull%2F5888%2Fhead

[marcphilipp]

@1vanan I'm glad you found a solution. Thanks for letting us know!