Junit5: Maven Surefire Plugin does not execute static nested classes

[rweisleder]

Overview

As suggested by @sbrannen and maybe related to #1343.

Discussion on SO: https://stackoverflow.com/a/47982480/4365460

Consider this class:

import org.junit.jupiter.api.Test;

class OuterTest {

    @Test
    void method() {
        System.out.println("OuterTest.method()");
    }

    static class InnerTest {

        @Test
        void method() {
            System.out.println("InnerTest.method()");
        }
    }
}

The tests in InnerTest are picked up by Gradle (junit-platform-gradle-plugin) and IntelliJ IDEA, but not with the Maven Surefire Plugin.

pom.xml:

<properties>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

    <junit-platform.version>1.1.0</junit-platform.version>
    <junit-jupiter.version>5.1.0</junit-jupiter.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>${junit-jupiter.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.19.1</version>
            <dependencies>
                <dependency>
                    <groupId>org.junit.platform</groupId>
                    <artifactId>junit-platform-surefire-provider</artifactId>
                    <version>${junit-platform.version}</version>
                </dependency>
                <dependency>
                    <groupId>org.junit.jupiter</groupId>
                    <artifactId>junit-jupiter-engine</artifactId>
                    <version>${junit-jupiter.version}</version>
                </dependency>
            </dependencies>
        </plugin>
    </plugins>
</build>

The issue is reproducible with 5.0.0, 5.1.0 and 5.2.0-M1 (and related version of JUnit Platform).

Related Issues

• #1343

[sbrannen]

Thanks for raising the issue!

We'll look into it.

[rweisleder]

The user guide isn't clear about static nested classes. The only note I found is about @Nested for inner classes. I thought this is the only way and that static nested classes are not possible.

[sbrannen]

Hmmm.... then we should probably document that better.

Static nested test classes have always been considered first-class citizens, both in JUnit 4 and JUnit Jupiter.

[sbrannen]

I thought this is the only way and that static nested classes are not possible.

In terms of testing your extension -- which was your original goal -- I personally recommend one of the following two approaches.

• Use a naming convention for test classes that are expected to fail (e.g., *TestCase instead of *Tests), and then make sure your "include class name pattern" does not match for the failing test cases.
  
  
  
  • This is exactly what we do in the test suite for JUnit 5.
  
  
• Introduce a special tag to mark such "intentionally failing" test cases and exclude that tag within your build.
  
  
  
  • This is a technique I recently introduced in the build for spring-test: https://github.com/spring-projects/spring-framework/commit/56774baa2a0a15711f3895bc9d5082b1e7351dab#diff-ab5a312ff1fc3fd79e07f9bf64045b1aR110
  
  
  • And here's the custom @FailingTestCase annotation that I use on such tests: https://github.com/spring-projects/spring-framework/commit/56774baa2a0a15711f3895bc9d5082b1e7351dab#diff-f7311322595ecccce960b074cb0627dbR38
  
  

[sbrannen]

This actually appears to be a bug within the Maven Surefire Plugin itself.

Namely, Surefire does not execute public static nested classes when using JUnit 4 either.

For example, running the following via Maven with only JUnit 4 support should result in a failure, but it doesn't since the nested test class is not executed. Perhaps it's not discovered or internally filtered out.

import static org.junit.Assert.assertEquals;

import org.junit.Test;

public class JUnit4Test {

    @Test
    public void test() {
        assertEquals(3, 1 + 2);
    }

    public static class NestedTest {

        @Test
        public void test() {
            assertEquals(3, 1 + 3);
        }
    }

}

So I'm starting to wonder how many teams in the world think Maven is executing _all_ of their tests when it actually is ignoring all static nested test classes. 🤔

[sbrannen]

@Tibor17, is this something your familiar with?

[rweisleder]

The Surefire Plugin excludes _nested_ classes by default: http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#excludes

By removing all default exclusions the nested classes get executed too, JUnit 4 and JUnit 5.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
    <configuration>
        <excludes>
            <exclude/>
        </excludes>
    </configuration>
</plugin>

See also https://stackoverflow.com/a/33813668/4365460

[sbrannen]

@rweisleder,

Wow! That obviously solves the mystery.

Thanks for sharing!!!

And... I learned something: didn't realize Maven Surefire excluded all nested classes by default. 😱

[sbrannen]

I am closing this issue, since it is technically not a _bug_.

Documentation will be addressed in #1378.