Junit5: Class-level @Test annotation

[jodastephen]

TestNG has a nice feature where the @Test annotation is added to the test class instead of the test method. When the class is annotated, all public void-returning methods on the class are treated as test methods, as per the documentation.

@Test
public class Test1 {

    public void test1() {
    }

    public void test2() {
    }

}

I'd like to see a mechanism to replicate this in JUnit 5. This would aid conversion from TestNG and be more DRY. It also avoids problems with annotating each method (eg. forgetting to add the annotation to one method in a large class, and the extra wasted vertical space.)

(Asked originally at Stack Overflow, and requested privately to raise an issue here)

[sbrannen]

FYI: if you're looking for a script to migrate from TestNG to JUnit Jupiter, feel free to use my "JUnit 3 to JUnit 4" migration script for inspiration. 😉

https://github.com/sbrannen/junit-converters/blob/master/junit3ToJUnit4.zsh

[jjDevPL]

Can I take it?

[sbrannen]

Can I take it?

What do you mean by "it"?

[jjDevPL]

I mean the feature. Can I implement this new annotation?

[sormuras]

I mean the feature. Can I implement this new annotation?

No -- the team decided against implementing this feature. Thus, I'll close this issue.

There are many ways to declare tests in Jupiter: @Test, @TestFactory, @TestTemplate (@RepeatedTest, @ParameterizedTest) -- we want them to be explicitly visible and not rely on a pattern that matches a method signature.

[jodastephen]

Can I use any of those to create a factory that operates by searching for a pattern in the method signature? While the team may not want to add that to the core, I certainly think its a better way to go (convention over configuration) and I don't mind writing some glue to do so. But I need some guidance as to what extension point is suitable.

[sormuras]

If you don't need real/full lifecycle support, you may try @TestFactory: https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests
Or give @ParameterizedTest with (static) MethodSource and/or an ArgumentsProvider a chance: https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests-sources-ArgumentsSource

[sormuras]

You may take some inspirations from here: https://github.com/junit-team/junit5/issues/1477#issuecomment-465928862

[sormuras]

Here you go:

import java.util.Arrays;
import java.util.stream.Stream;
import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;

class MyTests implements TestCase {
  public void test1() {}

  public void test2() {}

  public void test3() {}
}

interface TestCase {

  @TestFactory
  default Stream<DynamicTest> invokeAllPublicVoidTestMethods() {
    return Arrays.stream(getClass().getMethods())
        .filter(method -> method.getName().startsWith("test"))
        .filter(method -> method.getReturnType().equals(void.class))
        .map(method -> DynamicTest.dynamicTest(method.getName(), () -> method.invoke(this)));
  }
}

[jodastephen]

Thanks, I can expand from there as necessary.

[jodastephen]

Circling back on this. I have tried two separate approaches to tackle the problem:

• Using @TestFactory as above
• Using @ParameterizedTest with an @MethodSource providing Executable instances

with the check for it being a test method implemented as follows:

static boolean isTestMethod(Method method) {
    return Modifier.isPublic(method.getModifiers()) &&
            !Modifier.isStatic(method.getModifiers()) &&
            !Modifier.isAbstract(method.getModifiers()) &&
            !method.isBridge() &&
            !method.isSynthetic() &&
            method.getReturnType().equals(void.class) &&
            !isAnnotated(method, Test.class) &&
            !isAnnotated(method, TestTemplate.class) &&
            method.getAnnotation(BeforeAll.class) == null &&
            method.getAnnotation(AfterAll.class) == null &&
            method.getAnnotation(BeforeEach.class) == null &&
            method.getAnnotation(AfterEach.class) == null;
}

In both cases, the tests run OK, however there are issues:

• With @TestFactory, the @BeforeEach and @AfterEach annotations are not fired for each test (which rules this approach out)
• WIth @ParameterizedTest it can't handle the case where no tests are found (which rules this approach out)
• With both approaches, failures do not link to the correct method in the IDE (which rules both approaches out)

The last one is a killer. If a single test fails, I want to be able to double click on the test in the list and have the IDE open up the test method that failed. But using both approaches, the IDE opens up the shared method (invokeAllPublicVoidTestMethods() in the example code above). This IDE behaviour would be so annoying in daily work, that this whole approach doesn't work.

One possibility would be a new method:

`DynamicTest.dynamicTest(String displayName, Method methodToInvoke)`

The Method instance would be stored in DynamicTest and IDE code could be changed to look for it. I can raise a separate issue for that if it might be of interest, but it would clearly be lots of work for various teams to get the Method instance actually used by IDEs.

Moreover, I'd really like to auto-detect @ParameterizedTest as well, just using @MethodSource and the number of parameters to identify the test. But to do such a thing would involve duplicating lots of JUnit internal code. (Another sign that this feature request really is one that needs deep integration within JUnit, rather than being via an extension)

The only other option I can see to implement the feature would be to write my own test engine, as I really need to create TestDescriptor instances in the discovery phase. But this is of course not a realistic course of action.

Given all of the above, my migration from TestNG is going to have to annotate tens of thousands of test methods, each of which is a chance to miss a test method and reduce testing safety.

---

More generally, I'm still surprised that users of JUnit 5 don't constantly forget to annotate their tests. I know I do, and there is nothing worse than thinking you've written a test and are protected by it only to find out that you'd forgotten the annotation and it hasn't been running for the last year. Does anyone know of a maven plugin, or other tooling, that could watch out for test methods where the annotation has been forgotten?

Finally, while JUnit 5 has proven great in every other way so far, I do think it is a mistake not to offer users the __option__ to auto-detect test methods. Or alternatively some tooling that can flag up methods where the annotation is missing.

[marcphilipp]

FWIW you can specify a URI like method:org.junit.Foo#bar(java.lang.String, java.lang.String[]) and pass it to dynamicTest​(String displayName, URI testSourceUri, Executable executable) as the second argument to link the dynamic test to the executed method and get navigation support in the IDE.

[sbrannen]

And... we literally just documented that feature (URI Test Sources for Dynamic Tests) today: https://github.com/junit-team/junit5/commit/947291ef757e897a36a57d01c4a7027d4f034ccd 😉

[sormuras]

Blogged about it here https://sormuras.github.io/blog/2018-09-05-junit-5.3-dynamic-test-source

[sormuras]

https://twitter.com/jodastephen/status/1144615554579406848

import static java.util.stream.Collectors.joining;
import static org.junit.platform.commons.support.AnnotationSupport.isAnnotated;
import static org.junit.platform.commons.support.ModifierSupport.isPublic;

import java.lang.reflect.Method;
import java.util.ArrayList;
import org.junit.jupiter.api.extension.BeforeAllCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.platform.commons.annotation.Testable;

public class AssertTestableAnnotationOnAllPublicMethodsExtension implements BeforeAllCallback {
  @Override
  public void beforeAll(ExtensionContext extensionContext) {
    var methods = new ArrayList<Method>();
    for (var method : extensionContext.getRequiredTestClass().getDeclaredMethods()) {
      if (isPublic(method) && !isAnnotated(method, Testable.class)) {
        methods.add(method);
      }
    }
    if (methods.isEmpty()) {
      return;
    }
    var arrow = "\n  -> ";
    var strings = methods.stream().map(Method::toGenericString).collect(joining(arrow));
    throw new AssertionError(
        "Forgot @Test (or friends) annotation on "
            + methods.size()
            + " method(s):"
            + arrow
            + strings);
  }
}

Worth adding it to https://github.com/junit-team/junit5-samples/tree/master/junit5-jupiter-extensions?

[marcphilipp]

Since Jupiter methods don't have to be public, I'm not sure we should make it an official sample extension.

[sbrannen]

Although it's certainly handy if a project enforces that all testable methods are public and likewise that test classes otherwise contain no other public methods, I agree with Marc that it is likely not generally applicable enough to warrant using it as an official example.

For example, even if you switch to getMethods() instead of getDeclaredMethods() and filter out static methods and lifecycle methods (i.e., @BeforeEach and @AfterEach), you would still run into false positives for non-testable methods coming from interfaces (a la _testing traits_).

And yet... it's still a clever technique if that's exactly what you need. 😉