Junit5: Test methods cannot combine RepeatedTest and ParameterizedTest annotations

[SamCarlberg]

Overview

The title pretty much explains it. Test methods annotated with @RepeatedTest(n) and @ParameterizedTest will fail _n_ times with a ParameterResolutionException, then run the parameterized test as normal

Test class

class JUnitCannotCombineRepeatedAndParameterizedTests {

  @RepeatedTest(5)
  @ParameterizedTest
  @MethodSource("source")
  void test(String param) {
    System.out.println(param);
  }

  static Stream<Arguments> source() {
    return Stream.of(Arguments.of("foo"),
                     Arguments.of("bar"));
  }

}

Expected result

The parameterized test should run 5 times with the parameters generated by the source() method eg

[repetition 1 of 5]
  foo
  bar
[repetition 2 of 5]
  foo
  bar
...

Actual result

There are 5 repetitions of the test, each one failing with this exception:

org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [java.lang.String arg0] in executable [void JUnitCannotCombineRepeatedAndParameterizedTests.test(java.lang.String)].

After this, the test is run with the parameters generated by source()

This occurs on 5.0.0 and 5.1.0-M1

[sbrannen]

Well, that's actually the expected behavior.

You cannot have two @TestTemplate implementations executing at the same time for a single method.

In addition, it was never foreseen that one would _nest_ @TestTemplate implementations, which is essentially what you are proposing here.

So I'm a bit torn as to whether or not we should (or could) consider making a change like this.

@junit-team/junit-lambda, thoughts?

[TWiStErRob]

Combining different types of tests (dyanmic, repeated, parameterized) should be expected behavior, so 👍

Here's example:

fun ClosedRange<Int>.random() =
    ThreadLocalRandom.current().nextInt((endInclusive + 1) - start) + start

Ideally:

@ParameterizedTest
@CsvSource(value = ["0, 10", "15, 25", "-3, 4"])
@RepeatedTest(100)
@Test fun picksNumberInClosedRange(min: Int, max: Int) {
    val value = (min..max).random()

    assertThat(value, greaterThanOrEqualTo(min))
    assertThat(value, lessThanOrEqualTo(max))
}

an Ugly workaround:

@RepeatedTest(100)
@TestFactory fun picksNumberInClosedRangeParametersInline() =
    arrayOf(
        arrayOf(0, 10),
        arrayOf(15, 25),
        arrayOf(-3, 4)
    )
    .map { (min, max) -> dynamicTest("Between $min and $max") {
        val value = (min..max).random()

        assertThat(value, greaterThanOrEqualTo(min))
        assertThat(value, lessThanOrEqualTo(max))
    } }

which doesn't work because the RepeatedTest is simply ignored. So dropping all the annotation fancy, nice test naming and variable input sources 😞 I just create them myself:

@TestFactory fun picksNumberInClosedRangeJustDynamic() =
    arrayOf(
        arrayOf(0, 10),
        arrayOf(15, 25),
        arrayOf(-3, 4)
    )
    .map { (min, max) -> dynamicTest("Between $min and $max") {
        val value = (min..max).random()

        assertThat(value, greaterThanOrEqualTo(min))
        assertThat(value, lessThanOrEqualTo(max))
    } }
    .repeat(100)
}

fun Iterable<DynamicTest>.repeat(count: Int) =
    (1..count).asSequence().flatMap { this.asSequence() }.asIterable()

[leksak]

I'm writing code that could make use of this functionality right now

[marcphilipp]

This will be addressed by #871 which would allow to use @RepeatedContainer and/or @ParameterizedContainer on an enclosing, potentially @Nested, test class.

[honza-zidek]

There is a Stack Overflow issue How to repeat JUnit5 tests on class level? which would be solved by fixing this problem...

[danielhodder]

I'm not sure that #871 quite captures the same request as the one being requested here. The main use I want to use this capability for is combining different extensions together. I will give an example:

I have a TestTemplateInvocationContextProvider which creates a new databsae for each test, since I am doing database tests. It does this across several different database vendors, and setup scenarios. The connection details are provided to the tests with a ParameterResolver which provides a java.sql.Datasource

When writing my tests I often want to verify that a bunch of different 'bad' cases result in the correct exceptions so I want to use the @ParameterizedTest annotation; so I don't have to write each test out by hand. Here's an example:

@BeforeEach
public void setupService(Datasource datasource) {
// service init code here
}

@DatabseTest
@ParameterizedTest
@ValueSource(strings={"", null})
public void testServiceFailsIfInputIsNotValid(String invalidState) {
// verification code here
}

In this example I want the cartesion product of all the test execution contexts. Or I guess in different speek: for each nested context I want to run the whole setup again. I don't think the suite idea described in #871 will help that much since it really only seems to add one level, and that would help with one set. But what if I want to combine three, or more, of these extensions together. It would be nice if there was a way of changing the behavior of the TestTemplateTestDescriptor so that it would generate one test for each combination of TestTemplateExecutionContexts rather than just one for each.

[marcphilipp]

Do you need TestTemplateInvocationContextProvider or would an ArgumentsProvider suffice? If so, please voice your support on #1427. 🙂

[danielhodder]

I think maybe we could re-write to use that. The actual test harness implements BeforeAll, and AfterAll to manage some JVM global configuration, and AfterEach to clean up the databases created for each test case. It might be possible to create a similar thing with the ArgumentProvicer but the nice thing about the TestTemplateInvocationContextProvider is that each test can, itself, have it's set of Extensions.

[danielhodder]

It turns out this isn't that hard to do in principal so I've thrown together a PR that shows one way of solving the issue. It's open for feedback as much as anything.

[tommai78101]

Hello, can someone re-evaluate this issue? I see that this issue is closed, because it's superseded by another issue. However, there is a PR addressed to fix this issue, and no one has reviewed the PR as of 2021.

Is the current status, "open", because a PR is submitted, or is it "closed" and we need to invalidate the PR?

[sbrannen]

I see a branch with the proposal (https://github.com/danielhodder/junit5/tree/feature/1224_repeatable_and_parameterized_tests), but is there an actual PR associated with that branch?

@danielhodder ?

[danielhodder]

2409 is the PR for that branch. There is a description of the change there, but it amounts to changing how combinations of TestTemplates are handled.

[sbrannen]

Is the current status, "open", because a PR is submitted, or is it "closed" and we need to invalidate the PR?

The status of this issue is still closed.

PR #2409 will be assessed by the team in order to determine its fate.

[tommai78101]

Thanks for the clarification.