Junit5: Dynamic tests, nested twice

[fab1an]

Is there a way to create a TestFactory, that returns a TestFactory?

I have a test that reads test output from a third-party program and displays is.

So what i need is:

. Section #1
   . Dynamically named Test A
   . Dynamically Test B
   . Dynamically Something else
. Section #2
   . Dynamically named Test A
   . Dynamically Test B
   . Dynamically Something else

This could be done with a parameter on @TestFactory that takes a pattern to split the returned testname into a hierarchy:

@TestFactory("{0}/{1}/{rest}")
Iterable<DynamicTest> myTests() {
    return Arrays.asList(
        dynamicTest(
             "Section #1/Dynamically named Test A",
             () -> {}),
        dynamicTest(
             "Section #2/Dynamically named Test A",
             () -> {}),
    );
}

[sbrannen]

Is there a way to create a TestFactory, that returns a TestFactory?

It sounds to me like you are not asking for that but rather for a dynamic alteration of the hierarchy within the test plan in terms of reporting.

Is that assumption correct?

[fab1an]

I don't need arbitrary hierarchies or different depths, just 2 levels, both dynamic.

The use case is a program that reads test output from another program and displays the tests.

[sormuras]

How about a DynamicTest.pushContainer(String displayName) factory method that pushes a dynamic container node on top of the current tree node level. Along with DynamicTest.popContainer() to exit the current tree level. The initial container is defined by the @TestFactory annotated method name and can't be popped.

Pseudo-code using the example from the issue description:

@TestFactory
Iterable<DynamicTest> myTests() {
    return Arrays.asList(
        dynamicTest("Factory root-level test", ...),
        pushContainer("Section #1"),
        dynamicTest("Test A", ...),
        popContainer(),
        pushContainer("Section #2"),
        dynamicTest("Test B", ...)
        ...
    );
}

This would allow arbitrary nested levels and does not rely on String-parsing.

[fab1an]

There is a reason I suggested string-parsing: The JUnit team probably doesn't want to provide arbitrary nesting capabilities, so that tests adhere to certain simple standard-structure.

Otherwise people would start exploiting that and build systems-inside-systems, like libraries that take the nesting functionality and use different annotations.

[sormuras]

You can already nest test classes in test classes in test classes ... in test classes. As long it compiles.

Related issue: #604

[fab1an]

I don't think so. Depth is limited to 2 at the moment, isn't it?

[sormuras]

As long as you use @Nested Jupiter will find and execute them:

http://junit.org/junit5/docs/current/user-guide/#writing-tests-nested
https://travis-ci.org/junit-team/junit5/builds/220345400#L738

[marcphilipp]

We could introduce the notion of a _dynamic container_, e.g.:

@TestFactory
Iterable<DynamicExecutable> myTests() {
    return Stream.of("Section #1", "Section #2")
        .map(name -> dynamicContainer(name, () -> Stream.of(
                dynamicTest("Dynamically named Test A", () -> {}),
                dynamicTest("Dynamically Test B", () -> {}),
                dynamicTest("Dynamically Something else", () -> {}))));
}

[sbrannen]

There is a reason I suggested string-parsing: The JUnit team probably doesn't want to provide arbitrary nesting capabilities, so that tests adhere to certain simple standard-structure.

As @sormuras pointed out, _arbitrary nesting_ is already supported in JUnit Jupiter via @Nested.

Furthermore, arbitrary nesting is supported by the JUnit Platform (disregarding JUnit Jupiter). See frameworks like Specsy for examples.

So the key question is whether or not JUnit Jupiter should support arbitrary nesting of purely dynamic tests.

[sbrannen]

Otherwise people would start exploiting that and build systems-inside-systems, like libraries that take the nesting functionality and use different annotations.

That ship has already sailed: See https://github.com/junit-team/junit5/issues/48#issuecomment-291397577 for an example. 😉

[sbrannen]

@marcphilipp, would DynamicExecutable be a super type for DynamicContainer and DynamicTest?

[marcphilipp]

Yes, exactly. Maybe DynamicNode would be a better name.

[mfulton26]

I'm very excited for dynamicContainer. If there is anything I can do to assist (testing, etc.) please let me know.

[sormuras]

I almost got it working locally - will push a work-in-progress version for testing later today.

[sormuras]

Still work-in-progress as one test is disabled and there's a total lack of documentation, but here we go: https://travis-ci.org/junit-team/junit5/builds/226708262#L646

[sormuras]

Addressed by https://github.com/junit-team/junit5/commit/1a57e4f386a9f8cd6be0a08a4d914349d32a7f01 on master