Junit5: Dynamic containers: Support partial dynamic tree errors

[mfulton26]

Overview

Feature request. Dynamic containers provide the ability to create entire trees of tests and containers dynamically. However, if an exception is thrown while creating a sub portion of the tree then the entire tree results in an error and no test results are gathered.

Consider the following example:

class DemoTests {
    @TestFactory
    @DisplayName("1")
    Stream<DynamicNode> demo() {
        return Stream.of(
                dynamicContainer("1.1", Stream.of(
                        dynamicTest("1.1.1", () -> {
                        }),
                        dynamicTest("1.1.2", () -> {
                        })
                )),
                dynamicContainer("1.2", IntStream.rangeClosed(1, 10)
                        .mapToObj(number -> {
                            if (number == 9) {
                                throw new RuntimeException();
                            }
                            return dynamicTest("1.2." + number, () -> {
                            });
                        })
                )
        );
    }
}

The actual test result tree is

• :x: Test Results
  
  
  
  • :x: DemoTests
  
  
  • :x: 1
    
    
    
    
    
    • :x: Class Configuration
    
    
    
  
  

The desired test result tree is

• :x: Test Results
  
  
  
  • :x: DemoTests
  
  
  • :x: 1
    
    
    
    
    
    • ✔️ 1
    
    
    
    • ✔️ 1
    
    
    
    • ✔️ 2
    
    
    
    • :x: 2
    
    
    
    • :x: Class Configuration
    
    
    
  
  

Where part of the tree is executed.

Deliverables

• [x] Some way to provide a supplier of dynamic nodes instead of a list/stream of nodes directly.
• [ ] Support for executing available tests in dynamic trees of tests even when branches of the tree are not executable due to throwing an exception.

[marcphilipp]

How about adding an overload dynamicContainer(String, Supplier<Stream<? extends DynamicNode>>)?

[mfulton26]

That is strange that the case to Executable is necessary for the nested
dynamic tests.

I found in the DynamicContainer constructor that the stream is colleced
to an unmodifiable list in the constructor. I wonder if instead of adding a
Supplier overload if this collection to a list can be delayed to
elsewhere in the test engine so that if it throws an exception it can be
caught and dealt with appropriately. That way an overload is not needed and
test developers can define their streams as they normally would.

Thoughts?

On Tue, Jun 20, 2017 at 7:01 AM Christian Stein [e-mail]
wrote:

Having such an overload, one could write the example from the user
documentation as:

Stream.of("A", "B", "C")
.map(input -> dynamicContainer("Container " + input,
Stream.of(
dynamicTest("not null", (Executable)() -> assertNotNull(input)),
dynamicTest("fail on B", () -> { if ("B".equals(input)) fail(input); }),
dynamicContainer("properties",
() -> Stream.of(
dynamicTest("length > 0", (Executable) () -> assertTrue(input.length() > 0)),
dynamicTest("not empty", (Executable) () -> assertFalse(input.isEmpty()))
))
)));

Note the redundant but necessary (Executable) casts... strange.

The result looks like:

[image: dynamic_nodes_891]
https://user-images.githubusercontent.com/2319838/27332121-d94c8082-55c0-11e7-8855-8966255b4f5f.png

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/junit-team/junit5/issues/891#issuecomment-309731714,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVH3d30wbV4EKY8e5Pm1d74DaqaS_A7ks5sF7SCgaJpZM4N85o8
.

[marcphilipp]

Team decision: Store Stream in DynamicContainer and iterate over it lazily.