Zig: Proposal: Nested tests

[tgschultz]

Now that tests are namespaced, it seems useful to allow them to be nested within one another as well. Take this example from std.fmt:

test "fmt.format" {
    {
        const value: ?i32 = 1234;
        try testFmt("optional: 1234\n", "optional: {}\n", value);
    }
    {
        const value: ?i32 = null;
        try testFmt("optional: null\n", "optional: {}\n", value);
    }
    {
        const value: anyerror!i32 = 1234;
        try testFmt("error union: 1234\n", "error union: {}\n", value);
    }
    {
        const value: anyerror!i32 = error.InvalidChar;
        try testFmt("error union: error.InvalidChar\n", "error union: {}\n", value);
    }
    {
        const value: u3 = 0b101;
        try testFmt("u3: 5\n", "u3: {}\n", value);
    }
    {
        const value: u8 = 'a';
        try testFmt("u8: a\n", "u8: {c}\n", value);
    }
    {
        const value: u8 = 0b1100;
        try testFmt("u8: 0b1100\n", "u8: 0b{b}\n", value);
    }
//...
}

Which could use nesting to better document each test:

test "fmt.format" {
    test "optional w/ value" {
        const value: ?i32 = 1234;
        try testFmt("optional: 1234\n", "optional: {}\n", value);
    }
    test "optional w/ null" {
        const value: ?i32 = null;
        try testFmt("optional: null\n", "optional: {}\n", value);
    }
    test "error union w/ value" {
        const value: anyerror!i32 = 1234;
        try testFmt("error union: 1234\n", "error union: {}\n", value);
    }
    test "error union w/ error" {
        const value: anyerror!i32 = error.InvalidChar;
        try testFmt("error union: error.InvalidChar\n", "error union: {}\n", value);
    }
   test "sub-byte-width int" {
        const value: u3 = 0b101;
        try testFmt("u3: 5\n", "u3: {}\n", value);
    }
    test "character output" {
        const value: u8 = 'a';
        try testFmt("u8: a\n", "u8: {c}\n", value);
    }
    test "binary output" {
        const value: u8 = 0b1100;
        try testFmt("u8: 0b1100\n", "u8: 0b{b}\n", value);
    }
//...
}

Which could look like this in the output:

>zig test std.zig --test-filter "format"
Test 1/1 std.fmt.format...
    Test 1/7 std.fmt.optional w/ value...OK
    Test 2/7 std.fmt.optional w/ null...OK
    Test 3/7 std.fmt.error union w/ value...OK
    Test 4/7 std.fmt.error union w/ error...OK
    Test 5/7 std.fmt.sub-byte-width int...OK
    Test 6/7 std.fmt.character output...OK
    Test 7/7 std.fmt.binary output...OK
Ok
All Tests Passed

In this case each test could just be its own test block, but one can imagine situations where a test requires initialization of some data structures that will be reused across multiple tests, in which case it might make sense to do that in a single outer test block instead of every text block independently.

[Sahnvour]

Copy pasting setup code in multiple tests is a real pain point, so this would be an improvement. But if having shared "setup" code between different tests is a rationale for this feature, shouldn't we also consider the "teardown" part? In which case, we are actually designing a test framework.

[andrewrk]

I think this is where zig test stops and hands off to your own mechanisms. Namespacing can be done with structs. Setup/teardown can be done with function calls and defer. A zig test can call a into a test framework which has its own way of specifying tests and its own setup/teardown. One can set up their own test harnesses in the build system to do more complex things. This is one of those points where the toolchain and language have done enough, and it's time for userland to step in and take charge.