Zig: User-defined test runner

[data-man]

Usages:

• benchmarks (e.g. if the function name starts with "bench_")
• parallel execution
• testing time
• something other

Possible implementation:

test_runner.zig
``zig const std = @import("std"); const io = std.io; const builtin = @import("builtin"); const root = @import("root"); //the problem is here:root` doesn't exist in tests

pub const io_mode: io.Mode = builtin.test_io_mode;

var log_err_count: usize = 0;

pub fn main() anyerror!void {
const test_fn_list = builtin.test_functions;

if (@hasDecl(root, "testRunner")) {
   return root.testRunner(); //the problem is here: `root` doesn't exist in tests

...///the rest source
}

[FireFox317]

Or maybe this can be solved by having a compiler flag for zig test pointing to the test_runner in use. For example zig test --test-runner ./utils/test_runner.zig

[ityonemo]

I support something like this as it might make test integration less hackish in my "zigler" ffi library.

[demizer]

This would allow me to override the std.log function for my tests.

[data-man]

@demizer
Hacked implementation:
Sorry, nevermind. It's wrong.

const std = @import("std");
const root = @import("root");

pub fn main() void { //Trick is here!
}

test "root's trick" {
    std.debug.print("main is declared: {}\n", .{ @hasDecl(root, "main") });
}

[PavelVozenilek]

I'd proposed it in #567. Important thing for a custom test runner is ability to have parameters for individual tests and ability to process them by the test runner.

For example:

test (x=10 |10 ms <  time < 100 ms | whatever) // these are test parameters, to be parsed and used by test runner
{
  ...
}

Tests should not be restricted to some special "testing mode". I sometimes rerun tests also in release mode, to make sure release mode didn't screw up something. I also like to run "recent tests" (tests from recently modified source files) every time an application start - it is a handy feature to catch problems really early.

Another useful testing feature would be easy to use mocking: #500.

Parallel execution of tests is not a wise idea. Code not intended for concurrent run would now run in parallel...