Zig: Proposal: Add a way for a test to expect a panic

[mdsteele]

When I add safety checks to code (e.g. a panic that fires if the API user violates their end of the API contract), I want to be able to test those safety checks (e.g. by intentionally violating contract in the test, and checking that we get the expected panic). However, there doesn't seem to be any way to catch panics in Zig, even with a custom panic handler (which has to return noreturn, and thus cannot swallow the panic, or otherwise not halt/hang the program, as far as I can tell?), so this kind of testing doesn't seem to be possible. For comparison, Rust tests support this via the #[should_panic(expected = "substr")] annotation.

Some possible proposals:

1) Add a general mechanism for catching panics. Go has this in the form of recover, and Rust has catch_unwind. Zig could have a builtin to do a similar job. This builtin would be usable anywhere, but explicitly anti-recommended for ordinary error handling.

2) Provide a test-block-specific way to expect panics. Zig could have a builtin (e.g. @expectPanic(comptime expected_message_substr: []const u8) that would apply to the scope of the test block, and would cause the test to succeed if it panics with a matching message, or fail if it panics with the wrong message or doesn't panic at all.

3) Don't add this feature at all, and live with the fact that safety checks have to be tested at a higher level (e.g. by running each expect-panic-test as a separate subprocess and expecting that subprocess to crash, e.g. how {#code_begin|test_err|expected message#} in langref.html.in appears to work). Perhaps some mechanism could be added to std.build to make this a bit easier.

My personal preference would be for (2), since that makes the tests very easy to write. If we want (1) as a more general mechanism, then perhaps we could still have (2), or something like it (possibly in userspace), as a convenience. If we don't want to support catching panics, then (3) could work, but I worry that that would make writing safety tests painful enough that library authors would be discouraged from writing them.

[andrewrk]

This proposal would have the added benefit of allowing zig to self-host its runtime safety tests: https://github.com/ziglang/zig/blob/master/test/runtime_safety.zig

[PavelVozenilek]

For tests which call @panic or are rightfully expected to crash, add test annotation (as proposed in https://github.com/ziglang/zig/issues/1010#issuecomment-389230050, #567 and #513). Test runner would then fork, invoke just the crashing test and check the exit code. I'd say this is variant of the proposal # 3.

Do not add any _save-me-from-the-crash_ feature available to the user, that's the road to hell.

---

There's use case when you _do_ handle very, very rare situation, but add an assert there, just to make sure that this rare situation doesn't occur by some mistake in your regular code. For this special variant of assert should be invented, as proposed in https://github.com/ziglang/zig/issues/1304#issuecomment-408897007. I use not very helpful name assert2in C.

Example:

if (something-almost-impossible-happened) {
   assert2(false); // assert2 does nothing in stress tests, it fires when running ordinary code
   return 0;
}

[bronze1man]

save-me-from-the-crash feature available to the user

Please provide only one way to return and cache error.
Golang's recover feature just make programer process their error with panic and recover which is less code than if xxx return xxx.But you will need to define function with two version, one return error, another panic if error happen. And two version of one functions is the begin of hell.
But I think zig's error handle is easy enough which can just only use return error.

Test runner would then fork, invoke just the crashing test and check the exit code.

Crash process and catch by the parent process should be a good idea.

[binary132]

In my experience panic / recover is not really abused in Go much.

[isaachier]

@binary132, look at how the Go JSON parser cleverly uses recover to handle invalid input errors.

Edit: Speak of the devil... an article on this topic: https://eli.thegreenplace.net/2018/on-the-uses-and-misuses-of-panics-in-go.

[thejoshwolfe]

isn't this possible in userspace using setjmp (or equivalent) to effectively create a restore point and then longjmp out of your panic handler? Pretty sure this would never fly at comptime though.

[isaachier]

This relates more to the LLVM IR and how it implements panics. There is definitely a mechanism in the IR to throw and recover from exceptions, but Zig might not want to expose this in the language to avoid abuse.

Update: More details here: https://llvm.org/docs/ExceptionHandling.html.

[kristoff-it]

I think it's fairly common practice in Go to use recover to get a last chance to log remotely the final error message in a program before crashing.

Without a way to do so, any logging instrumentation added to an executable will not be able to record everything and in some environments it can be a problem if you can't do remote logging properly. An alternative could be to use an external process to do the remote logging (so that it can also read the Zig executable's final message), but it's a nuisance, especially when deploying to a container.

If there are no bad technical implications, I would recommend leaving the option of using some kind of recover to the user also at run-time, not just for tests, even if the odd json parser library might abuse it.

[andrewrk]

Hi Loris, this is not documented yet (See #1517) but you can override the default panic function in the root source file. In the panic function you define what happens when an assertion fails. The default behavior is here: https://github.com/ziglang/zig/blob/56e07622c692f70eb10836b86c5fda02c53e2394/std/special/panic.zig

If you specify a panic handler in the root source file, like this:

const builtin = @import("builtin");
pub fn panic(msg: []const u8, error_return_trace: ?*builtin.StackTrace) noreturn {
    // your implementation here
}

Here you can do this logging instrumentation and then call the default panic handler, or do some other strategy.

[kristoff-it]

Oh I see, so, if understand correctly, it can already be done but only at top level, where recover can instead be called at any level.

Could there be a legitimate reason for a library to want to hide panics?
I honestly don't know. Food for thought, at least for me.

Thank you for the clarification :)

[thejoshwolfe]

It would be rare for a library to handle panics or to expect panics to be handled. such a library would probably advertise itself as a library specifically for managing application state, such as a logging library that "owns" stdout, stderr, and the panic handler. in such a scenario, the library would document that any application that wishes to take advantage of the panic handling code should define a top-level panic handler that calls into the library's implementation.

The reason there's no library-local support for handling panic is that crash-related situations have app-wide impact, and managing that is the responsibility of the app.

[Manuzor]

What about zig DLLs loaded from C or something else? It's not uncommon in game code for example to have an executable that loads other subsystems (e.g. renderer) or the core game itself from DLLs. If the executable itself was written in C and one of those DLLs was written in zig, what would happen if a zig panic occurs?

I'm thinking of a scenario where parts of a larger existing system would be rewritten one by one because porting the whole thing might not be feasible. Or maybe source code is not available for porting.

[andrewrk]

If the executable itself was written in C and one of those DLLs was written in zig, what would happen if a zig panic occurs?

Zig's panic system is set up to cross object file boundaries in this way, but I don't think this use case is solved yet. The first thing that I would like to try is to have the panic handler be a function exported with weak linkage, which means that the C code could choose to override the panic function by declaring a function with the same symbol name.

Another potential thing would be to do #1439 and #2189 and then have std/special/panic.zig support something like pub panic = null; in the root source file. This would make it a linker error if the C code in this use case forgot to declare the panic handler.

I think that addresses static linking. To address DLLs, I'll have to experiment a little bit. I'm not sure how external function references work in DLLs when they are provided by the loading module.