Nimble: Any way to test that fatalError() is called?

[abbeycode]

I'm testing failure cases in my code, and it doesn't look like Nimble provides a way to assert that a given block will call fatalError(). Is that correct? If so, it seems that would be a useful thing to have.

This Stack Overflow question has a couple of answers that might provide solutions: http://stackoverflow.com/questions/32873212/unit-test-fatalerror-in-swift

[modocache]

I believe it's true that Nimble doesn't provide this functionality. I think it would be very interesting to support this, but I'm not sure it's possible. The Swift stdlib's unit tests have a way to do this; perhaps we can borrow some concepts from there?

[modocache]

@morganchen12 was kind enough to point out that this is being discussed on the Swift mailing lists.

The mailing list discussion is rooted in XCTest, which is an interesting topic indeed. Nimble tries to limit its dependency on XCTest. Perhaps we could pioneer an approach and then submit an evolution proposal for it to be added to XCTest as well?

[abbeycode]

I started looking into how fatalError() is implemented. This is what I found:

swift/stdlib/public/core/Assert.swift:137:

/// Unconditionally print a `message` and stop execution.
@_transparent @noreturn
public func fatalError(
  @autoclosure message: () -> String = String(),
  file: StaticString = __FILE__, line: UInt = __LINE__
) {
  _assertionFailed("fatal error", message(), file, line)
}

_assertionFailed(...) is defined in swift/stdlib/public/core/AssertCommon.swift:115:

/// This function should be used only in the implementation of user-level
/// assertions.
///
/// This function should not be inlined because it is cold and it inlining just
/// bloats code.
@noreturn @inline(never)
@_semantics("stdlib_binary_only")
func _assertionFailed(
  prefix: StaticString, _ message: String,
  _ file: StaticString, _ line: UInt
) {
  // ...
  // Writes out the message in a safe way
  // ...
  Builtin.int_trap()
}

I can't see where int_trap() is defined, though, and how we might be able to intercept it.

[morganchen12]

I would imagine Builtin.int_trap() is just llvm.trap(), which is platform-dependent and just emits an undefined trap instruction or abort() if in doubt. I don't think it can be intercepted in a way that isn't painful.

[abbeycode]

I thought I had something...

[abbeycode]

I've been working on incorporating the solution proposed in a Stack Overflow answer, but need some guidance.

The problem is that it's a two-part solution. The first part overrides fatalError() in client code, in a way that unit tests can swap out the implementation. I don't believe Nimble currently has any component meant to be linked into a client project though, correct? Would a solution that required that be out of bounds?

You would basically need to compile a single Swift file into the client code. I'm not sure how we would approach packaging this type of change for distribution.

[morganchen12]

@abbeycode I don't think that's a perfect solution either--even if we do find a way to inject a file into the code we're trying to test, it'll create duplicate symbols within the same module if they've also tried to add that same file and in either case it won't work if they're calling Swift.fatalError() instead of just fatalError(). Ideally we want our solution to be completely agnostic of the codebase we're testing.

However, there is some good news: http://cocoawithlove.com/blog/2016/02/02/partial-functions-part-two-catching-precondition-failures.html

[jeffh]

Yeah, @morganchen12's proposal is preferred. I don't really like having a test-helper library that is linked in the main app.

Since we can't fork(), mach exception handling is probably the best solution. Otherwise fatalError() may trigger a SIGILL, although I haven't verified that or not.

[abbeycode]

@morganchen12 That's awesome. I'm beginning to implement a solution using Matt's code now.

[abbeycode]

Would it be preferrable to use Matt's code as a submodule, or to copy/appropriate it?

[modocache]

Just a quick note: I haven't had time to think about it in depth, but no matter how we incorporate fatalError testing, we should give attribution to @mattgallagher in some way. It looks like his code is going to have a heavy influence on the code we'll be including in Nimble either way. We should probably also reach out to @mattgallagher and make sure he's OK with whatever approach we end up taking--once we decide what that is.

[morganchen12]

Agree 100% with crediting Matt 👍

Joe Groff mentioned on the mailing list that it'd be best to test fatal error failures using the tools we have now since compiler support for catching such failures would be expensive. If we want Nimble to work on Linux where Mach exceptions aren't available, we'll probably need to use a fork()-based approach.

Landon Fuller has some other great writing on handling Mach exceptions (1, 2).

[mattgallagher]

By all means, copy my code. Provided you pull the copyright headers along with it, there's no other attribution required.

@abbeycode Personal opinion: subtrees are usually better than submodules, where possible. They're no more tricky than copying and pasting files into your source tree. Which you're welcome to do as well, if it simplifies things.

@morganchen12 If Linux support is valuable to you, you could catch the SIGILL with a signal handler. I've updated my post with a comment about why this is a pain in the butt but it remains possible.

[mattgallagher]

@modocache If your question was: "Am I happy with you copying the approach of my code but not the actual code itself?"... Sure. I don't blog about it because I'm feeling proprietary :-) In that case, a link back to me is not technically required but always appreciated.

[modocache]

@mattgallagher Great! I didn't have a specific question, just wanted to make sure once we decided on something we ran it past you. Glad to have your input!! Up for a code review once we have something operational? :wink:

[abbeycode]

I made a pull request to implement this, when someone has a chance to take a look! I've got some issues and questions.