Nimble: Swift Assertions in Xcode 9.1

[blessingLopes]

What did you do?

I used expect { () -> Void in fatalError() }.to(throwAssertion()) in a test, which
throws a fatal error, completely stops the tests suite execution, and we get a

expected to raise any exception, got no exception

test failed message

What did you expect to happen?

expect { () -> Void in fatalError() }.to(throwAssertion()) to succeed.

What actually happened instead?

expect { () -> Void in fatalError() }.to(throwAssertion()) fails.

Environment

List the software versions you're using:

• Nimble: 7.0
• Xcode Version: 9.1 (9B55)
• Swift Version: 4.0

Please also mention which package manager you used and its version. Delete the
other package managers in this list:

• Cocoapods: 1.3.1

Maybe I'm missing something or doing something wrong, my deep apologies if that's the case.
Thanks for any feedback!

[szweier]

Seeing the same issue seems to be Xcode 9.1 specific.

[VojtaStavik]

This is happening only when the test suite is run in the host application. For standalone tests everything works OK.

[bryantjustin]

You can continue the tests by clicking on "Continue program execution" but this gets tedious if you have a few of these type of tests.

[msewell]

As a sort-of workaround for getting stuck on debugger breakpoints for fatal errors when running tests, you can run your tests without debugging support. Disable 'Debug Executable' in your scheme by going to _Edit Scheme > Test > Info_ and unchecking 'Debug Executable'.

[bryantjustin]

@msewell Thanks for the tip. I just confirmed it and turning off "Debug Executable" stops this from happening. I guess if we need to actually have the debugger up for a test, we'll probably want to isolate using fdescribe(), fcontext(), or fit() to avoid having to go through all the fatalErrors.

[blessingLopes]

Many thanks for the feedback and useful suggestions!

[ghost]

Maybe, you should mention this in documentation? Solution was not obvious for me.

[msewell]

@MortyMerr: Disabling debugging during test runs isn't really a solution (it hardly even is a workaround). This issue still needs a "real" fix.

[rafaelnobrepd]

A side effect of disable debugging during tests (which is quite inconvenient as is), is that it seems to affect coverage reports (always reports 0% when debug executable is disabled).
Unfortunately its a real pain, to the point that I'm considering removing the assertion expectations

[mvandervelden]

Haven't had this issue before, until I started using Xcode 9.3 / Swift 4.1, where it fails consistently. Can confirm that disabling debugging during tests works, but as mentioned before, it's not a suitable workaround.

[RetepV]

Woops. And I am finding this issue as well, after having upgraded to Xcode 9.3.

            it("Assertion from inch unit test") {
               expect {
                    try HeightConverter.stringForProfileHeightUnit(HeightUnit.inch)
                }.to(throwAssertion())
            }

Code to test executes the following assert:

            assert(false, "Profile only defines cm or ft. Anything else is a bug.")

In Xcode 9.2, the test pass fine. In Xcode 9.3, the line actually asserts and the test run fails because it detects an assertion (which it's actually supposed to detect).

[drekka]

Hitting this with 9.3 as well.

It appears from my testing that Apple has modified the way the debugger handles program exits and is aggressively halting at the point where the code exits. Thus making it impossible for Nimble to trap it.

Turning off debugging of the executable does indeed halt the problem but is not really a solution as it effectively disables test debugging.

I'm not sure that this can be addressed either with Nimble or XCTest functionality.

[danielaRiesgo]

Hello. Just wanted to know if there is an update about this?

I'm having the same problem. Worked in XCode 9.2 but not in XCode 9.3.

[drekka]

Problem is persisting in Xcode 9.4. I suspect this will not be fixed and Nimble will have to find an alternative solution. For myself, I've gone with the approach of using a local override of fatalError(...) and then adding code to set a mock equivalent during testing. Unfortunately I cannot publish the code here as I did it for a client and they would have to approve it (unlikely!), however I can say it was based on https://medium.com/@marcosantadev/how-to-test-fatalerror-in-swift-e1be9ff11a29

[ikesyo]

Looks like we can adopt https://github.com/mattgallagher/CwlPreconditionTesting/commit/ce96cb1d81644b92c66fd2b622260905526a7c5f to address the issue.

Related to https://github.com/apple/swift/pull/11329.

[ikesyo]

Fixed by #545.