Quick: Provide an API around continueAfterFailure for XCTestCases

[jeffh]

More historic context is on Nimble#69.

By default, XCTestCase sets this as YES, which makes this continue running the test, even when a failure is reached.

I'm posing this a few questions:

1. Should Quick provide an API that wraps this behavior (eg - global application of a QuickConfiguration setting)?
2. what is the default value for this setting, if we're providing one.

I'm my votes are yes for the first one and continueAfterFailure = true (whatever XCTest defaults to).

[modocache]

I agree on both points! :+1:

[marcelofabri]

@modocache Any directions on where how to implement this? Where to bridge from Configuration to the current XCTestCase?

[modocache]

See QuickConfiguration, I think that's the Objective-C bridgeable class that end users use. Also see the documentation on "configuring Quick". Also, /cc @Quick/contributors!

[marcelofabri]

I think I wasn't clear. I wanted to know if there's a place in Quick where there're both a Configuration and a QuickSpec so I could read a value from Configuration and set on QuickSpec.

I found that World is where the Configuration is kept, but haven't found how to go from there to a QuickSpec.

[marcelofabri]

Digging more, I think it could work if we added the QuickSpec as a param to Example's run and send it on addInstanceMethodForExample:classSelectorNames: from QuickSpec.

Would that be a welcomed change?

[marcelofabri]

Actually, configuring continueAfterFailure = false doesn't really seem to be working with Quick:

class MySpec: QuickSpec {
    override func setUp() {
         continueAfterFailure = false
    }
    override func spec() {
         it("fails only once") {
             fail("Only I appear")
             fail("And I don't")
         }
    }
}

I get both failures 😬

[iby]

Are there any workarounds for this in 2019? Tried overriding the continueAfterFailure property and changing it from beforeEach/All – no luck.

I'm guessing this flag tells the XCTest assertion to not report anything if the previous one failed. Also guessing that Nimble assertions work differently from XCTest ones and simply ignore this flag. Is this the reason why it's not working?

[damirstuhec]

Wondering the same thing. This would be great to have!

[andrewchang-bird]

The only workaround I’ve found is to call stop() twice on the current XCTestRun. This causes an assertion failure within xctest that stops the current test from executing further, but allows other test cases to continue.

Here’s what I’m doing which supports both vanilla XCTest and Nimble:

class TestKiller: NSObject, XCTestObservation {
  override init() {
    super.init()
    XCTestObservationCenter.shared.addTestObserver(self)
  }

  private(set) var testCase: XCTestCase?
  func testCase(_ testCase: XCTestCase,
                didFailWithDescription description: String,
                inFile filePath: String?,
                atLine lineNumber: Int) {
    testCase.continueAfterFailure = false
    self.testCase = testCase
  }
}

let testKiller = TestKiller()
XCTFail("Foo bar error")

testKiller.testCase?.testRun?.stop()
testKiller.testCase?.testRun?.stop()

assert(false) // This will never be executed.