Iglistkit: Migrate to XCTWaiter for async tests?

[rnystrom]

Idea from #608

https://developer.apple.com/reference/xctest/xctwaiter
https://developer.apple.com/reference/xctest/xctwaiterdelegate

What do these do? Are they better? Worse? Internal and Travis CI both support 10.3, so we can start using these, just no idea what they do.

[jessesquires]

just no idea what they do.

lol

[modocache]

I don't think the new API will solve the problem you two mentioned in #608; the new API don't have anything to do with timeouts specifically. Maybe use a macro or configuration value that uses a short timeout locally, and a longer timeout on CI?

As for what the new API do, here's a write-up I posted in an internal Facebook-only group:

1. Enforce the order of expectations

You can call -[XCTestCase waitForExpectatons:timeout:enforceOrder:] in order to test not only that your expectations were fulfilled, but also that they were fulfilled in a specific order:

XCTestExpectaton *expectationOne = [[XCTestExpectation alloc] initWithDescription:@"one"];
XCTestExpectaton *expectationTwo = [[XCTestExpectation alloc] initWithDescription:@"two"];

[myObject doSomethingAsyncWithCallback:^{
    [expectationOne fulfill];
    [myObject doAnotherAsyncThingWithCallback:^{
        [expectationTwo fulfill];
    }];
}];

// This test will fail if expectationTwo is fulfilled before expectationOne.
[self waitForExpectations:@[expectationOne, expectationTwo] timeout:1.0 enforceOrder:YES];

2. More flexibility when waiting

Until now, if you waited for an expectation that was not fulfilled, your test would fail -- period. Now, by using the new XCTWaiter class with a nil XCTWaiterDelegate, you may receive a result enum instead:

XCTWaiter *waiter = [[XCTWaiter alloc] initWithDelegate:nil];
XCTWaiterResult result = [waiter waitForExpectations:@[expectationOne, expectationTwo] timeout:1.0 enforceOrder:YES];
XCTAssertEqual(result, XCTWaiterResultIncorrectOrder);

This allows you to write tests for whatever you want -- you can verify that expectations are made in an incorrect order, or that expectations are not fulfilled at all.

3. Verifying an expectation is never called

You may have tests such as the following:

[myObject doSomethingAsyncWithSuccess:^{
    // ...
} failure:^{
    XCTFail(@"This should not be called!");
}];

sleep(10);

This test verifies that the failure callback is never called, but it must wait in order to make sure the failure callback has the opportunity to be called at all.

You may now use the -[XCTestExpectation inverted] property to verify that an expectation is not fulfilled:

XCTestExpectation *expectation = [[XCTestExpectation alloc] initWithDescription:@"failure callback is not called"];
expectation.inverted = YES;

[myObject doSomethingAsyncWithSuccess:^{
    // ...
} failure:^{
    [expectation fulfill];
}];

[self waitForExpectationsWithTimeout:10.0 handler:nil];

However, please be careful when verifying the order of inverse expectations. The following test is not capable of passing:

XCTestExpectation *one = // ...
one.inverted = YES;
XCTestExpectation *two = // ...

XCTWaiter *waiter = // ...
[waiter waitForExpectations:@[one, two] timeout:1.0 enforceOrder:YES];

Here XCTest naively tries to prove that first one is not fulfilled, and then two is fulfilled. You can't prove a negative, so this test is a paradox. In fact, I wouldn't recommend using enforceOrder with any inversed expectations: it doesn't make sense to say "I expect X to not happen, then Y to not happen, then Z to happen," and XCTest behaves in unintuitive ways when faced with such expectations.

4. Allowing "over-fulfillment" of expectations

By default, expectations assert if they're fulfilled more than once. Use the new -[XCTestExpectation expectedFulfillmentCount] and -[XCTestExpectation assertForOverFulfill] boolean properties to allow them to be fulfilled more than once, or an unlimited number of times.

[jessesquires]

@modocache 🙌 🙌 🙌 🙌 🙌

looks like (2) and (3) would allow us to rely less on OCMock

[rnystrom]

Should we do this?

[modocache]

Up to you. I believe there have been a few additions to the XCTestExpectation API since I posted above, which I could go into more if you'd like.

However, if you're thinking of migrating in order to prevent CI-only failures, then I don't think the new API will help. You'll still have to be mindful of the fact that CI machines are probably more resource-constrained, and so you'll still have to use a different timeout for async tests on CI than when developing.

@jessesquires mentioned other reasons to do this, like using expectations in order to test that certain code paths are not taken. I think that's reasonable, but then again, is OCMock any less stable than XCTest? Both are pretty much fundamental dependencies for nearly every Cocoa/iOS test suite.

My personal take: one could make the argument that rewriting some of the tests with this new XCTest API is a good idea, but I think most people would agree that there are more impactful things to do with your time.

[jessesquires]

Especially since we're already invested in using OCMock, I think there's little to gain here. Close?