Quick: Add a test to Quick's test suite that verifies afterSuite is called

[norio-nomura]

Reproducing steps:

1. Checkout 5fb172b
2. Set break point at AfterSuiteTests.swift:10 and AfterSuiteTests+ObjC.m:12
3. Run test

Expected result:

1. Debugger stopped at break points set on step 2

Actual result:

1. Debugger did not stop at break points set on step 2

[takecian]

Example.swift:88

if !world.isRunningAdditionalSuites && numberOfExamplesRun >= world.includedExampleCount {
            world.suiteHooks.executeAfters() // <- never executed
        }

I will investigate more..

[takecian]

QuickSpecRunner.m:12(Test code for Quick) sets isRunningAdditionalSuites = YES and no one sets isRunningAdditionalSuites = NO

XCTestRun *qck_runSuite(XCTestSuite *suite) {
    [World sharedWorld].isRunningAdditionalSuites = YES;

At Example.swift:3, numberOfExamplesRun seems incremented through all TestSuites. numberOfExamplesRun will not be set to 0 after running suite.
numberOfExamplesRun will be greater than world.includedExampleCount.

private var numberOfExamplesRun = 0

I think cause of afterSuite not called is above, how do we solve it..

world.numberOfExamplesRun seems have count of all of specs.

[takecian]

To fix this issue, I think that we need to change how to count examples and set isRunningAdditionalSuites to NO(false) .

In addition, as of now, some tests seems executed twice, they were executed as a part of FunctionaTests and their TestSuite. Is it expected behavior?

class FunctionalTests_AfterSuite_AfterSuiteSpec: QuickSpec {
    override func spec() {
    ....
    }
}

class FunctionalTests_AfterSuite_Spec: QuickSpec {
    override func spec() {
        it("is executed before afterSuite") {
         ....  <- Executed twice. 'A part of FunctionalTests' and 'a part of AfterSuiteTests'
        }
    }
}

class AfterSuiteTests: XCTestCase, XCTestCaseProvider {
    ...
    func testAfterSuiteIsNotExecutedBeforeAnyExamples() {
        // Execute the spec with an assertion after the one with an afterSuite.
        let result = qck_runSpecs([
            FunctionalTests_AfterSuite_AfterSuiteSpec.self,
            FunctionalTests_AfterSuite_Spec.self
            ])

        // Although this ensures that afterSuite is not called before any
        // examples, it doesn't test that it's ever called in the first place.
        XCTAssert(result.hasSucceeded)
    }
}

What do you think, @modocache ? It would be helpful for any comment 😄

[modocache]

So, just so I'm clear: this problem only occurs in our test suite? Or is this a regression in afterSuite? That is, does this affect Quick's end users? Just curious.

FunctionalTests_AfterSuite_AfterSuiteSpec is run once via XCTest's automatic discovery, and once within testAfterSuiteIsNotExecutedBeforeAnyExamples, so yes, it's expected that it will run twice.

Looks like I originally added isRunningAdditionalSuites, and the code you mention in your comment, two years ago: https://github.com/Quick/Quick/commit/9719468ece992f95ec4fcccda7dfcc94a07661a3. Here's what I wrote in that commit message:

Running examples multiple times causes the afterSuite logic to misfire. Add a flag to World in order to prevent afterSuite from running in such a case.

So it looks like that code exists for the explicit purpose of preventing afterSuite from executing in some situations.

Perhaps our test suite simply doesn't verify that afterSuite is called? I would love to add a test for this behavior, but I guess two years ago I wasn't smart enough to figure out how to do so. @takecian, could you:

1. Tell me whether afterSuite is executed for our users?
2. Think about some ways to verify afterSuite is called as part of Quick's CI? Is it possible?

[noisypigeon]

Do you mind if I assign this issue to you, @takecian?

[takecian]

@istx25 Sure!

[takecian]

Tell me whether afterSuite is executed for our users?

I wrote sample tests as follows(v0.9.3), in case of XCTest's automatic discovery, afterSuite are called.

@testable import QuickSample

class QuickSampleTests: QuickSpec {
    override func spec(){
        afterSuite { 
            print("afterSuite is called")  <- executed
        }

        it("sample test") { 
            XCTAssertTrue(true)
        }
    }
}

class QuickSample2Tests: QuickSpec {
    override func spec(){
        afterSuite {
            print("afterSuite2 is called")  <- executed
        }

        it("sample2 test") {
            XCTAssertTrue(true)
        }
    }
}

However, in case of containing own TestSuites, I checked value of numberOfExamplesRun in Example.swift while running our test suite. numberOfExamplesRun is not be set to '0' after completion of each TestSuite.

So if I set isRunningAdditionalSuites to No, afterSuite will be called every time at some point in our test suites. numberOfExamplesRun will greater than world.includedExampleCount at some point.

if !world.isRunningAdditionalSuites && numberOfExamplesRun >= world.includedExampleCount {
            world.suiteHooks.executeAfters() 
}

Expected behavior of numberOfExamplesRun is set to 0 after running of each TestSuite, is my understanding correct?

Think about some ways to verify afterSuite is called as part of Quick's CI? Is it possible?

Sure. As you guess, current test code just verifies that afterSuite is not called while running Example('it'). I will find how to count number of examples in TestSuite and where to set numberOfExamplesRun to 0.

[modocache]

@takecian Yup, everything in your comment sounds correct to me! Good luck trying to find a way to test afterSuite in our own test suite!

As for the original issue, that afterSuite is not called in our test suite -- that's correct, and that's how I wrote this to work. Maybe the title of this issue should be updated to "Add a test to Quick's test suite that verifies afterSuite is called".

[takecian]

@modocache okay, i'll update title and find a way to test afterSuite.

[noisypigeon]

Closing since there is a PR pending review with this implemented. I'll try to get around to reviewing this ASAP @takecian! I replied on the PR as well. :)