Quick: Tests not executing in Xcode 7 beta 6

[zbeckman]

We are seeing a very reproducible problem (not 100%, I would estimate around 50-70%) where Xcode 7 beta 6 fails to detect and execute tests. Instead, it seems that the test spec is detected, but none of the tests defined within the spec (e.g., with context and it, etc.) are actually run.

The result is that the test "runs," and is "successful," but it never actually executes any tests... so we are getting a false positive. Here's a screenshot that demonstrates the problem:

It seems to _temporarily_ reset if we clean the build directory and clean derived data... but, the problem usually returns as soon as we execute the tests more than once, or if we change the test (and require a recompile). Based on these observations, I'll say the problem crops up every time we compile a test... and only stays away if we get it working, and don't touch the compiled tests. Just guessing at this point, but that seems to be the behavior.

[jeffh]

Hey @zbeckman,

It seems like that Quick can't generate the test method for XCTest to run.

Is this when you're running the entire test suite or attempt to run an individual tests class (eg - QuickSpec)?

Out of curiosity, how long is the full path to that source file?

[zbeckman]

This occurs both when trying to run individual tests, and when trying to run the entire suite. I've also seen a situation where some tests show up, but seem to be missing the "status" button (the play/red/green button to the right of the test). In these cases, those individual tests seem to be unusable, but other tests are usable. Let me know if you would like a screenshot.

Most of the time a clean / clean project folder / delete derived data fixes the issues and I can execute all tests. Then, it depends on how much changes how quickly – the more I change, the more quickly it devolves. Then repeat to get back to a usable state.

The full path for that file is:

/Users/zbeckman/Projects/Glimpulse/iOS/project/Glimpulse/GlimpulseTests/<file>

As of yesterday, I cleaned out all "old style" non-Quick tests. So now we have fewer tests, and none of them are XCTest cases. I'm not sure if it is coincidence but it seems like the system has been behaving a bit better since them. Might just be because 20 test classes vanished...

[alexjg]

I'm experiencing this issue as well, in my case the first run of the tests after xcode starts actually runs the tests but any test run thereafter does not. Cleaning the project or deleting derived date makes no difference, the only way to get the tests to run again is to restart xcode.

[adsantos]

I'm also having this problem in XCode 7.0 using Quick 0.6.0.

[adsantos]

If I build the project and run the entire suite, then it works fine. Running individual tests or not doing build before running the entire suite results in the tests not running.

[kos9kus]

I faced with the bug, when updated xCode 7.0
The bug makes the Quick almostly unusable. Please could you fix that?

[modocache]

Absolutely, this is the number one item on my to-do list. Hang tight, and sorry for the frustration.

[jonah-williams]

I see a help wanted label here and my current project could certainly use this. What are our next steps here? Do we understand the problem yet or is it still unclear why this frequently fails?

[modocache]

Using Quick v0.6.0 and the official release of Xcode 7, Quick's own test suite, which includes tests written using QuickSpec, produces correct test output--all tests pass, and each one has a test method name.

If I build the project and run the entire suite, then it works fine. Running individual tests or not doing build before running the entire suite results in the tests not running.

Thanks, @adsantos! Running an individual test was the only way I was able to reproduce this problem. It also appears this was originally reported in #363. It looks like Xcode 7 runs individual tests a little differently than previous versions.

There also seem to be reports in this thread that indicate the tests don't run even when running the entire suite, but I'm not able to reproduce that bug. I'll start by attempting to tackle the bug with running individual tests.

What are our next steps here?

Thanks, @jonah-williams! It would be great to have either:

1. A fix for the individual test issue I describe above
2. A clear way to reproduce the issues people describe with running the entire test suite

[modocache]

I believe this may have been fixed by #388, which calls +[XCTestCase testInvocations] explicitly. It appears Xcode 7 no longer calls that method when running individual tests.

Could someone on this thread confirm that Xcode 7 works fine with the current HEAD of the master branch of Quick, fdedd1e959? It's looking pretty good for me.

[adsantos]

@modocache I tested the current HEAD in XCode 7 and I still find problems. Here is what I've done:

• Step1: Run individual tests. Result: the tests didn't run.
• Step2: Run the entire suite. Result: all OK.
• Step3: Run the entire suite again. Result: all OK.
• Step4: Run individual tests. Result: all OK.
• Step5: Run individual tests again. Result: One less test was run than in the previous step.

If I continue running the individual tests, I always get x-1 tests run, compared to the previous run.

If I make changes to the tests, I have to run the entire suite again, or otherwise running individual tests won't work.

[modocache]

@adsantos Thanks for checking!

• Step1: Run individual tests. Result: the tests didn't run.

I believe this may be related to #219. Unfortunately, Quick just doesn't support this at the moment--before you can run individual tests via Xcode, you must run the entire test suite once.

Reminder: If you want to run individual tests, the recommended way to do so is by using Quick's focused tests, via fit, fcontext, or fdescribe.

Let me know if I've misunderstood this step.

• Step4: Run individual tests. Result: all OK.
• Step5: Run individual tests again. Result: One less test was run than in the previous step.

Can you explain exactly what kind of "individual test" you're running, and how? Are you selecting and running a single test _method_ in the Xcode test navigator? Or a single test _class_? Are you running the same test again by using Xcode > Product > Perform Action > Test Again ^⌥⌘G? Or are you selecting the same test in the test navigator again?

[adsantos]

@modocache In step1, step4 and step5, I run all the tests of a specific class by clicking on the checkmark on the left of class MyClass: QuickSpec. In step2 and step3, I run all the tests in all classes via XCode > Product > Test.

[modocache]

@adsantos Gotcha. Last night as I investigated I was right-clicking and selecting "Test 'test_method_name'" from the contextual menu. If I recall, the checkmarks didn't always work... I'll look into this some more."

If you don't mind me asking, is there a reason you'd like to use the buttons in Xcode, instead of using fit?

[adsantos]

@modocache I'm working on an existing project which has unit tests written with XCTest. I added another class which uses Quick. I frequently only want to run the tests I wrote in that class and not the old XCTests. The possible ways to run the tests on a specific class that I'm aware of are the following:

• Click on the checkmark on the left of class MyClass: QuickSpec
• Xcode > Product > Perform Action > Run "MyTests"
• In the Test Navigator, right click on MyTests and click Run "MyTests"

In all these approaches, I get one less test run than in the previous run. Is there another way of running the tests of a specific class?

I could also remove the other test files from the target and this way only execute my tests when running all the tests, but I don't want to do that.

[modocache]

Got it. Still investigating.

I frequently only want to run the tests I wrote in that class and not the old XCTests.

I usually write my tests like so:

class MyClassSpec: QuickSpec {
  override func spec() {
    describe("MyClass") {
      // ...
    }
  }
}

And to run the entire set of expectations in this spec, I prepend an f to the top-level describe closure.

Do things work any differently for you if, instead of clicking on the checkmark in the gutter, you select the test class from the test navigator, right-click, and click "Run 'MyClassSpec'", like in the GIF below?

In any case, I'm still looking into this. Sorry again!

[adsantos]

@modocache If I use fdescribe and run all the tests, the other tests written with XCTest are still run, rather than only running the quick tests within the fdescribe block.

If I select the test class from the test navigator, right-click and click "Run 'My Class Spec'", I get the same problem described before: the first time, x tests are run; the second time, x-1 tests are run, and so on - always one less test is run.

[jkhustler]

Seeing this also on Xcode 7.0.1. The tests for a spec are built and run correctly the first time when I right click the test target and run it. The second time, I just see this under my specs:

unspecifiedMethod_-[(THE_NAME_OF_THE_SPEC) (null)]()

Please let us know if you want us to run and report results or if there is anything we can do to help.

[fnazarios]

Same error above. My tests run correctly first time, but second just see this: unspecifiedMethod_-[(THE_NAME_OF_THE_SPEC) (null)]()

:(

[eimantas]

I have just witnessed this issue on Xcode 7.1 with Quick 0.8.0.

Any update on this issue?

[trimolesi]

I have the same error, the tests run correctly first time, after this it always show "Test succeeded", but not run nothing.
Xcode Version 7.1 (7B91b)
pod 'Quick', '~> 0.8.0'
pod 'Nimble', '3.0.0'

[jeffh]

I'd love a sample project that exhibits this issue with specific directions of steps to take to reproduce it.

[wm-j-ray]

Same issues here, but intermittently. For the most part, tests in a quickSpec do not run dependably.

I can clean project, delete derive's, shut down and restart Xcode and tests will run but then seem to stop. Sometimes all the tests will run, then other times a only a subset will run.

Sorry I can't be more specific. For what it's worth, I'm using Carthage. Here's a pure vanilla subset. Exact same configuration as trimolesi but under Carthage.

When and if first test runs and fails as it should. 2nd test (naming) doesn't run:

describe("#createProject") {
            beforeEach {
            }

            it("will allow you to create a project") {
                // given
                // when
                // then
                expect(1.2).to(beCloseTo(1.1, within: 0.1))
                expect(3) > 5
                expect("seahorse").to(contain("sea"))
                expect(["Atlantic", "Pacific"]).toNot(contain("Mississippi"))
            }



            it("will allow you to name a project") {
                // given
                // when
                // then
                expect(1.2).to(beCloseTo(1.1, within: 0.1))
                expect(3) > 5
                expect("seahorse").to(contain("sea"))
                expect(["Atlantic", "Pacific"]).to(contain("Mississippi"))
            }


        }

[kos9kus]

I'm experiencing the same glitch, the first time running is ok and that's it, then next tests are always succeeded. A solution to the problem is re-open a workspace completely.

[brittonmobile]

In case this helps anyone else, I was unable to get the tests to run when an actual device was set as the target. Switching it to a simulator did work.

[wm-j-ray]

Either device or simulator doesn't change or help. I have tried this with App Store version of Xcode 7.1, Xcode 7.2b2 and now today with 7.2b3. The issue as I can see is the same. Tests are not running.

I thought it could be related to nesting describes but apparently not. I am completely perplexed and can't really determine a pattern for debugging. I wish I could be more help. Related to adsanto's comments, I didn't see any n-1 pattern (but maybe my test harness is not big enough yet). I also so no real difference if I ran 'all tests', started at the QuickSpec class level, or at the individual test.

Anybody have any ideas?

Here is log output from the above stub from previous. I ran all project tests (a mixture of xctest, quick and nimble. Note that only one of the tests from the first it block runs. This is with a clean install of AppStore version of xCode 7.1.1 7B1005)

Test Suite 'ProjectSpec' started at 2015-11-12 18:26:03.861
Test Case '-[WTDataStoreTests.ProjectSpec         Project_MetaData___createProject__will_allow_you_to_create_a_project_UserswmjrayDropboxNiteWor   kWTDataStoreWTDataStoreTestsModelsCoreDataProjectSpecswift_35]' started.
Test Case '-[WTDataStoreTests.ProjectSpec   Project_MetaData___createProject__will_allow_you_to_create_a_project_UserswmjrayDropboxNiteWor  kWTDataStoreWTDataStoreTestsModelsCoreDataProjectSpecswift_35]' passed (0.162 seconds).
Test Suite 'ProjectSpec' passed at 2015-11-12 18:26:04.024.
 Executed 1 test, with 0 failures (0 unexpected) in 0.162 (0.163) seconds
Test Suite 'SetUpCoreDataTests' started at 2015-11-12 18:26:04.025

[dspaces1]

I just started using Quick for swift and got the same issue on both Xcode Version 7.2 (7C68) and Version 7.3 beta (7D111g). My first individual test fail properly the first time and succeed every time after unless I either restart my workspace or run the entire suite.

An updates on this issue?

[modocache]

@dspaces1 Yes, thanks to @sharplet and #452, I believe this issue has been fixed. I also explain in #439:

Keep in mind what has been fixed: all tests should now be run, every time. However, not all tests are always displayed in the Xcode UI.

Sorry for the wait everyone! All tests should always run now, but if that's not the case please open another issue or comment on this one.

The fix is available in master. I'll release a new version of Quick soon so that you can include it using your favorite package manager.

[amjad-Alhusseini]

Hi,

Im trying to run my quick spec tests on XCode 7.3 whether individually or through test entire suite with no luck, though I'm using the latest quick version 0.9.1.

Any ideas?

[slashrover]

@amjad-Alhusseini Hi, Have you sorted it out ?

[amjad-Alhusseini]

@slashrover Yes, i need to add build phase to copy framework files, this sorted it out.
It used to work without copying these files in previous Xcode versions.

[tom-sparo]

@amjad-Alhusseini can you be more specific about what you're doing to get this to work?

[amjad-Alhusseini]

@tom-sparo Sure, So when i implemented quick framework on my project (which uses Carthage as my dependency manager), i didn't use to copy the framework files as a build phase as specified in step 5 as shown in this link.
Hence, After upgrading to XCode 7.3 for some reason it doesn't run these tests, so i revised the steps again and it turned out that i was missing this step.

[miketihonchik]

I am still experiencing the issue using XCode Version 7.3.1 (7D1014). The tests run fine first time after opening project, however, every time after that I get following results:

Test Suite 'Selected tests' started at 2016-06-23 11:16:14.714 Test Suite 'Selected tests' passed at 2016-06-23 11:16:14.716. Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds

The only way to force tests to run again, is to restart workspace.

[briankeane]

Also experiencing... Xcode 7.3 (7D175).

Test Suite 'Selected tests' started at 2016-06-26 10:17:49.950
Test Suite 'Quick.framework' started at 2016-06-26 10:17:49.951
Test Suite 'Quick.framework' passed at 2016-06-26 10:17:49.952.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'playolaIphoneTests.xctest' started at 2016-06-26 10:17:49.952
Test Suite 'playolaIphoneTests.xctest' passed at 2016-06-26 10:17:49.953.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2016-06-26 10:17:49.953.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.003) seconds

[Dan-Andoni-BJSS]

Xcode 7.3.1 The same for me

Test Suite 'All tests' started at 2016-06-28 15:30:38.394
Test Suite 'Quick.framework' started at 2016-06-28 15:30:38.395
Test Suite 'Quick.framework' passed at 2016-06-28 15:30:38.396.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'XConnectSDKTests.xctest' started at 2016-06-28 15:30:38.397
Test Suite 'XConnectSDKTests.xctest' passed at 2016-06-28 15:30:38.397.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'All tests' passed at 2016-06-28 15:30:38.398.
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.005) seconds

[briankeane]

Not sure if this helps anyone else out, but it turns out I had one test with all of my expectations inside of a callback... correcting that problem not only fixed the single test but it also for some reason caused all my tests to run normally. Here's the before and after:

before:

it ("returns an updated user")
{
    auth.removeSpin("testSpinID")
    .then
    {
        user -> Void in
        expect(user.id).to(equal(DataMocker.users[0].id))  // test succeeds before this runs
    }
}

and after:

it ("returns an updated user")
{
    var completed:Bool = false
    auth.removeSpin("testSpinID")
    .then
    {
        user -> Void in
        expect(user.id).to(equal(DataMocker.users[0].id))
        completed = true
    }
    expect(completed).toEventually(equal(true))
}

[ravindrana]

XCode 8. Still facing this. In the Swift file, only 1 or 2 tests running, rest of them not considered as tests at all. any help ?

[noisypigeon]

@ravindrana: Please open a new issue with more specific details about your situation. 😸

[JeffESchmitz]

Same issue as @ravindrana with Xcode 8. Spec tests don't show up.

[noisypigeon]

Hi @JeffESchmitz. Thanks for reaching out. As I asked @ravindrana to do, I'd really appreciate it if you could file a new issue with more specific details about your situation, possible a project that I can use to replicate the problem and the exact Xcode version you use so I can attempt at replicating the conditions. Cheers!

[noisypigeon]

I'm going to lock this issue. Please create a new issue with updated examples and details about the problem if you are experience this again.