Quick: Xcode says it has run the tests but not even one

[shravankumarbj]

Hi,

Its strange that when I open Xcode fresh and run a test file whole unit tests in project will run with the file tests also. And if I focus or without focussing run a file again Xcode says it has run the test and passes, even though last run had failed test and it is still failing Xcode says it has passed, but it doesn't reach any break point in the tests written. I am using Xcode 7.3.1. I am using package managers but not with respect to any testing.

I really like Quick and nimble which helps me to write BDD tests but now I am in a situation that I am not able to fix the previous failing tests due to the respective main code change.

Can any one help me on these

[shravankumarbj]

Any help on these.??? Or any link where I can rectify this????

[modocache]

Oops, sorry for the slow response! Just to clarify: are you running the entire test suite (⌘U)? Or just a single test file via the Xcode test navigator?

Both should work, but I have seen reports of single test execution causing trouble in the past -- see #373. I would recommend always running the entire suite. To run only one test, use fit.

If you're running the entire suite, but still experiencing problems, could you provide more detail on your project? A sample project that demonstrates the problem would be best. As is, the setup you describe works for me, so I'm not sure how to help.

[dtrenz]

Same issue here. If I can't get it sorted, I will try to get an example project up to help reproduce.

[dtrenz]

@modocache — Example project that reproduces the issue for me: https://github.com/dtrenz/QuickExample

[dtrenz]

OK, this is getting strange. It's working this morning. I haven't restarted my machine, or even Xcode. It just started working again. Huh.

[modocache]

If you can narrow it down, that would be very much appreciated. I have a feeling this is going to come down to some very interesting Apple Radars.

[shravankumarbj]

Its happening same to me. I cannot create example project becoz of time constraint. If I restart XCode it will test it otherwise it says all tests passed but it will not even one test passed or it will hit any break point. If I run one QuickSpec file it runs all unit tests entire suite. I am pretty sure its Xcode problem but not able to identify whats the problem.

[acecilia]

Hi,
I am experiencing similar problems in my project. I am completely new to Quick and Nimble (been using it for 40 minutes) so I am using a really simple example. Using Carthage with the copy phase of the frameworks properly set. Xcode version 7.3.1 (7D1014). The problem:

After launching xCode I go Product->Test and this is the result: (GOOD)

02:43:39.469 xctest[3303:53598] _XCT_testBundleReadyWithProtocolVersion:minimumVersion: reply received
02:43:39.486 xctest[3303:53598] _IDE_startExecutingTestPlanWithProtocolVersion:16
Test Suite 'All tests' started at 2016-07-13 02:43:39.501
Test Suite 'Quick.framework' started at 2016-07-13 02:43:39.504
Test Suite 'Quick.framework' passed at 2016-07-13 02:43:39.507.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.003) seconds
Test Suite 'RxSwiftAddons-Test.xctest' started at 2016-07-13 02:43:39.509
Test Suite 'TwoWayBindingSpec' started at 2016-07-13 02:43:39.511
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_7]' started.
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_7]' passed (0.003 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:28: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0] : expected to equal <0>, got <7>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0]' failed (0.003 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:34: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7] : expected to equal <7>, got <3>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7]' failed (0.002 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:38: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0] : expected to equal <0>, got <3>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0]' failed (0.013 seconds).
Test Suite 'TwoWayBindingSpec' failed at 2016-07-13 02:43:39.537.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.020 (0.026) seconds
Test Suite 'RxSwiftAddons-Test.xctest' failed at 2016-07-13 02:43:39.538.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.020 (0.029) seconds
Test Suite 'All tests' failed at 2016-07-13 02:43:39.539.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.020 (0.039) seconds

Test session log:
    /Users/andres/Library/Developer/Xcode/DerivedData/RxSwiftAddons-beauhyeinceyftdqstzhjycznril/Logs/Test/F4954D43-557B-40F8-820C-FDB5FB472F0D/Session-2016-07-13_02:43:33-81oCyZ.log

Program ended with exit code: 1

Then, without closing xCode and without touching anything (really, ANYTHING) I do again Product->Test and I got this: (BAD, VERY BAD)

02:43:59.638 xctest[3356:54280] _XCT_testBundleReadyWithProtocolVersion:minimumVersion: reply received
02:43:59.652 xctest[3356:54280] _IDE_startExecutingTestPlanWithProtocolVersion:16
Test Suite 'All tests' started at 2016-07-13 02:43:59.662
Test Suite 'Quick.framework' started at 2016-07-13 02:43:59.663
Test Suite 'Quick.framework' passed at 2016-07-13 02:43:59.664.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Test Suite 'RxSwiftAddons-Test.xctest' started at 2016-07-13 02:43:59.665
Test Suite 'TwoWayBindingSpec' started at 2016-07-13 02:43:59.666
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_7]' started.
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_7]' passed (0.002 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:28: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0] : expected to equal <0>, got <7>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_7__Is_0]' failed (0.002 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:34: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7] : expected to equal <7>, got <3>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_7]' failed (0.006 seconds).
Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0]' started.
/Users/andres/Desktop/RxSwiftAddons/Tests/Common/TwoWayBinding/TwoWayBinding-Tests.swift:38: error: -[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0] : expected to equal <0>, got <3>

Test Case '-[RxSwiftAddons_Test.TwoWayBindingSpec The_number__Is_set_to_3__Is_0]' failed (0.003 seconds).
Test Suite 'TwoWayBindingSpec' failed at 2016-07-13 02:43:59.685.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.014 (0.019) seconds
Test Suite 'RxSwiftAddons-Test.xctest' failed at 2016-07-13 02:43:59.686.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.014 (0.021) seconds
Test Suite 'All tests' failed at 2016-07-13 02:43:59.688.
     Executed 4 tests, with 3 failures (0 unexpected) in 0.014 (0.026) seconds

Test session log:
    /Users/andres/Library/Developer/Xcode/DerivedData/RxSwiftAddons-beauhyeinceyftdqstzhjycznril/Logs/Test/E2B5B5D9-B1FA-4478-BCC8-4BAC37C30180/Session-2016-07-13_02:43:56-h7VMy4.log

Program ended with exit code: 1

I hope this information helps you with the issue!

PD: (what a long comment, sorry for this ><, but maybe it is helpful) Full code:

import Quick
import Nimble

class TwoWayBindingSpec: QuickSpec {
    override func spec() {
        var number = 0

        beforeEach {
            number = 1
        }

        describe("The number") {
            context("Is set to 7") {
                it("Is 7") {
                    number = 7
                    expect(number) == 7
                }
                it("Is 0") {
                    number = 7
                    expect(number) == 0
                }
            }
            context("Is set to 3") {
                it("Is 7") {
                    number = 3
                    expect(number) == 7
                }
                it("Is 0") {
                    number = 3
                    expect(number) == 0
                }
            }
        }
    }
}

[SimonKaeseler]

Any news on this?

[MP0w]

if you have troubles running a single test or a single test file you might check this out #557
Can you guys ( @shravankumarbj @SimonKaeseler ?) check if your test target name contains a space?
@acecilia looks like the second time all tests (4) were performed but one is not shown by Xcode. This is a known issue (Xcode issue) #427 (BTW since your target has a space in the name you might also be affected by #557 )

[SimonKaeseler]

@MP0w Ive removed the spaces (actually created a new project without spaces) and it seems to be working now - how ever there is an inconsistency between he number of tests listed in the test navigator and the number of executed tests in the output. BUT, it seems that it runs all tests

[MP0w]

@simonkaeseler ty, good to know. Then I'm pretty sure the issue it's only happening for target containing a space. I'm working on a PR that uses the principalClass of the test bundle to get the module name...

[SimonKaeseler]

FYI: If i run all tests it doesn't seem to find the ones that are under some of the subtest classes. If i, however, run them individually it can find them. If you need more specific information about the tests them selves let me know.

EDIT: After commenting the autogenerated testfile, the "test all" feature does not find any of the Quick tests. The tests can be run by manually testing the class

[shravankumarbj]

Actually this issue popped up because I had two Xcodes. When I deleted beta
version then I had no issues. but still I cannot run individual tests I can
only run complete tests. But this works not blocking thats the releif

On Sat, Oct 29, 2016 at 1:50 AM, Willow Alexandra [email protected]
wrote:

Any news on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Quick/Quick/issues/534#issuecomment-257018556, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAjaIkMFdpV7dsCyVlEFer-mYolClJZcks5q4lkNgaJpZM4IjHs2
.

[lnanek]

Seeing the same problem in Xcode 8.1 with these versions:

    pod 'Quick', '~> 0.10.0'
    pod 'Nimble', '~> 5.1.1'

The tests in the QuickSpec subclass only get run when all tests are run. If I run just the file or the tests in the file in Xcode, nothing is run. fit not feasible since there are 822 tests in this project that takes minutes to run and not all are QuickSpec.

[srikanth-vm]

@MP0w Is this fix part of Quick 0.9.3 as well, as thats the version supported for Swift 2.3

[tbaranes]

Quick 1.0 fixed this for me 🎉

[MP0w]

@srikanth-vm I didn't do it for swift 2.3 not sure if someone else did. You can check #638 and port it or I can also do it ( probably not today)

[paulfreeman]

I'm getting this problem with Xcode 8.2 beta, Swift 3.0, Quick v1.0.0, Nimble v5.1.1 installed by Carthage from Github. The problem is very inconsistent. Sometimes a restart fixes it, then it goes bad again.

I had about 20 XCTests in 4 files before setting up Quick and Nimble. Now I have an additional 26 Quick tests in 4 files as well.

What I am seeing (note I am running testing with debugging set to on and off, I've tried both to see if it makes a difference).

Running ALL tests works consistently (but is too slow for my needs)

Running one test by selecting the run icon from the Test Navigator does not work often. When it does work it appears to causes ALL the QuickSpec tests to run, but not other XCTests in the test target

Running one test by selecting the run icon on the first line of the QuickSpec class works occasionally but soon 'goes bad' and stops running and I get 'Executed 0 tests' for the suite.

Note that a restart of Xcode and simulator occasionally fixes this but not consistently. Sometimes the single test run from the QuickSpec file just starts working again with no apparent intervention on my part.

I have also tried to remove all the XCTests from the target just leaving the QuickSpec's but this made no difference, still getting the 'Executed 0 tests' from both running in the test navigator or from the run icon in the file.

[hirad]

@paulfreeman This may not be an issue with Quick necessarily. I just added a UI test target to my project, and so far I haven't added Quick and Nimble to it - just plain XCTests. And I can't run just one test case. Using Xcode's UI for running one test case, always runs all test cases. (This is in Xcode 8.2 beta, Swift 2.3).

[paulfreeman]

@hirad My comments concern Unit Testing only not UI Testing,

To exclude possibly incorrect logic in thinking about this...

Test 1

I created a test project in XCode 8.2 with just two XCTests in it using Swift 3.0 as the language.

I can test either XCTest separately or together via the test navigator. This would seem to remove Xcode 8.2 itself as the sole culprit.

Test 2
But... belt and braces as there is doubt here, maybe it could be something else in my project (which is moderately large and complex) so I created a branch of my project and DELETED Quick and Nimble from my Unit Test target. I removed the QuickSpec tests from the test target so it would build leaving only the XCTests

Now everything works correctly again, I can run all the XCTests as a set, as individual suites or individually either from the button in the test navigator, from the contextual menu or by clicking the button in the test file. I can disable parts of suites and run the suites with fewer tests, re-enable them etc. etc.

Test 3

So I checked out my development branch again to reinstate the install of Quick and Nimble. Then I removed all the Quickspec tests from the test target. Trashed derivedData directory. Rebuilt test target. Now just running the XCTests, the problem is still there.

Test 4

Leaving everything else as is, just remove the two Quick/Nimble from build dependencies. XCTests now all work again.

[MP0w]

@paulfreeman do you have a project to replicate the issue? Can you set some breakpoints to check what's going on?
For me was just the space in the target name (fixed now) and was really easy to debug.
BTW using fit/fdescribe is just better but running 0 tests should probably warn/fail because someone might not notice it

[paulfreeman]

@MPOw My test target is a bit boring and has a simple name so that isn't the issue.

Thanks for the tip on fit and fdescribe, as a workaround I've put the Quick tests into a different test target so I can run them as a batch but keeping my XCTests in a separate target so I can run them with the full test navigator capabilities.

I can't show the main project, its client confidential, but my small test project shows at least one of the problems. Initially I created this just to check Xcode 8.2 was able to execute XCTests in all methods from the navigator.

I have now added Quick and Nimble and a single QuickSpec with two minimalistic tests in it.

Following this addition I can no longer run a single XCTest, only an entire suite (file). It's not quite the 0 tests issue, but it shows that something is not quite alright, as the functionality of the test navigator has changed.

trashtesty.zip

[MP0w]

I checked @benediktreichert issues and is basically the same problem I had with space in the bundle name but for -
If you have a dash that's probably the issue.

See https://github.com/Quick/Quick/issues/651#issuecomment-264723986

[MP0w]

As @paulfreeman mention "Following this addition I can no longer run a single XCTest, only an entire suite (file). It's not quite the 0 tests issue, but it shows that something is not quite alright, as the functionality of the test navigator has changed."

this is another issue I see, Quick run the whole suite/file and not a single test this is another issue. @paulfreeman Can you open it?

[paulfreeman]

@MPOw Thanks, yes it could be a '-' causing the problem in my case also. There was I being too literal minded about the ' ' problem, I should have realised.

I will open a new issue for the other problem.

[iby]

-?! Nonononono… I refuse to believe.

[iby]

After calming myself down… here a quick and easy fix – use identifier transformation with PRODUCT_NAME, i.e., $(TARGET_NAME:identifier). This will transform spaces and other wonderful characters into underscores, which makes Xcode and AppCode happy about running selected tests.

[jeffh]

This should also be fixed on the latest version of Quick IIRC

On Feb 18, 2017, 11:54 AM -0800, Ian Bytchek [e-mail], wrote:
>

After calming myself down… here a quick and easy fix – use identifier transformation with PRODUCT_NAME, i.e., $(TARGET_NAME:identifier). This will transform spaces and other wonderful characters into underscores, which makes Xcode and AppCode happy about running selected tests.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (https://github.com/Quick/Quick/issues/534#issuecomment-280870496), or mute the thread (https://github.com/notifications/unsubscribe-auth/AAEMCHUBXiQz6Jv_I_0NoPzIYJNB2sHMks5rd0x5gaJpZM4IjHs2).

[iby]

That's what I thought from the above comments, but the issue remained after update and clean build. It's been following me forever and I'm 95% certain it's Xcode related, not just Quick. No?

[jeffh]

Oh right, I forgot about that case again, never mind.

• Jeff

On Feb 18, 2017, 12:04 PM -0800, Ian Bytchek [e-mail], wrote:

That's what I thought from the above comments, but the issue remained after update and clean build. It's been following me forever and I'm 95% certain it's Xcode related, not just Quick. No?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[shaps80]

Not sure if this is relevant in this case -- but perhaps could help towards discovering the underlying issue.

I recently started up a project and noticed my tests don't execute when they are not inside an it() statement. I didn't test whether describe() or context() made any impact. But I had simply forgotten to use them and instead tried to just add a single expect() statement. Tests didn't run.

Perhaps that's by design though?

[jeffh]

Hey Shaps,

Could you elaborate more on that particular test case? Were you using the xctest-style of writing tests or did you put the expectation inside the spec() call?

Sent from my iPhone

On Feb 26, 2017, at 2:56 PM, Shaps [e-mail] wrote:

Not sure if this is relevant in this case -- but perhaps could help towards discovering the underlying issue.

I recently started up a project and noticed my tests don't execute when they are not inside an it() statement. I didn't test whether describe() or context() made any impact. But I had simply forgotten to use them and instead tried to just add a single expect() statement. Tests didn't run.

Perhaps that's by design though?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[shaps80]

Here's a sample project that shows what I did and how Xcode doesn't pickup the test. However, this is obviously NOT how you'd actually write the test.

Nimble-Test.zip

[shaps80]

Ok, so that last project didn't have an actual iOS app inside which seems to make a difference. If you run the tests in this project you'll see that nothing really happens even though the test should fail.

Test.zip

[ikesyo]

Closing this as part of issue cleanup.