Quick: Tests partially appearing in test inspector or not at all

[mutech]

Test inspector shows only some tests (in this case here 4/5 but sometimes it only shows a single test with unspecifiedMethod_(null) as test name, sometimes 3/5 and other combinations). I think it's a coincidence that it does not show the failed test only, right before it did not show a succeeded test, can't reproduce it anymore.

I cleared derived data, rebuild in various combinations. Results are fuzzy.

Using Xcode 7.1, the framework I'm testing is an Objective-C framework.

Here the test inspector view:

The corresponding test results

04:51:56.174 xctest[83634:11803643] _XCT_testBundleReadyWithProtocolVersion:minimumVersion: reply received
04:51:56.176 xctest[83634:11803643] _IDE_startExecutingTestPlanWithProtocolVersion:16
Test Suite 'All tests' started at 2015-11-22 04:51:56.218
Test Suite 'AKABeaconQuickTests.xctest' started at 2015-11-22 04:51:56.219
Test Suite 'UITextViewBindingsSpec' started at 2015-11-22 04:51:56.220
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__does_not_change_its_initial_text_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_68]' started.
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__does_not_change_its_initial_text_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_68]' passed (0.009 seconds).
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__updates_text_to_bound_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_76]' started.
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__updates_text_to_bound_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_76]' passed (0.000 seconds).
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_model_changes__updates_text_to_new_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_85]' started.
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_model_changes__updates_text_to_new_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_85]' passed (0.046 seconds).
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_view_changes__binding_context__updates_model_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_97]' started.
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_view_changes__binding_context__updates_model_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_97]' passed (0.001 seconds).
Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_no_longer_observing_changes__reverted_text_to_initial_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_106]' started.
/Users/mu/git/aka-ios-beacon/AKABeacon/AKABeaconQuickTests/UITextViewBindingsSpec.swift:107: error: -[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_no_longer_observing_changes__reverted_text_to_initial_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_106] : failed - expected to equal <Initial text>, got <new view Value>

Test Case '-[AKABeaconQuickTests.UITextViewBindingsSpec UITextView__textBinding__when_bound__when_observing_changes__when_no_longer_observing_changes__reverted_text_to_initial_value_UsersmugitakaiosbeaconAKABeaconAKABeaconQuickTestsUITextViewBindingsSpecswift_106]' failed (0.025 seconds).
Test Suite 'UITextViewBindingsSpec' failed at 2015-11-22 04:51:56.304.
     Executed 5 tests, with 1 failure (0 unexpected) in 0.081 (0.084) seconds
Test Suite 'AKABeaconQuickTests.xctest' failed at 2015-11-22 04:51:56.305.
     Executed 5 tests, with 1 failure (0 unexpected) in 0.081 (0.085) seconds
Test Suite 'Quick.framework' started at 2015-11-22 04:51:56.355
Test Suite 'Quick.framework' passed at 2015-11-22 04:51:56.356.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'All tests' failed at 2015-11-22 04:51:56.356.
     Executed 5 tests, with 1 failure (0 unexpected) in 0.081 (0.138) seconds


Test session log:
    /var/folders/75/z598xsqx299616b9hxbwg1jc0000gn/T/com.apple.dt.XCTest-status/Session-2015-11-22_04:51:52-XKfIi5.log

Program ended with exit code: 1

I'm also attaching the test class, just in case I made a stupid obvious mistake:

import Quick
import Nimble
import AKABeacon

class UITextViewBindingsSpec: QuickSpec {

    @objc
    class TestModel: NSObject, AKABindingContextProtocol {
        dynamic var text: String

        override init() {
            text = "Default text";
        }

        init(text: String) {
            self.text = text
        }

        func dataContextPropertyForKeyPath(keyPath: String?, withChangeObserver valueDidChange: AKAPropertyChangeObserver?) -> AKAProperty? {
            return AKAProperty.init(ofWeakKeyValueTarget: self, keyPath: keyPath, changeObserver: valueDidChange);
        }

        func dataContextValueForKeyPath(keyPath: String) -> AnyObject? {
            return dataContextPropertyForKeyPath(keyPath, withChangeObserver: nil)?.value;
        }

        func rootDataContextPropertyForKeyPath(keyPath: String?, withChangeObserver valueDidChange: AKAPropertyChangeObserver?) -> AKAProperty? {
            return self.dataContextPropertyForKeyPath(keyPath, withChangeObserver: valueDidChange);
        }

        func rootDataContextValueForKeyPath(keyPath: String) -> AnyObject? {
            return rootDataContextPropertyForKeyPath(keyPath, withChangeObserver: nil)?.value;
        }

        func controlPropertyForKeyPath(keyPath: String, withChangeObserver valueDidChange: AKAPropertyChangeObserver?) -> AKAProperty? {
            return nil;
        }

        func controlValueForKeyPath(keyPath: String) -> AnyObject? {
            return nil;
        }
    }

    override func spec() {
        describe("UITextView") {
            let initialText = "Initial text";
            let textView = UITextView(frame: CGRect());
            textView.text = initialText;

            context("textBinding") {
                let property = Selector("textBinding_aka")
                textView.textBinding_aka = "text"
                let expression: AKABindingExpression! = textView.aka_bindingExpressionForProperty(property)!

                context("when bound") {
                    let bindingContext = TestModel();
                    let initialModelValue = bindingContext.text;

                    let binding = try! AKABinding_UITextView_textBinding(
                        target: textView,
                        property: property,
                        expression: expression,
                        context: bindingContext,
                        delegate: nil);

                    let textAfterBindingCreation = textView.text;

                    it("does not change its initial text") {
                        expect(textAfterBindingCreation).to(equal(initialText));
                    }

                    context("when observing changes") {
                        binding.startObservingChanges()
                        let textAfterStartObservingChanges = textView.text;

                        it("updates text to bound value") {
                            expect(textAfterStartObservingChanges).to(equal(initialModelValue))
                        }

                        context("when model changes") {
                            let changedModelValue = "new value"
                            bindingContext.text = changedModelValue
                            let textAfterModelChange = textView.text

                            it("updates text to new value") {
                                expect(textAfterModelChange).to(equal(changedModelValue))
                            }
                        }

                        context("when view changes") {
                            let changedViewValue = "new view Value"
                            textView.text = changedViewValue;
                            textView.delegate?.textViewDidChange!(textView);
                            let modelValueAfterViewChange = bindingContext.text;

                            describe("binding context") {
                                it("updates model value") {
                                    expect(modelValueAfterViewChange).to(equal(changedViewValue))
                                }
                            }
                        }

                        context("when no longer observing changes") {
                            binding.stopObservingChanges()
                            let textAfterStopObservingChanges = textView.text
                            it("reverted text to initial value") {
                                expect(textAfterStopObservingChanges).to(equal(initialText));
                            }
                        }
                    }
                }
            }
        }
    }
}

[mutech]

Here is the other case described above, where only a single test case is displayed with undefined name:

12:22:11.153 xctest[91812:12407866] _XCT_testBundleReadyWithProtocolVersion:minimumVersion: reply received
12:22:11.155 xctest[91812:12407866] _IDE_startExecutingTestPlanWithProtocolVersion:16
Test Suite 'Selected tests' started at 2015-11-22 12:22:11.470
Test Suite 'UILabelBindingsSpec' started at 2015-11-22 12:22:11.471
Test Case '-[AKABeaconQuickTests.UILabelBindingsSpec (null)]' started.
Test Case '-[AKABeaconQuickTests.UILabelBindingsSpec (null)]' passed (0.005 seconds).


Test session log:
    /var/folders/75/z598xsqx299616b9hxbwg1jc0000gn/T/com.apple.dt.XCTest-status/Session-2015-11-22_12:22:06-3tX3SC.log

Test Suite 'UILabelBindingsSpec' passed at 2015-11-22 12:22:11.478.
     Executed 1 test, with 0 failures (0 unexpected) in 0.005 (0.006) seconds
Test Suite 'Selected tests' passed at 2015-11-22 12:22:11.478.
     Executed 1 test, with 0 failures (0 unexpected) in 0.005 (0.008) seconds
Program ended with exit code: 0

Pasting the code for this one too (not sure if the behavior is related to the code at all, but well...):

import Quick
import Nimble
import AKABeacon

class UILabelBindingsSpec: QuickSpec {
    override func spec() {
        describe("UILabelView") {
            context("when bound to a text") {
                let initialText = "Initial text";
                let view = UILabel(frame: CGRect());
                view.text = initialText;

                let property = Selector("textBinding_aka")
                view.textBinding_aka = "textValue"
                let expression: AKABindingExpression! = view.aka_bindingExpressionForProperty(property)!

                let bindingContext = TestModel();
                let initialModelValue = bindingContext.textValue;

                let bindingType: AKABinding.Type = expression.bindingType as! AKABinding.Type;
                let binding = try! bindingType.init (
                    target: view,
                    property: property,
                    expression: expression,
                    context: bindingContext,
                    delegate: nil);

                let textAfterBindingCreation = view.text;

                it("does not change its initial text") {
                    expect(textAfterBindingCreation).to(equal(initialText));
                }

                context("when observing changes") {
                    binding.startObservingChanges()
                    let textAfterStartObservingChanges = view.text;

                    it("updates text to bound value") {
                        expect(textAfterStartObservingChanges).to(equal(initialModelValue))
                    }

                    context("when model changes") {
                        let changedModelValue = "new value"
                        bindingContext.textValue = changedModelValue
                        let textAfterModelChange = view.text

                        it("updates text to new value") {
                            expect(textAfterModelChange).to(equal(changedModelValue))
                        }
                    }

                    context("when no longer observing changes") {
                        binding.stopObservingChanges()
                        let textAfterStopObservingChanges = view.text
                        it("reverts text to initial value") {
                            expect(textAfterStopObservingChanges).to(equal(initialText));
                        }
                    }
                }
            }

            context("when bound to a date") {
                let initialText = "Initial text";
                let view = UILabel(frame: CGRect());
                view.text = initialText;

                let property = Selector("textBinding_aka")
                view.textBinding_aka = "dateValue { dateFormatter: dateFormatter }"
                let expression: AKABindingExpression! = view.aka_bindingExpressionForProperty(property)!

                let bindingContext = TestModel();
                let initialFormattedModelValue = bindingContext.dateFormatter.stringFromDate(bindingContext.dateValue);

                let bindingType: AKABinding.Type = expression.bindingType as! AKABinding.Type;
                let binding = try! bindingType.init (
                    target: view,
                    property: property,
                    expression: expression,
                    context: bindingContext,
                    delegate: nil);

                let textAfterBindingCreation = view.text;

                it("does not change its initial text") {
                    expect(textAfterBindingCreation).to(equal(initialText));
                }

                context("when observing changes") {
                    binding.startObservingChanges()
                    let textAfterStartObservingChanges = view.text;

                    it("updates text to bound value") {
                        expect(textAfterStartObservingChanges).to(equal(initialFormattedModelValue))
                    }

                    context("when model changes") {
                        let changedModelValue = NSDate()
                        bindingContext.dateValue = changedModelValue
                        let changedFormattedModelValue = bindingContext.dateFormatter.stringFromDate(changedModelValue)
                        let textAfterModelChange = view.text

                        it("updates text to new value") {
                            expect(textAfterModelChange).to(equal(changedFormattedModelValue))
                        }
                    }

                    context("when no longer observing changes") {
                        binding.stopObservingChanges()
                        let textAfterStopObservingChanges = view.text
                        it("reverts text to initial value") {
                            expect(textAfterStopObservingChanges).to(equal(initialText));
                        }
                    }
                }
            }
        }
    }
}

EDIT: After removing derived data, then clean and build for tests and run tests I get this:

Looks better, but there is still a test missing in UITextView...

[justinmakaila]

I'm currently experiencing something similar to this... None of my tests are showing up in the inspector, and when I run them, the immediately fail, without hitting any of the breakpoints I have set in spec().

Any advice?

[Gabriel-Lacatus]

I can also confirm some weird behaviour in this area. It's not a terrible issue, more like an annoyance really. What happens in my case is:

• if I have N if statements in a Spec class, after building the project I can only see the first N-1 (i.e. in the order they appear in code) in the Test Navigator
• when I start the tests, for a brief period while the tests are running, I can see all N tests listed in Test Navigator, then all N icons go green (i.e. tests pass)
• immediately after XCode finishes running the tests, the last test disappears from the Test Navigator and only the first N-1 remain there
• if I close XCode and reopen it then all tests are gone from Test Navigator; running the tests make them show up as they are being executed, then as soon as they finish the last one disappears again

[ReedD]

:+1: I have the same problem. Adding one extra dummy test ensures all of my real tests written above it run.

[modocache]

If you use Specta or Quick and are disappointed with tests disappearing in Xcode's Test Navigator, tests lingering in the Test Navigator despite no longer being defined, or even tests not being run, please read and duplicate this radar: https://openradar.appspot.com/26028557.

Xcode does not provide the Specta or Quick maintainers with an API to address these problems. The best way to contribute to fixing these issues is to communicate to the Apple XCTest team that this is a use case we wish they'd support better. The best way I know to do that is to ask you all to re-submit https://openradar.appspot.com/26028557. 🙇

[Jeehut]

I investigated this issue back in January (my conclusions are posted in #439) and setup an example project. Basically I can confirm the observations made by @Gabriel-Lacatus and found a workaround:

Simply add an it case that is last when all tests names were sorted alphabetically like this:

it("zzz"){} // workaround to prevent real tests from disappearing from navigator

I hope this helps and the issue will disappear sooner rather than later.

[modocache]

Thanks for the workaround, @Dschee! Please also make sure to file a radar if you haven't already--if Apple provided us with some additional tools, we would be able to avoid this issue without requiring an it("zzz").

[doozMen]

I have a way to reproduce this all the time. If you build Quick and Nimble using Carthage this behaviour occurs. Not tests. But if you build it using Xcode in Debug then it works all the time, for me. So I anded up defining Quick and Nimble as a depended build via a subproject.

[ikesyo]

Closing this as part of issue cleanup.

[danielrobleM]

Hi, any solution ?
I don't understand how to use the it ("zzz") {} to fix it. Tests keep disappearing