Quick: `swift test` doesn't actually run tests on macOS

[thislooksfun]

• [x] I have read CONTRIBUTING and have done my best to follow them.

What did you do?

1. Created a simple Spec:

import Quick
import Nimble

class ThingsSpec: QuickSpec {
  override func spec() {
    describe("test") {
      it("does the thing") {
        expect(true).to(beTrue())
      }
    }
  }
}

1. Ran swift test

What did you expect to happen?

I expected it to properly run the test spec, and output the results.

What actually happened instead?

Instead, I got the following output:

Test Suite 'All tests' started at 2017-01-14 00:03:05.037
Test Suite '[REDACTED]PackageTests.xctest' started at 2017-01-14 00:03:05.038
Test Suite '[REDACTED]PackageTests.xctest' passed at 2017-01-14 00:03:05.038.
         Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'All tests' passed at 2017-01-14 00:03:05.038.
         Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds

Environment

List the software versions you're using:

• Quick: 1.0.0
• Nimble: 5.1.1
• Xcode Version: Irrelevant
• Swift Version: Apple Swift version 3.0.2 (swiftlang-800.0.63 clang-800.0.42.1)
• Swift Package Manager 3.0.2 (swiftpm-11750)

Project that demonstrates the issue

I can't link directly to the project in question, as it is closed source, but I had the same problem when attempting to verify the issue with ReactiveCocoa/ReactiveSwift .

[gperdomor]

I confirm the issue, i have a vapor project and tried to configure tests with Quick and finally had to use with XCTests... Related to #675

[thislooksfun]

I should also mention this is on macOS, haven't tried Linux yet.

[ikesyo]

As reported in https://github.com/Quick/Quick/issues/600, Quick can not run with SwiftPM on macOS yet.

[thislooksfun]

So swift test support is Linux only right now?

[ikesyo]

Unfortunately yes.

[thislooksfun]

Well that's a shame. I'm going to close this for now, feel free to re-open it.

Also, I'm assuming not, but do you have any idea when it'll be implemented?

[gperdomor]

@thislooksfun @ikesyo Two things...

• 
  
  1. This is still an issue, and sooner or latter should be solved, so... Open again please.
  
  
• 
  
  1. I made a quick POC in Ubuntu 16.10 and don't run test :(
  
  

I'm doing something wrong?, can you provide a demo or something?

[ikesyo]

@gperdomor

I made a quick POC in Ubuntu 16.10 and don't run test :(

You need to prepare Tests/LinuxMain.swift as follows:

import XCTest
import Quick
@testable import FooBarTests

Quick.QCKMain([
    FooBarSpec.self,
])

See also:

• https://github.com/apple/swift-corelibs-xctest/blob/swift-DEVELOPMENT-SNAPSHOT-2017-01-15-a/Documentation/Linux.md
• https://github.com/apple/swift-package-manager/blob/swift-DEVELOPMENT-SNAPSHOT-2017-01-15-a/Documentation/Reference.md#test-suite-layouts

[thislooksfun]

@gperdomor good point, re-opened.

[thislooksfun]

@ikesyo Actually, would you be interested in an external CLI interface for test running, as an alternative to swift test? I could try to throw one together over the next couple days if you want.

(Sorry for double-comment, only just thought of this)

[ikesyo]

Actually, would you be interested in an external CLI interface for test running, as an alternative to swift test? I could try to throw one together over the next couple days if you want.

I'm not in that direction (but other @Quick/core members may be). I think that the right direction should be supporting SwiftPM on macOS and I'm tackling that locally.

[thislooksfun]

Oh, I fully agree. Supporting SwiftPM natively would be the best-case option, however, when reading one of docs you linked to above, I came across this statement:

When running on the Objective-C runtime, XCTest is able to find all of your tests by simply asking the runtime for the subclasses of XCTestCase. It then finds the methods that start with the string test. This functionality is not currently present when running on the Swift runtime.

Which seems to state that on macOS it basically forces XCTest, so until there is a hook somewhere to add custom testing libraries it looks like it wont be possible.

I was suggesting having a cross-platform CLI that would run the tests _for now_, and then can be deprecated / merged into SwiftPM once that support actually exists.

[thislooksfun]

Tell you what, I'm going to make one anyway for my own use. If/when it's in a usable state I'll check back in. I'll also link to the repo from here once it's up, for anyone coming in the future.

[gperdomor]

@ikesyo can you try this please? not working for me
QuickTest.zip

[ikesyo]

@gperdomor expect(...)... must be in it as follows. You placed it in describe directly.

class MySpec: QuickSpec {
    override func spec() {
        describe("MySpec") {
                it("should run tests") {
                expect(false).to(beTruthy())
            }
        }
    }
}

[gperdomor]

@ikesyo I get this:
Linking ./.build/debug/QuickTestsPackageTests.xctest /usr/bin/ld.gold: error: /home/gperdomor/Desktop/QuickTest/.build/debug/App.build/main.swift.o: multiple definition of 'main' /usr/bin/ld.gold: /tmp/LinuxMain-4b0cc6.o: previous definition here clang: error: linker command failed with exit code 1 (use -v to see invocation) <unknown>:0: error: link command failed with exit code 1 (use -v to see invocation) <unknown>:0: error: build had 1 command failures error: exit(1): /swift-3.0.1/usr/bin/swift-build-tool -f /home/gperdomor/Desktop/QuickTest/.build/debug.yaml test

[ikesyo]

@gperdomor Could you please try swift build --clean first then swift test (that is a link failure and should not be related to Quick itself).

And also could you please file a new issue for further questions? This issue is dedicated for SwiftPM support on macOS, so Linux related ones should be separated.

[gperdomor]

[ikesyo]

@gperdomor Could you please file a new issue?

[ddunbar]

A question on this came up in SwiftPM Slack discussion today... is this something blocked on Quick or blocked on SwiftPM? It isn't clear to me from the discussion here...

[modocache]

I'll take another look at #600 and this issue, and in the next 2-3 days try to write up exactly what would need to be done, in either Quick or in SwiftPM, in order to allow users to successfully run swift test.

[modocache]

On second thought, I don't think I'll have the time to do this. Sorry! If someone else picked it up that'd be grand.

[ikesyo]

I try to explain...

• SwiftPM does not support a mixed-language target of Swift and Objective-C. Quick on macOS consists of the both languages. The Swift part depends on the Objective-C part and vice versa.
  
  
  
  • Because of the language limitation that Objective-C classes cannot extend Swift classes, QuickSpec must be defined in Objective-C.
  
  
  • We need to use NSInvocation related APIs of XCTestCase in XCTest on macOS, but NSInvocation cannot be used from Swift. This is also the reason why we use Objective-C.
  
  
  • Even if we split our codebase into Swift one and Objective-C one, SwiftPM does not allow an Objective-C target to use a Swift target (but the opposite is allowed).
  
  
• In SwiftPM on Linux, we are using pure Swift implementation (we don't care Objective-C users in this case) which is based on LinuxMain.swift entry point , testCase and XCTMain APIs. Thanks to those, we can define QuickSpec in Swift and can run examples without any use of NSInvocations.
• Even if we choose the same pure Swift implementation in SwiftPM on macOS, swift test on macOS runs using ObjC XCTest so we must provide examples of a spec through NSInvocationss and that can't be achieved. There is no LinuxMain.swift, testCase and XCTMain APIs in ObjC XCTest.

[ddunbar]

This can't be done cleanly yet, but could you have an Obj-C implementation target which:

1. Was excluded on Linux (it will have to have another non-empty C file so that you can still depend on it)
2. Is depended on by the actual Swift target.
3. Just provides the hooks needed to create the dynamic NSInvocations
   ?

[ikesyo]

Thanks @ddunbar. I submitted #687 as the PoC of that idea.

[ddunbar]

Awesome!