Quick: [Xcode 8] Does test filtering work with Quick?

[modocache]

Someone with Xcode 8 installed should try the new xcodebuild test filters with Quick. Does everything work as expected?

xcodebuild test -only-testing:MyTestBundle/MyTestSuite/MySpec
xcodebuild test -only-testing:MyTestBundle/MyTestSuite
xcodebuild test -only-testing:MyTestBundle

xcodebuild test -skip-testing:MyOtherTestBundle/MyTestSuite/MySpec

[noisypigeon]

I can't install betas on my main work partition, but I'm going to setup a partition with macOS and the beta Xcode to test soon.

[modocache]

Awesome, thanks @istx25! Let me know how it goes.

[omnianoureldin]

Hello, I was wondering if there is any progress here. I tried the filtering with xCode 8 and it doesn't seem to work.

[noisypigeon]

I still haven't had the chance to get around to this. My Mac was acting up around this and then I forgot. So no progress yet. Sorry!

[MP0w]

I tried it out:

-only-testing:TEST_BUNDLE/MODULE_NAME.SPEC_CLASS_NAME works
-only-testing:TEST_BUNDLE/MODULE_NAME.SPEC_CLASS_NAME/TEST_NAME didn't work
however I was not able to "only test" a single test even for regular (XC)tests. Looks like whatever I add after ...CLASS_NAME/ doesn't matter

[kmahar]

Related to this, is there anyway to use the --filter arg to swift test to only run some tests by name? I have test suites broken up by name (ex. ClientTests), and when ClientTests was an XCTestCase I could pass in --filter ClientTests to just run that group quickly. Now that I've changed ClientTests to be a QuickSpec no tests are run at all when I pass in that option.

[phatblat]

I played around with this using swift 5.2.2 and the full test names can be found using the --list-tests option.

↪ swift test --list-tests
PinKitTests.PostSpec/post__has_a_description
PinKitTests.PostSpec/post__has_an_extended
PinKitTests.PostSpec/post__has_a_hash
PinKitTests.PostSpec/post__has_a_description_2
PinKitTests.PostSpec/post__has_a_meta
PinKitTests.PostSpec/post__is_shared
PinKitTests.PostSpec/post__has_tags
PinKitTests.PostSpec/post__has_a_time
PinKitTests.PostSpec/post__has_a_toread_flag
PinKitTests.PostsAddSpec/posts_add__has_a_result_code
PinKitTests.PostsAddSpec/posts_add__parsing__can_be_parsed_from_json
PinKitTests.PostsDeleteSpec/posts_delete__has_a_result_code
PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
phatblat at octodec in ~/dev/swift/PinKit on master [✓]

Providing the full test name (or just the ) to the --filter option seems to do less work because there is a lot less output than when running without the filter. However it doesn't appear to actually run any tests due to the "Executed 0 tests" output.

↪ swift test --filter PinKitTests.PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Test Suite 'Selected tests' started at 2020-06-09 13:28:41.011
Test Suite 'PinKitPackageTests.xctest' started at 2020-06-09 13:28:41.012
Test Suite 'PinKitPackageTests.xctest' passed at 2020-06-09 13:28:41.012.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:28:41.012.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

xcodebuild (v11.5) has the same issue when running a subset of tests. It runs a test suite called 'Selected tests' and the summary shows it "Executed 0 tests".

↪ xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package -only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json
Command line invocation:
    /Applications/Xcode-11.5.app/Contents/Developer/usr/bin/xcodebuild test -workspace PinKit.xcworkspace -scheme PinKit-Package "-only-testing:PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"

User defaults from command line:
    IDETestRunOnlyIdentifiers = (
    "PinKitTests/PostsDeleteSpec/posts_delete__parsing__can_be_parsed_from_json"
)

...
Test Suite 'Selected tests' started at 2020-06-09 13:38:47.655
Test Suite 'PinKitTests.xctest' started at 2020-06-09 13:38:47.656
Test Suite 'PinKitTests.xctest' passed at 2020-06-09 13:38:47.656.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2020-06-09 13:38:47.656.
     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
...

Test session results, code coverage, and logs:
    /Volumes/DerivedData/PinKit-akgpefidgquoryhfyxurznnvcfrr/Logs/Test/Test-PinKit-Package-2020.06.09_13-38-46--0600.xcresult

** TEST SUCCEEDED ** [1.737 sec]

Something in the swift test command is able to see the individual Quick tests. I wonder if that mechanism can be used to help expose the filtered set of "Selected tests" from Quick to the XCTest runtime.