Fastlane-plugin-test_center: The auto-generated only_testing list now includes properties from non-testcase classes that start with "test"

[armando-leal]

I don't think this was happening on release 3.0.4, but now on 3.1.0 any properties in non-testcase class that belong to the test target will be included in the only_testing test cases list. This will eventually cause the whole run to fail.

Here is an example list I got after adding a swift class to the AtomicBoyTests target. This class includes a testing property which is causing the run to fail.

| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample2", |
| | "AtomicBoyUITests/AtomicBoyUITests/ |
| | testExample", |
| | "AtomicBoyUITests/SwiftAtomicBoyUIT |
| | ests/testExample", |
| | "AtomicBoyUITests/SwiftAtomicBoyUIT |
| | ests/testing.getter : Swift.Int\n", |
| | "AtomicBoyUITests/SwiftAtomicBoyUIT |
| | ests/testing : Swift.Int\n", |
| | "AtomicBoyUITests/TestObject/testin |
| | g.getter : __ObjC.XCTestCase\n", |
| | "AtomicBoyUITests/TestObject/testin |
| | g.materializeForSet : |
| | __ObjC.XCTestCase\n", |
| | "AtomicBoyUITests/TestObject/testin |
| | g.setter : __ObjC.XCTestCase\n", |
| | "AtomicBoyUITests/TestObject/testin |
| | g : __ObjC.XCTestCase\n"]

[lyndsey-ferguson]

Can you provide an example of the Swift class? Also, can you provide the way you're calling the multi_scan action? It should be removing out the weird, obviously not-method strings.

[andriko13]

I'm getting the same issue, after adding a class var "testingLocale". It erroneously appears in the only_testing list, and after renaming it, disappears - pretty easy to reproduce.

[lyndsey-ferguson]

Ok, I added a class variable and I do not see the issue. Here is what I am doing:
In the AtomicBoyTests.m file:

+++ b/AtomicBoy/AtomicBoyTests/AtomicBoyTests.m
@@ -9,12 +9,19 @@
 #import <XCTest/XCTest.h>
 #import <stdlib.h>

-@interface AtomicBoyTests : XCTestCase
+static BOOL _testingLocale = YES;

+@interface AtomicBoyTests : XCTestCase
++(BOOL)testingLocale;
 @end

 @implementation AtomicBoyTests

++ (BOOL)testingLocale {
+    return _testingLocale;
+}
+

From the command line:

bundle exec test_multiscan_workspace

And here is the console log:

[✔] 🚀
+-----------------------------+---------+----------------------------------------+
|                                  Used plugins                                  |
+-----------------------------+---------+----------------------------------------+
| Plugin                      | Version | Action                                 |
+-----------------------------+---------+----------------------------------------+
| fastlane-plugin-test_center | 3.1.0   | suppressed_tests suppress_tests        |
|                             |         | suppress_tests_from_junit              |
|                             |         | tests_from_xctestrun tests_from_junit  |
|                             |         | multi_scan collate_junit_reports       |
+-----------------------------+---------+----------------------------------------+

[12:34:34]: You have require'd a gem, if this is a third party gem, please use `fastlane_require 'fastlane/actions/scan'` to ensure the gem is installed locally.
[12:34:34]: Driving the lane 'test_multiscan_workspace' 🚀
[12:34:34]: ------------------------
[12:34:34]: --- Step: multi_scan ---
[12:34:34]: ------------------------

+----------------------+----------------------+
| Summary for multi_scan (test_center v3.1.0) |
+----------------------+----------------------+
| try_count            | 3                    |
+----------------------+----------------------+

[12:34:34]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:34:34]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:34:36]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:34:38]: Found simulator "iPhone 5s (11.1)"
[12:34:38]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:34:38]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:34:39]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:34:40]: Found simulator "iPhone 5s (11.1)"

+------------------------+-----------------------------------------------------------------------+
|                                    Summary for scan 2.84.0                                     |
+------------------------+-----------------------------------------------------------------------+
| workspace              | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A  |
|                        | tomicBoy.xcworkspace                                                  |
| scheme                 | AtomicBoy                                                             |
| build_for_testing      | true                                                                  |
| derived_data_path      | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo  |
|                        | y-awhdzysurqhqyrcsnwebnamaphxb                                        |
| clean                  | true                                                                  |
| skip_build             | false                                                                 |
| output_directory       | ./fastlane/test_output                                                |
| output_types           | html,junit                                                            |
| buildlog_path          | ~/Library/Logs/scan                                                   |
| include_simulator_logs | false                                                                 |
| open_report            | false                                                                 |
| skip_slack             | false                                                                 |
| slack_only_on_failure  | false                                                                 |
| use_clang_report_name  | false                                                                 |
| fail_build             | true                                                                  |
| xcode_path             | /Applications/Xcode9.1.0.app                                          |
+------------------------+-----------------------------------------------------------------------+

[12:34:40]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy -destination 'platform=iOS Simulator,id=885F68C7-1032-4593-A206-6A3A011E53A4' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb' clean build-for-testing | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty  --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180321-32907-ojle1w'
[12:34:40]: ▸ Loading...
[12:34:41]: ▸ Cleaning AtomicBoy/AtomicBoy [Debug]
[12:34:41]: ▸ Check Dependencies
[12:34:41]: ▸ Clean Succeeded
[12:34:41]: ▸ Building AtomicBoy/AtomicBoy [Debug]
[12:34:41]: ▸ Check Dependencies
[12:34:42]: ▸ Compiling GameScene.m
[12:34:42]: ▸ Compiling GameViewController.m
[12:34:42]: ▸ Compiling main.m
[12:34:42]: ▸ Compiling AppDelegate.m
[12:34:42]: ▸ Linking AtomicBoy
[12:34:42]: ▸ Copying AtomicBoy/GameScene.sks
[12:34:42]: ▸ Copying AtomicBoy/Actions.sks
[12:34:42]: ▸ Compiling LaunchScreen.storyboard
[12:34:46]: ▸ Compiling Main.storyboard
[12:34:46]: ▸ Processing Info.plist
[12:34:46]: Running Tests: ▸ Touching AtomicBoy.app
[12:34:46]: ▸ Signing /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb/Build/Products/Debug-iphonesimulator/AtomicBoy.app
[12:34:46]: ▸ Building AtomicBoy/AtomicBoyUITests [Debug]
[12:34:46]: ▸ Check Dependencies
[12:34:46]: ▸ Building AtomicBoy/AtomicBoyTests [Debug]
[12:34:46]: ▸ Check Dependencies
[12:34:46]: ▸ Processing Info.plist
[12:34:46]: ▸ Copying /Applications/Xcode9.1.0.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/PrivateFrameworks/IDEBundleInjection.framework
[12:34:46]: ▸ Copying /Applications/Xcode9.1.0.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks/XCTest.framework
[12:34:46]: ▸ Linking AtomicBoyTests
[12:34:46]: Running Tests: ▸ Touching AtomicBoyTests.xctest
[12:34:46]: ▸ Signing /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb/Build/Products/Debug-iphonesimulator/AtomicBoy.app/PlugIns/AtomicBoyTests.xctest
[12:34:46]: ▸ Test build Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

[12:34:47]: ------------------------------------------------------
[12:34:47]: --- Starting test run on testable 'AtomicBoyTests' ---
[12:34:47]: ------------------------------------------------------
[12:34:48]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:34:48]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:34:48]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:34:50]: Found simulator "iPhone 5s (11.1)"
[12:34:50]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:34:50]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:34:51]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:34:52]: Found simulator "iPhone 5s (11.1)"

+------------------------+-----------------------------------------------------------------------+
|                                    Summary for scan 2.84.0                                     |
+------------------------+-----------------------------------------------------------------------+
| workspace              | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A  |
|                        | tomicBoy.xcworkspace                                                  |
| scheme                 | AtomicBoy                                                             |
| test_without_building  | true                                                                  |
| derived_data_path      | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo  |
|                        | y-awhdzysurqhqyrcsnwebnamaphxb                                        |
| skip_build             | false                                                                 |
| output_types           | html,junit                                                            |
| output_files           | report.html,report.junit                                              |
| buildlog_path          | ~/Library/Logs/scan                                                   |
| include_simulator_logs | false                                                                 |
| open_report            | false                                                                 |
| skip_slack             | false                                                                 |
| slack_only_on_failure  | false                                                                 |
| use_clang_report_name  | false                                                                 |
| clean                  | false                                                                 |
| only_testing           | ["AtomicBoyTests/AtomicBoyTests/testExample",                         |
|                        | "AtomicBoyTests/AtomicBoyTests/testPerformanceExample"]               |
| output_directory       | ./fastlane/test_output/results-AtomicBoyTests                         |
| fail_build             | true                                                                  |
| xcode_path             | /Applications/Xcode9.1.0.app                                          |
+------------------------+-----------------------------------------------------------------------+

[12:34:52]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy -destination 'platform=iOS Simulator,id=885F68C7-1032-4593-A206-6A3A011E53A4' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb' -only-testing:AtomicBoyTests/AtomicBoyTests/testExample -only-testing:AtomicBoyTests/AtomicBoyTests/testPerformanceExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty  --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180321-32907-rt4kur'
[12:34:52]: ▸ Loading...
[12:34:54]: ▸ 2018-03-21 12:34:54.360 xcodebuild[33059:5205641]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[12:34:54]: ▸ /var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-1A0BC104-5AD6-4DE5-B6EA-CC27406036D4/AtomicBoyTests-6503BAD2-CC90-4F8A-86CB-594F48437D74/Session-AtomicBoyTests-2018-03-21_123454-gp4Txj.log
[12:34:54]: ▸ 2018-03-21 12:34:54.362 xcodebuild[33059:5205629] [MT] IDETestOperationsObserverDebug: (1068B03C-EBAE-499C-94C2-B19A3D82896A) Beginning test session AtomicBoyTests-1068B03C-EBAE-499C-94C2-B19A3D82896A at 2018-03-21 12:34:54.362 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7fa5465d0e50> {
[12:34:54]: ▸ SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
[12:34:54]: ▸ } (11.1 (15B87))
[12:35:05]: ▸ Selected tests
[12:35:05]: ▸ Test Suite AtomicBoyTests.xctest started
[12:35:05]: ▸ AtomicBoyTests
[12:35:05]: ▸ ✗ testExample, ((false) is true) failed
[12:35:05]: ▸ ◷ testPerformanceExample measured (0.000 seconds)
[12:35:05]: ▸ ✓ testPerformanceExample (0.417 seconds)
[12:35:05]: ▸ AtomicBoyTests
[12:35:05]: ▸ testExample, ((false) is true) failed
[12:35:05]: ▸ /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoyTests/AtomicBoyTests.m:38
[12:35:05]: ▸ ```
[12:35:05]: ▸     if (arc4random_uniform(6) < 3) {
[12:35:05]: ▸         XCTAssertTrue(false);
[12:35:05]: ▸     }
[12:35:05]: ▸   ```
[12:35:05]: ▸    Executed 2 tests, with 1 failure (0 unexpected) in 0.419 (0.421) seconds
[12:35:05]: ▸
[12:35:05]: ▸ ** TEST EXECUTE FAILED **
2018-03-21 12:34:54.360 xcodebuild[33059:5205641]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-1A0BC104-5AD6-4DE5-B6EA-CC27406036D4/AtomicBoyTests-6503BAD2-CC90-4F8A-86CB-594F48437D74/Session-AtomicBoyTests-2018-03-21_123454-gp4Txj.log
2018-03-21 12:34:54.362 xcodebuild[33059:5205629] [MT] IDETestOperationsObserverDebug: (1068B03C-EBAE-499C-94C2-B19A3D82896A) Beginning test session AtomicBoyTests-1068B03C-EBAE-499C-94C2-B19A3D82896A at 2018-03-21 12:34:54.362 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7fa5465d0e50> {
SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyTests.xctest started
AtomicBoyTests
✗ testExample, ((false) is true) failed
◷ testPerformanceExample measured (0.000 seconds)
✓ testPerformanceExample (0.417 seconds)


AtomicBoyTests
testExample, ((false) is true) failed
/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoyTests/AtomicBoyTests.m:38

if (arc4random_uniform(6) < 3) {
    XCTAssertTrue(false);
}

```

     Executed 2 tests, with 1 failure (0 unexpected) in 0.419 (0.421) seconds

* TEST EXECUTE FAILED *
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 2 |
| Number of failures | 1 |
+--------------------+---+

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+

+------------------------+-----------------------------------------------------------------------+
| Summary for scan 2.84.0 |
+------------------------+-----------------------------------------------------------------------+
| workspace | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A |
| | tomicBoy.xcworkspace |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo |
| | y-awhdzysurqhqyrcsnwebnamaphxb |
| skip_build | false |
| output_types | html,junit |
| output_files | report-2.html,report-2.junit |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyTests/AtomicBoyTests/testExample"] |
| output_directory | ./fastlane/test_output/results-AtomicBoyTests |
| fail_build | true |
| xcode_path | /Applications/Xcode9.1.0.app |
+------------------------+-----------------------------------------------------------------------+

2018-03-21 12:35:24.327 xcodebuild[33245:5207884] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-DA68D4AB-4F22-49D2-8603-094E13A8432C/AtomicBoyTests-816B9B33-A4F1-412C-B50A-14A4E1ED0A12/Session-AtomicBoyTests-2018-03-21_123524-zsZDEE.log
2018-03-21 12:35:24.328 xcodebuild[33245:5207878] [MT] IDETestOperationsObserverDebug: (B91F3757-C97F-4DCD-B743-5CAA871B6F4F) Beginning test session AtomicBoyTests-B91F3757-C97F-4DCD-B743-5CAA871B6F4F at 2018-03-21 12:35:24.327 with Xcode 9B55 on target {
SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyTests.xctest started
AtomicBoyTests
✗ testExample, ((false) is true) failed

AtomicBoyTests
testExample, ((false) is true) failed
/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoyTests/AtomicBoyTests.m:38

    if (arc4random_uniform(6) < 3) {
        XCTAssertTrue(false);
    }
  ```


         Executed 1 test, with 1 failure (0 unexpected) in 0.003 (0.004) seconds

** TEST EXECUTE FAILED **
[12:35:44]: Exit status: 65
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 1 |
| Number of failures | 1 |
+--------------------+---+

[12:35:44]: Re-running scan on only failed tests
[12:35:45]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:35:45]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:35:46]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:35:47]: Found simulator "iPhone 5s (11.1)"
[12:35:47]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:35:47]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:35:48]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:35:49]: Found simulator "iPhone 5s (11.1)"

+------------------------+-----------------------------------------------------------------------+
|                                    Summary for scan 2.84.0                                     |
+------------------------+-----------------------------------------------------------------------+
| workspace              | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A  |
|                        | tomicBoy.xcworkspace                                                  |
| scheme                 | AtomicBoy                                                             |
| test_without_building  | true                                                                  |
| derived_data_path      | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo  |
|                        | y-awhdzysurqhqyrcsnwebnamaphxb                                        |
| skip_build             | false                                                                 |
| output_types           | html,junit                                                            |
| output_files           | report-3.html,report-3.junit                                          |
| buildlog_path          | ~/Library/Logs/scan                                                   |
| include_simulator_logs | false                                                                 |
| open_report            | false                                                                 |
| skip_slack             | false                                                                 |
| slack_only_on_failure  | false                                                                 |
| use_clang_report_name  | false                                                                 |
| clean                  | false                                                                 |
| only_testing           | ["AtomicBoyTests/AtomicBoyTests/testExample"]                         |
| output_directory       | ./fastlane/test_output/results-AtomicBoyTests                         |
| fail_build             | true                                                                  |
| xcode_path             | /Applications/Xcode9.1.0.app                                          |
+------------------------+-----------------------------------------------------------------------+

[12:35:50]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy -destination 'platform=iOS Simulator,id=885F68C7-1032-4593-A206-6A3A011E53A4' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb' -only-testing:AtomicBoyTests/AtomicBoyTests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty  --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report-3.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report-3.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180321-32907-10hp7f0'
[12:35:50]: ▸ Loading...
[12:35:51]: ▸ 2018-03-21 12:35:51.778 xcodebuild[33421:5210029]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[12:35:51]: ▸ /var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-C10D06D1-9D09-4634-A8FA-A1111B72AD79/AtomicBoyTests-9913A79D-DAC8-415F-B998-DBAC200D2017/Session-AtomicBoyTests-2018-03-21_123551-4hWvHb.log
[12:35:51]: ▸ 2018-03-21 12:35:51.779 xcodebuild[33421:5210014] [MT] IDETestOperationsObserverDebug: (AA29208D-F365-41CD-8BF3-C3E6FCDD6593) Beginning test session AtomicBoyTests-AA29208D-F365-41CD-8BF3-C3E6FCDD6593 at 2018-03-21 12:35:51.779 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7fc663ba2b20> {
[12:35:51]: ▸ SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
[12:35:51]: ▸ } (11.1 (15B87))
[12:35:59]: ▸ Selected tests
[12:35:59]: ▸ Test Suite AtomicBoyTests.xctest started
[12:35:59]: ▸ AtomicBoyTests
[12:35:59]: ▸ ✓ testExample (0.001 seconds)
[12:35:59]: ▸    Executed 1 test, with 0 failures (0 unexpected) in 0.001 (0.003) seconds
[12:35:59]: ▸
[12:36:00]: ▸ Test execute Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 1 |
| Number of failures | 0 |
+--------------------+---+

[12:36:11]: --------------------------------------------------------
[12:36:11]: --- Starting test run on testable 'AtomicBoyUITests' ---
[12:36:11]: --------------------------------------------------------
[12:36:12]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:36:12]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:36:13]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:36:14]: Found simulator "iPhone 5s (11.1)"
[12:36:14]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:36:14]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:36:15]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:36:16]: Found simulator "iPhone 5s (11.1)"

+------------------------+-----------------------------------------------------------------------+
|                                    Summary for scan 2.84.0                                     |
+------------------------+-----------------------------------------------------------------------+
| workspace              | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A  |
|                        | tomicBoy.xcworkspace                                                  |
| scheme                 | AtomicBoy                                                             |
| test_without_building  | true                                                                  |
| derived_data_path      | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo  |
|                        | y-awhdzysurqhqyrcsnwebnamaphxb                                        |
| skip_build             | false                                                                 |
| output_types           | html,junit                                                            |
| output_files           | report.html,report.junit                                              |
| buildlog_path          | ~/Library/Logs/scan                                                   |
| include_simulator_logs | false                                                                 |
| open_report            | false                                                                 |
| skip_slack             | false                                                                 |
| slack_only_on_failure  | false                                                                 |
| use_clang_report_name  | false                                                                 |
| clean                  | false                                                                 |
| only_testing           | ["AtomicBoyUITests/AtomicBoyUITests/testExample2",                    |
|                        | "AtomicBoyUITests/AtomicBoyUITests/testExample"]                      |
| output_directory       | ./fastlane/test_output/results-AtomicBoyUITests                       |
| fail_build             | true                                                                  |
| xcode_path             | /Applications/Xcode9.1.0.app                                          |
+------------------------+-----------------------------------------------------------------------+

[12:36:16]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy -destination 'platform=iOS Simulator,id=885F68C7-1032-4593-A206-6A3A011E53A4' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample2 -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty  --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180321-32907-1xs239r'
[12:36:16]: ▸ Loading...
[12:36:18]: ▸ 2018-03-21 12:36:18.829 xcodebuild[33597:5212138]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[12:36:18]: ▸ /var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-7C44C734-AC9B-4F69-BA9D-D3918187F18B/AtomicBoyUITests-B548312A-2029-4C6A-9792-DBCCCC865549/Session-AtomicBoyUITests-2018-03-21_123618-q5RGq8.log
[12:36:18]: ▸ 2018-03-21 12:36:18.830 xcodebuild[33597:5212127] [MT] IDETestOperationsObserverDebug: (D0BDC863-F7EE-40D6-B9E1-D9345389478F) Beginning test session AtomicBoyUITests-D0BDC863-F7EE-40D6-B9E1-D9345389478F at 2018-03-21 12:36:18.830 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7fb21e070030> {
[12:36:18]: ▸ SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
[12:36:18]: ▸ } (11.1 (15B87))
[12:36:32]: ▸ Selected tests
[12:36:32]: ▸ Test Suite AtomicBoyUITests.xctest started
[12:36:32]: ▸ AtomicBoyUITests
[12:36:37]: ▸ ✗ testExample, ((false) is true) failed
[12:36:41]: ▸ ✓ testExample2 (4.534 seconds)
[12:36:41]: ▸ AtomicBoyUITests
[12:36:41]: ▸ testExample, ((false) is true) failed
[12:36:41]: ▸ AtomicBoyUITests.m:40
[12:36:41]: ▸    Executed 2 tests, with 1 failure (0 unexpected) in 9.621 (9.623) seconds
[12:36:41]: ▸
[12:36:42]: ▸ ** TEST EXECUTE FAILED **
2018-03-21 12:36:18.829 xcodebuild[33597:5212138]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-7C44C734-AC9B-4F69-BA9D-D3918187F18B/AtomicBoyUITests-B548312A-2029-4C6A-9792-DBCCCC865549/Session-AtomicBoyUITests-2018-03-21_123618-q5RGq8.log
2018-03-21 12:36:18.830 xcodebuild[33597:5212127] [MT] IDETestOperationsObserverDebug: (D0BDC863-F7EE-40D6-B9E1-D9345389478F) Beginning test session AtomicBoyUITests-D0BDC863-F7EE-40D6-B9E1-D9345389478F at 2018-03-21 12:36:18.830 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7fb21e070030> {
SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyUITests.xctest started
AtomicBoyUITests
✗ testExample, ((false) is true) failed
✓ testExample2 (4.534 seconds)


AtomicBoyUITests
testExample, ((false) is true) failed
AtomicBoyUITests.m:40


         Executed 2 tests, with 1 failure (0 unexpected) in 9.621 (9.623) seconds

** TEST EXECUTE FAILED **
[12:36:51]: Exit status: 65
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 2 |
| Number of failures | 1 |
+--------------------+---+

[12:36:51]: Re-running scan on only failed tests
[12:36:52]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:36:52]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:36:53]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:36:55]: Found simulator "iPhone 5s (11.1)"
[12:36:55]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean                | true                |
+----------------------+---------------------+

[12:36:55]: $ xcodebuild -list -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace
[12:36:56]: $ xcodebuild -showBuildSettings -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy
[12:36:57]: Found simulator "iPhone 5s (11.1)"

+------------------------+-----------------------------------------------------------------------+
|                                    Summary for scan 2.84.0                                     |
+------------------------+-----------------------------------------------------------------------+
| workspace              | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/A  |
|                        | tomicBoy.xcworkspace                                                  |
| scheme                 | AtomicBoy                                                             |
| test_without_building  | true                                                                  |
| derived_data_path      | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBo  |
|                        | y-awhdzysurqhqyrcsnwebnamaphxb                                        |
| skip_build             | false                                                                 |
| output_types           | html,junit                                                            |
| output_files           | report-2.html,report-2.junit                                          |
| buildlog_path          | ~/Library/Logs/scan                                                   |
| include_simulator_logs | false                                                                 |
| open_report            | false                                                                 |
| skip_slack             | false                                                                 |
| slack_only_on_failure  | false                                                                 |
| use_clang_report_name  | false                                                                 |
| clean                  | false                                                                 |
| only_testing           | ["AtomicBoyUITests/AtomicBoyUITests/testExample"]                     |
| output_directory       | ./fastlane/test_output/results-AtomicBoyUITests                       |
| fail_build             | true                                                                  |
| xcode_path             | /Applications/Xcode9.1.0.app                                          |
+------------------------+-----------------------------------------------------------------------+

[12:36:57]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcworkspace -scheme AtomicBoy -destination 'platform=iOS Simulator,id=885F68C7-1032-4593-A206-6A3A011E53A4' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-awhdzysurqhqyrcsnwebnamaphxb' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty  --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180321-32907-bwctbg'
[12:36:57]: ▸ Loading...
[12:36:59]: ▸ 2018-03-21 12:36:59.381 xcodebuild[33800:5214701]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[12:36:59]: ▸ /var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/com.apple.dt.XCTest/IDETestRunSession-59E7DF7C-C2B9-4ADB-B469-7351589182B7/AtomicBoyUITests-D76DDE63-D79E-4625-9C12-10971748DC96/Session-AtomicBoyUITests-2018-03-21_123659-eRrQI0.log
[12:36:59]: ▸ 2018-03-21 12:36:59.382 xcodebuild[33800:5214694] [MT] IDETestOperationsObserverDebug: (8D060B18-1DF6-4B23-9611-4C91C9283380) Beginning test session AtomicBoyUITests-8D060B18-1DF6-4B23-9611-4C91C9283380 at 2018-03-21 12:36:59.381 with Xcode 9B55 on target <DVTiPhoneSimulator: 0x7f82bab4e100> {
[12:36:59]: ▸ SimDevice: iPhone 5s (885F68C7-1032-4593-A206-6A3A011E53A4, iOS 11.1, Booted)
[12:36:59]: ▸ } (11.1 (15B87))
[12:37:08]: ▸ Selected tests
[12:37:08]: ▸ Test Suite AtomicBoyUITests.xctest started
[12:37:08]: ▸ AtomicBoyUITests
[12:37:13]: ▸ ✓ testExample (4.843 seconds)
[12:37:13]: ▸    Executed 1 test, with 0 failures (0 unexpected) in 4.843 (4.846) seconds
[12:37:13]: ▸
[12:37:14]: ▸ Test execute Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 1 |
| Number of failures | 0 |
+--------------------+---+


+-------------------+-----------------------------------------------------------------------+
|                                    multi_scan results                                     |
+-------------------+-----------------------------------------------------------------------+
| result            | true                                                                  |
| total_tests       | 4                                                                     |
| passing_testcount | 4                                                                     |
| failed_testcount  | 0                                                                     |
| failed_tests      | []                                                                    |
| total_retry_count | 3                                                                     |
| report_files      | ["/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/  |
|                   | test_output/report.junit",                                            |
|                   | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/t  |
|                   | est_output/results-AtomicBoyUITests/report.junit",                    |
|                   | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/t  |
|                   | est_output/results-AtomicBoyTests/report.junit"]                      |
+-------------------+-----------------------------------------------------------------------+


+------+------------+-------------+
|        fastlane summary         |
+------+------------+-------------+
| Step | Action     | Time (in s) |
+------+------------+-------------+
| 1    | multi_scan | 167         |
+------+------------+-------------+

[12:37:21]: fastlane.tools finished successfully 🎉

#######################################################################
# fastlane 2.86.2 is available. You are on 2.84.0.
# You should use the latest version.
# Please update using `bundle update fastlane`.
#######################################################################

2.86.2 Fixes get_version_number to auto-select target when there is only one
* Fixes get_version_number to auto-select target when there is only one (#12121) via Josh Holtz

2.86.1 Fix for crash when download_dsyms can't find a version
- Fix a crashing bug in download_dsyms when can't match a version (#12095) via Takeru Chuganji

2.86.0 Lots of action fixes and internal improvements
* Add git diff option to ensure_git_status_clean (#12063) via Nicolas Braun
* Add the git_submodule_update action (#12064) via Nicolas Braun
* Ignore FastlaneSwiftRunner.xcodeproj for detecting iOS project's of users (#12075) via Fumiya Nakamura
* Take xcconfig option into account when generating `xcodebuild -showBuildSettings` command. (#12067) via Chris Campbell
* Fixes app_store_build_number for finding latest version (#12073) via Josh Holtz
* snapshot - get actual sim device types instead of inferring by name (#12039) via Josh Holtz
* Fastlane.swift - Allow actions with zero options (#12065) via Josh Holtz
* get_version_number now uses xcodeproj gem (#12037) via Josh Holtz
* deliver - option for rejecting app when submitting if already waiting for review (#10722) via Rishabh Tayal
* Removed and/or for conditionals with rubo (#12049) via Josh Holtz
* download_dysm will now find more trains and versions by switching to tunes API (#12068) via Josh Holtz
* [spaceship] Raise a meaningful error if the device name is too long (#12059) via Thi
* Running dotenv before any lanes called so disable color can be applied to everything (#12047) via Josh Holtz
* Fix auto generated files link (#12033) via Iulian Onofrei
* Use xcodeproj to introspect Xcode projects (#12027) via Danielle Tomlinson
* Reinitialize Spaceship::Tunes.client different teams (#11970) via Matthew Nespor
* supply can call track_promote without passing an APKs (#12021) via Josh Holtz

To see all new releases, open https://github.com/fastlane/fastlane/releases

Please update using `bundle update fastlane`

None of the only_testing parameters include mangled test identifiers. Can one of you include exactly what you're doing so that I can reproduce this and fix it?

Thanks!

[lyndsey-ferguson]

I'm looking for the exact source code change, the command used to run fastlane and an example of the Fastfile.

[armando-leal]

I added the following two swift classes to AtomicBoyUITests

//
//  SwiftAtomicBoyUITests.swift
//  AtomicBoyUITests
//
//

import XCTest

class SwiftAtomicBoyUITests: XCTestCase {

    func testExample() {
        let _ = TestObject(testcase: self)
        if (arc4random_uniform(6) < 3) {
            XCTAssertTrue(false)
        }
    }

}

//
//  TestObject.swift
//  AtomicBoyUITests
//
//

import UIKit
import XCTest

class TestObject: NSObject {

    var testing : XCTestCase

    convenience init(testcase: XCTestCase) {
        self.init()
        self.testing = testcase
    }

    override init() {
        self.testing = XCTestCase()
        super.init()
    }
}

[lyndsey-ferguson]

Thank you, exactly what I needed

[lyndsey-ferguson]

Fixed in v3.1.1

[lyndsey-ferguson]

Make sure you are getting xctest_list v1.1.3. That is what I used to :build, and :publish the gem.

[armando-leal]

Wow that was quick! Thanks for resolving this :)

[lyndsey-ferguson]

@armando-leal does this work for you? I'd love a confirmation.

[armando-leal]

I'm testing it out right now. I'll let you know.

[lyndsey-ferguson]

🤞

[armando-leal]

Issue resolved 👍

Thanks again!