Fastlane-plugin-test_center: multi_scan succeeds when retrying test failures from multiple test targets

[asos-chrisblackmore]

New Issue Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example(s) for the action(s) I am using
• [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍

Issue Description

As the title suggests, when an Xcode project contains multiple test targets, and some of those targets contain test failures, multi_scan will retry a specific test case for the incorrect test suite, resulting in no test failures being reported.

I've attached an Xcode project to demonstrate the issue (run fastlane iOS unit_test to observe) - but consider the following test setup (the first 2 tests failing, the last test succeeding):

TestTarget1.TestTarget1Tests.testTarget1Test() ❌
TestTarget2.TestTarget2Tests.testTarget2Test() ❌
TestTarget3.TestTarget3Tests.testTarget3test() ✅

The first test attempt correctly reports the following:

Failing tests:
    TestTarget1:
        TestTarget1Tests.testTarget1Test()
    TestTarget2:
        TestTarget2Tests.testTarget2Test()

** TEST EXECUTE FAILED **
[18:11:00]: Exit status: 65
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 3 |
| Number of failures | 2 |
+--------------------+---+

However, the second test attempt passes the following to `xcodebuild`:

only_testing: ["TestTarget3/TestTarget2Tests"]

..as you can see, it's pulling the last executed test target `TestTarget3`, and the last test failure `testTarget2Test`. I confirmed in my personal Xcode project that it indeed always pulls the last test target execution, whether it contained failures or not. This confuses `xcodebuild` apparently, because it actually succeeds, since 0 tests were actually executed:

Test Suite TestTarget3.xctest started
[32;1m   Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
[0m
xcodebuild[47322:925591] [MT] IDETestOperationsObserverDebug: 62.869 elapsed -- Testing started completed.
xcodebuild[47322:925591] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
xcodebuild[47322:925591] [MT] IDETestOperationsObserverDebug: 62.869 sec, +62.869 sec -- end
Test execute Succeeded

+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

..`multi_scan` then goes on to incorrectly report that there are 0 failures. Like I said, please observe the Xcode project attached, that will hopefully explain the issue much better than what I have! [MultiScanDemo.zip](https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/files/5568834/MultiScanDemo.zip) ##### Complete output when running fastlane, including the stack trace and command used

asglh-ml-13805:MutliScanDemo chris.blackmore$ fastlane ios unit_test
[✔] 🚀 
[18:05:16]: fastlane detected a Gemfile in the current directory
[18:05:16]: However, it seems like you didn't use `bundle exec`
[18:05:16]: To launch fastlane faster, please use
[18:05:16]: 
[18:05:16]: $ bundle exec fastlane ios unit_test
[18:05:16]: 
[18:05:16]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
+-----------------------------+---------+--------------------------------------------------------------------------------+
|                                                      Used plugins                                                      |
+-----------------------------+---------+--------------------------------------------------------------------------------+
| Plugin                      | Version | Action                                                                         |
+-----------------------------+---------+--------------------------------------------------------------------------------+
| fastlane-plugin-test_center | 3.14.8  | suppressed_tests suppress_tests tests_from_xcresult suppress_tests_from_junit  |
|                             |         | tests_from_xctestrun quit_core_simulator_service collate_test_result_bundles   |
|                             |         | tests_from_junit multi_scan test_options_from_testplan collate_html_reports    |
|                             |         | collate_junit_reports collate_json_reports testplans_from_scheme               |
|                             |         | collate_xcresults                                                              |
+-----------------------------+---------+--------------------------------------------------------------------------------+

[18:05:19]: Driving the lane 'ios unit_test' 🚀
[18:05:19]: ------------------------
[18:05:19]: --- Step: multi_scan ---
[18:05:19]: ------------------------
[18:05:19]: As of Xcode 11, test_result bundles created in the output directory are actually symbolic links to an xcresult bundle

+------------------------+---------------------+
| Summary for multi_scan (test_center v3.14.8) |
+------------------------+---------------------+
| collate_reports        | true                |
| invocation_based_tests | true                |
| try_count              | 3                   |
+------------------------+---------------------+

[18:05:19]: Using deprecated option: '--custom_report_file_name' (Use `--output_files` instead)
[18:05:19]: Using deprecated option: '--custom_report_file_name' (Use `--output_files` instead)
[18:05:19]: Resolving Swift Package Manager dependencies...
[18:05:19]: $ xcodebuild -resolvePackageDependencies -scheme MutliScanDemo -project ./MultiScanDemo.xcodeproj
[18:05:20]: ▸ Command line invocation:
[18:05:20]: ▸     /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -resolvePackageDependencies -scheme MutliScanDemo -project ./MultiScanDemo.xcodeproj
[18:05:21]: ▸ resolved source packages: 
[18:05:21]: $ xcodebuild -showBuildSettings -scheme MutliScanDemo -project ./MultiScanDemo.xcodeproj
[18:05:23]: Found simulator "iPhone 8 (14.0)"

+------------------------------------------------+-------------------------------------------------------------------------------------------------------+
|                                                                Summary for scan 2.168.0                                                                |
+------------------------------------------------+-------------------------------------------------------------------------------------------------------+
| buildlog_path                                  | ./build/logs                                                                                          |
| output_directory                               | ./build/output                                                                                        |
| output_types                                   | junit                                                                                                 |
| output_files                                   | report.junit                                                                                          |
| scheme                                         | MutliScanDemo                                                                                         |
| fail_build                                     | true                                                                                                  |
| reset_simulator                                | true                                                                                                  |
| suppress_xcode_output                          | false                                                                                                 |
| project                                        | ./MultiScanDemo.xcodeproj                                                                             |
| skip_detect_devices                            | false                                                                                                 |
| force_quit_simulator                           | false                                                                                                 |
| disable_slide_to_type                          | true                                                                                                  |
| reinstall_app                                  | false                                                                                                 |
| clean                                          | false                                                                                                 |
| open_report                                    | false                                                                                                 |
| derived_data_path                              | /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg |
| should_zip_build_products                      | false                                                                                                 |
| use_clang_report_name                          | false                                                                                                 |
| disable_concurrent_testing                     | false                                                                                                 |
| build_for_testing                              | true                                                                                                  |
| slack_use_webhook_configured_username_and_icon | false                                                                                                 |
| slack_username                                 | fastlane                                                                                              |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png                                                   |
| skip_slack                                     | false                                                                                                 |
| slack_only_on_failure                          | false                                                                                                 |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                                                                     |
| include_simulator_logs                         | false                                                                                                 |
| skip_build                                     | false                                                                                                 |
| xcode_path                                     | /Applications/Xcode.app                                                                               |
+------------------------------------------------+-------------------------------------------------------------------------------------------------------+

[18:05:23]: Resetting iPhone 8
[18:05:24]: Disabling 'Slide to Type' iPhone 8
[18:05:24]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/chris.blackmore/Library/Developer/CoreSimulator/Devices/9A2F581B-43CF-4806-9F78-76AB46124EDA/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
[18:05:24]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme MutliScanDemo -project ./MultiScanDemo.xcodeproj -derivedDataPath /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg -destination 'platform=iOS Simulator,id=9A2F581B-43CF-4806-9F78-76AB46124EDA' build-for-testing | tee '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/logs/MultiScanDemo-MutliScanDemo.log' | xcpretty  --report junit --output '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/report.junit' --report junit --output '/var/folders/q5/c8mcjfnn4ns3vhvg6c97d9_r0000gq/T/junit_report20201119-47860-84j2qw' 
[18:05:24]: ▸ Loading...
[18:05:29]: ▸     Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'TestTarget1' from project 'MultiScanDemo')
[18:05:29]: ▸     Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'TestTarget3' from project 'MultiScanDemo')
[18:05:29]: ▸     Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'TestTarget2' from project 'MultiScanDemo')
[18:05:29]: ▸ Linking TestTarget3
[18:05:30]: ▸ Linking TestTarget1
[18:05:30]: ▸ Linking TestTarget2
[18:05:31]: ▸ Test build Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

[18:05:31]: Single try testing for device 'iPhone 8'
[18:05:32]: Starting scan #1.

+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+
|                                                                         Summary for scan 2.168.0                                                                          |
+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+
| buildlog_path                                  | ./build/logs                                                                                                             |
| output_directory                               | /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output                                                  |
| output_types                                   | junit                                                                                                                    |
| output_files                                   | report.junit                                                                                                             |
| scheme                                         | MutliScanDemo                                                                                                            |
| fail_build                                     | true                                                                                                                     |
| reset_simulator                                | true                                                                                                                     |
| suppress_xcode_output                          | false                                                                                                                    |
| project                                        | ./MultiScanDemo.xcodeproj                                                                                                |
| skip_detect_devices                            | false                                                                                                                    |
| force_quit_simulator                           | false                                                                                                                    |
| disable_slide_to_type                          | true                                                                                                                     |
| reinstall_app                                  | false                                                                                                                    |
| xctestrun                                      | /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg/Build/Products/Mu  |
|                                                | tliScanDemo_iphonesimulator14.0-x86_64.xctestrun                                                                         |
| clean                                          | false                                                                                                                    |
| open_report                                    | false                                                                                                                    |
| derived_data_path                              | /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg                    |
| should_zip_build_products                      | false                                                                                                                    |
| use_clang_report_name                          | false                                                                                                                    |
| disable_concurrent_testing                     | true                                                                                                                     |
| xcargs                                         |  -parallel-testing-enabled NO -resultBundlePath                                                                          |
|                                                | '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult'                                 |
| slack_use_webhook_configured_username_and_icon | false                                                                                                                    |
| slack_username                                 | fastlane                                                                                                                 |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png                                                                      |
| skip_slack                                     | false                                                                                                                    |
| slack_only_on_failure                          | false                                                                                                                    |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                                                                                        |
| include_simulator_logs                         | false                                                                                                                    |
| skip_build                                     | false                                                                                                                    |
| xcode_path                                     | /Applications/Xcode.app                                                                                                  |
| build_for_testing                              | false                                                                                                                    |
| result_bundle                                  | false                                                                                                                    |
+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+

[18:05:32]: Resetting iPhone 8
[18:05:32]: Shutting down 9A2F581B-43CF-4806-9F78-76AB46124EDA
[18:05:35]: Disabling 'Slide to Type' iPhone 8
[18:05:35]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/chris.blackmore/Library/Developer/CoreSimulator/Devices/9A2F581B-43CF-4806-9F78-76AB46124EDA/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
[18:05:35]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=9A2F581B-43CF-4806-9F78-76AB46124EDA' -derivedDataPath /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg -disable-concurrent-testing -xctestrun '/Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg/Build/Products/MutliScanDemo_iphonesimulator14.0-x86_64.xctestrun'  -parallel-testing-enabled NO -resultBundlePath '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult'  test-without-building | tee '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/logs/MultiScanDemo-MutliScanDemo.log' | xcpretty  --report junit --output '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/report.junit' --report junit --output '/var/folders/q5/c8mcjfnn4ns3vhvg6c97d9_r0000gq/T/junit_report20201119-47860-lequox' 
[18:05:35]: ▸ Loading...
[18:05:37]: ▸ 2020-11-19 18:05:37.150 xcodebuild[48030:956995]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[18:05:37]: ▸ /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget1-49EC4AE2-D810-4652-ABA9-30E867566E71/TestTarget1-0D8471C0-967F-4E7D-AF25-F1692D08C005/Session-TestTarget1-2020-11-19_180537-ovFLzl.log
[18:05:37]: ▸ 2020-11-19 18:05:37.151 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (CC4C3197-95E4-4098-A9FB-576F99869638) Beginning test session TestTarget1-CC4C3197-95E4-4098-A9FB-576F99869638 at 2020-11-19 18:05:37.151 with Xcode 12A7209 on target  {
[18:05:37]: ▸       SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Shutdown)
[18:05:37]: ▸ } (14.0 (18A372))
[18:05:46]: ▸ 2020-11-19 18:05:46.830 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (CC4C3197-95E4-4098-A9FB-576F99869638) Finished requesting crash reports. Continuing with testing.
[18:10:40]: ▸ All tests
[18:10:40]: ▸ Test Suite TestTarget1.xctest started
[18:10:40]: ▸ TestTarget1Tests
[18:10:40]: ▸     ✗ testTarget1Test, failed - This is the failure reason
[18:10:40]: ▸ TestTarget1.TestTarget1Tests
[18:10:40]: ▸   testTarget1Test, failed - This is the failure reason
[18:10:40]: ▸   /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/MultiScanDemoTests/TestTarget1Tests.swift:9
[18:10:40]: ▸   ```
[18:10:40]: ▸ [38;5;230m    [39m[38;5;221;01mfunc[39;00m[38;5;230m [39m[38;5;153mtestTarget1Test[39m[38;5;87m()[39m[38;5;230m [39m[38;5;87m{[39m[38;5;230m
[18:10:40]: ▸ [38;5;230m        [39m[38;5;155;01mXCTFail[39;00m[38;5;87m([39m[38;5;229;01m"This is the failure reason"[39;00m[38;5;87m)[39m[38;5;230m
[18:10:40]: ▸ [38;5;230m    [39m[38;5;87m}[39m[38;5;230m
[18:10:40]: ▸ [38;5;230m[39m  ```
[18:10:40]: ▸ [31m   Executed 1 test, with 1 failure (0 unexpected) in 0.204 (0.205) seconds
[18:10:40]: ▸ [0m
[18:10:41]: ▸ 2020-11-19 18:10:41.089 xcodebuild[48030:972735]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[18:10:41]: ▸ /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget2-275DC755-52B1-4CDF-B8FD-589F5B30931A/TestTarget2-C382B143-25EC-4B77-B951-5F55756F2CA0/Session-TestTarget2-2020-11-19_181041-iW9r6u.log
[18:10:41]: ▸ 2020-11-19 18:10:41.089 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (16D071A1-66A5-4848-9B88-7326629883F7) Beginning test session TestTarget2-16D071A1-66A5-4848-9B88-7326629883F7 at 2020-11-19 18:10:41.089 with Xcode 12A7209 on target  {
[18:10:41]: ▸       SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Booted)
[18:10:41]: ▸ } (14.0 (18A372))
[18:10:41]: ▸ 2020-11-19 18:10:41.092 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (16D071A1-66A5-4848-9B88-7326629883F7) Finished requesting crash reports. Continuing with testing.
[18:10:41]: ▸ All tests
[18:10:41]: ▸ Test Suite TestTarget2.xctest started
[18:10:41]: ▸ TestTarget2Tests
[18:10:41]: ▸     ✗ testTarget2Test, failed - This is the failure reason
[18:10:41]: ▸ TestTarget2.TestTarget2Tests
[18:10:41]: ▸   testTarget2Test, failed - This is the failure reason
[18:10:41]: ▸   /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/MultiScanDemoTests/TestTarget2Tests.swift:10
[18:10:41]: ▸   ```
[18:10:41]: ▸ [38;5;230m    [39m[38;5;221;01mfunc[39;00m[38;5;230m [39m[38;5;153mtestTarget2Test[39m[38;5;87m()[39m[38;5;230m [39m[38;5;87m{[39m[38;5;230m
[18:10:41]: ▸ [38;5;230m        [39m[38;5;155;01mXCTFail[39;00m[38;5;87m([39m[38;5;229;01m"This is the failure reason"[39;00m[38;5;87m)[39m[38;5;230m
[18:10:41]: ▸ [38;5;230m    [39m[38;5;87m}[39m[38;5;230m
[18:10:41]: ▸ [38;5;230m[39m  ```
[18:10:41]: ▸ [31m   Executed 1 test, with 1 failure (0 unexpected) in 0.028 (0.030) seconds
[18:10:41]: ▸ [0m
[18:10:41]: ▸ 2020-11-19 18:10:41.884 xcodebuild[48030:972892]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[18:10:41]: ▸ /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget3-2DC0E65C-57F2-4101-AAB9-288217225846/TestTarget3-77DE35D4-55A0-4B39-94D5-8646DD34CE74/Session-TestTarget3-2020-11-19_181041-IP0ZO4.log
[18:10:41]: ▸ 2020-11-19 18:10:41.884 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE) Beginning test session TestTarget3-B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE at 2020-11-19 18:10:41.885 with Xcode 12A7209 on target  {
[18:10:41]: ▸       SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Booted)
[18:10:41]: ▸ } (14.0 (18A372))
[18:10:41]: ▸ 2020-11-19 18:10:41.888 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE) Finished requesting crash reports. Continuing with testing.
[18:10:42]: ▸ All tests
[18:10:42]: ▸ Test Suite TestTarget3.xctest started
[18:10:42]: ▸ TestTarget3Tests
[18:10:42]: ▸     ✓ testTarget3Test (0.002 seconds)
[18:10:42]: ▸ [32;1m     Executed 1 test, with 0 failures (0 unexpected) in 0.002 (0.004) seconds
[18:10:42]: ▸ [0m
[18:10:57]: ▸ 2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 320.303 elapsed -- Testing started completed.
[18:10:57]: ▸ 2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[18:10:57]: ▸ 2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 320.303 sec, +320.303 sec -- end
[18:10:57]: ▸ Failing tests:
[18:10:57]: ▸   TestTarget1:
[18:10:57]: ▸       TestTarget1Tests.testTarget1Test()
[18:10:57]: ▸   TestTarget2:
[18:10:57]: ▸       TestTarget2Tests.testTarget2Test()
[18:10:57]: ▸ ** TEST EXECUTE FAILED **
2020-11-19 18:05:37.150 xcodebuild[48030:956995]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget1-49EC4AE2-D810-4652-ABA9-30E867566E71/TestTarget1-0D8471C0-967F-4E7D-AF25-F1692D08C005/Session-TestTarget1-2020-11-19_180537-ovFLzl.log
2020-11-19 18:05:37.151 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (CC4C3197-95E4-4098-A9FB-576F99869638) Beginning test session TestTarget1-CC4C3197-95E4-4098-A9FB-576F99869638 at 2020-11-19 18:05:37.151 with Xcode 12A7209 on target  {
        SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Shutdown)
} (14.0 (18A372))
2020-11-19 18:05:46.830 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (CC4C3197-95E4-4098-A9FB-576F99869638) Finished requesting crash reports. Continuing with testing.
All tests
Test Suite TestTarget1.xctest started
TestTarget1Tests
    ✗ testTarget1Test, failed - This is the failure reason


TestTarget1.TestTarget1Tests
  testTarget1Test, failed - This is the failure reason
  /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/MultiScanDemoTests/TestTarget1Tests.swift:9
  ```
    func testTarget1Test() {
        XCTFail("This is the failure reason")
    }
  ```


     Executed 1 test, with 1 failure (0 unexpected) in 0.204 (0.205) seconds

2020-11-19 18:10:41.089 xcodebuild[48030:972735]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget2-275DC755-52B1-4CDF-B8FD-589F5B30931A/TestTarget2-C382B143-25EC-4B77-B951-5F55756F2CA0/Session-TestTarget2-2020-11-19_181041-iW9r6u.log
2020-11-19 18:10:41.089 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (16D071A1-66A5-4848-9B88-7326629883F7) Beginning test session TestTarget2-16D071A1-66A5-4848-9B88-7326629883F7 at 2020-11-19 18:10:41.089 with Xcode 12A7209 on target  {
        SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Booted)
} (14.0 (18A372))
2020-11-19 18:10:41.092 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (16D071A1-66A5-4848-9B88-7326629883F7) Finished requesting crash reports. Continuing with testing.
All tests
Test Suite TestTarget2.xctest started
TestTarget2Tests
    ✗ testTarget2Test, failed - This is the failure reason


TestTarget2.TestTarget2Tests
  testTarget2Test, failed - This is the failure reason
  /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/MultiScanDemoTests/TestTarget2Tests.swift:10
  ```
    func testTarget2Test() {
        XCTFail("This is the failure reason")
    }
  ```


     Executed 1 test, with 1 failure (0 unexpected) in 0.028 (0.030) seconds

2020-11-19 18:10:41.884 xcodebuild[48030:972892]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget3-2DC0E65C-57F2-4101-AAB9-288217225846/TestTarget3-77DE35D4-55A0-4B39-94D5-8646DD34CE74/Session-TestTarget3-2020-11-19_181041-IP0ZO4.log
2020-11-19 18:10:41.884 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE) Beginning test session TestTarget3-B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE at 2020-11-19 18:10:41.885 with Xcode 12A7209 on target  {
        SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Booted)
} (14.0 (18A372))
2020-11-19 18:10:41.888 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: (B043A64A-6E6B-4D81-B1C3-7B5DD54AEDCE) Finished requesting crash reports. Continuing with testing.
All tests
Test Suite TestTarget3.xctest started
TestTarget3Tests
    ✓ testTarget3Test (0.002 seconds)


     Executed 1 test, with 0 failures (0 unexpected) in 0.002 (0.004) seconds

2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 320.303 elapsed -- Testing started completed.
2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2020-11-19 18:10:57.439 xcodebuild[48030:956921] [MT] IDETestOperationsObserverDebug: 320.303 sec, +320.303 sec -- end
Failing tests:
    TestTarget1:
        TestTarget1Tests.testTarget1Test()
    TestTarget2:
        TestTarget2Tests.testTarget2Test()

** TEST EXECUTE FAILED **
[18:11:00]: Exit status: 65
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 3 |
| Number of failures | 2 |
+--------------------+---+

[18:11:02]: Testing batches for device 'iPhone 8'
[18:11:02]: Starting test run 1
[18:11:06]: Starting scan #1 with 1 tests.

+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+
|                                                                         Summary for scan 2.168.0                                                                          |
+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+
| buildlog_path                                  | ./build/logs                                                                                                             |
| output_directory                               | /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output                                                  |
| output_types                                   | junit                                                                                                                    |
| output_files                                   | report.junit                                                                                                             |
| scheme                                         | MutliScanDemo                                                                                                            |
| fail_build                                     | true                                                                                                                     |
| reset_simulator                                | true                                                                                                                     |
| suppress_xcode_output                          | false                                                                                                                    |
| project                                        | ./MultiScanDemo.xcodeproj                                                                                                |
| skip_detect_devices                            | false                                                                                                                    |
| force_quit_simulator                           | false                                                                                                                    |
| disable_slide_to_type                          | true                                                                                                                     |
| reinstall_app                                  | false                                                                                                                    |
| only_testing                                   | ["TestTarget3/TestTarget2Tests"]                                                                                         |
| xctestrun                                      | /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg/Build/Products/Mu  |
|                                                | tliScanDemo_iphonesimulator14.0-x86_64.xctestrun                                                                         |
| clean                                          | false                                                                                                                    |
| open_report                                    | false                                                                                                                    |
| derived_data_path                              | /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg                    |
| should_zip_build_products                      | false                                                                                                                    |
| use_clang_report_name                          | false                                                                                                                    |
| disable_concurrent_testing                     | true                                                                                                                     |
| xcargs                                         |  -parallel-testing-enabled NO -resultBundlePath                                                                          |
|                                                | '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult'                                 |
| slack_use_webhook_configured_username_and_icon | false                                                                                                                    |
| slack_username                                 | fastlane                                                                                                                 |
| slack_icon_url                                 | https://fastlane.tools/assets/img/fastlane_icon.png                                                                      |
| skip_slack                                     | false                                                                                                                    |
| slack_only_on_failure                          | false                                                                                                                    |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                                                                                        |
| include_simulator_logs                         | false                                                                                                                    |
| skip_build                                     | false                                                                                                                    |
| xcode_path                                     | /Applications/Xcode.app                                                                                                  |
| build_for_testing                              | false                                                                                                                    |
| result_bundle                                  | false                                                                                                                    |
+------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+

[18:11:06]: Resetting iPhone 8
[18:11:06]: Shutting down 9A2F581B-43CF-4806-9F78-76AB46124EDA
[18:11:09]: Disabling 'Slide to Type' iPhone 8
[18:11:09]: $ /usr/libexec/PlistBuddy -c "Add :KeyboardContinuousPathEnabled bool false" /Users/chris.blackmore/Library/Developer/CoreSimulator/Devices/9A2F581B-43CF-4806-9F78-76AB46124EDA/data/Library/Preferences/com.apple.keyboard.ContinuousPath.plist >/dev/null 2>&1
[18:11:09]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=9A2F581B-43CF-4806-9F78-76AB46124EDA' -derivedDataPath /Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg -disable-concurrent-testing -xctestrun '/Users/chris.blackmore/Library/Developer/Xcode/DerivedData/MultiScanDemo-cheknywehazmorgabtneeyuhudwg/Build/Products/MutliScanDemo_iphonesimulator14.0-x86_64.xctestrun'  -parallel-testing-enabled NO -resultBundlePath '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult'  -only-testing:TestTarget3/TestTarget2Tests test-without-building | tee '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/logs/MultiScanDemo-MutliScanDemo.log' | xcpretty  --report junit --output '/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/report.junit' --report junit --output '/var/folders/q5/c8mcjfnn4ns3vhvg6c97d9_r0000gq/T/junit_report20201119-47860-zusytk' 
[18:11:09]: ▸ Loading...
[18:11:11]: ▸ 2020-11-19 18:11:11.527 xcodebuild[48397:978064]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[18:11:11]: ▸ /Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult/Staging/1_Test/Diagnostics/TestTarget3-A105EDC7-2929-4F19-890D-C6336296B2F6/TestTarget3-4F8C1BCA-051D-4893-9731-61E40211A45B/Session-TestTarget3-2020-11-19_181111-3Va11W.log
[18:11:11]: ▸ 2020-11-19 18:11:11.528 xcodebuild[48397:977936] [MT] IDETestOperationsObserverDebug: (C5C39FDF-A266-41D1-949C-1A090CD14315) Beginning test session TestTarget3-C5C39FDF-A266-41D1-949C-1A090CD14315 at 2020-11-19 18:11:11.528 with Xcode 12A7209 on target  {
[18:11:11]: ▸       SimDevice: iPhone 8 (9A2F581B-43CF-4806-9F78-76AB46124EDA, iOS 14.0, Shutdown)
[18:11:11]: ▸ } (14.0 (18A372))
[18:11:21]: ▸ 2020-11-19 18:11:21.284 xcodebuild[48397:977936] [MT] IDETestOperationsObserverDebug: (C5C39FDF-A266-41D1-949C-1A090CD14315) Finished requesting crash reports. Continuing with testing.
[18:12:14]: ▸ Selected tests
[18:12:14]: ▸ Test Suite TestTarget3.xctest started
[18:12:14]: ▸ [32;1m     Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
[18:12:14]: ▸ [0m
[18:12:14]: ▸ 2020-11-19 18:12:14.307 xcodebuild[48397:977936] [MT] IDETestOperationsObserverDebug: 62.794 elapsed -- Testing started completed.
[18:12:14]: ▸ 2020-11-19 18:12:14.307 xcodebuild[48397:977936] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[18:12:14]: ▸ 2020-11-19 18:12:14.307 xcodebuild[48397:977936] [MT] IDETestOperationsObserverDebug: 62.794 sec, +62.794 sec -- end
[18:12:14]: ▸ Test execute Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+


+-------------------+-----------------------------------------------------------------------------------------------+
|                                                multi_scan results                                                 |
+-------------------+-----------------------------------------------------------------------------------------------+
| result            | true                                                                                          |
| total_tests       | 0                                                                                             |
| passing_testcount | 0                                                                                             |
| failed_testcount  | 0                                                                                             |
| failed_tests      | []                                                                                            |
| total_retry_count | 1                                                                                             |
| report_files      | ["/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/report.junit",      |
|                   | "/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.test_result",  |
|                   | "/Users/chris.blackmore/Desktop/MultiScanDemo/MultiScanDemo/build/output/Debug.xcresult"]     |
+-------------------+-----------------------------------------------------------------------------------------------+


+------+------------+-------------+
|        fastlane summary         |
+------+------------+-------------+
| Step | Action     | Time (in s) |
+------+------------+-------------+
| 1    | multi_scan | 421         |
+------+------------+-------------+

[18:12:20]: fastlane.tools just saved you 7 minutes! 🎉
asglh-ml-13805:MutliScanDemo chris.blackmore$ 

Environment

✅ fastlane environment ✅


Stack

| Key                         | Value                                       |

| --------------------------- | ------------------------------------------- |

| OS                          | 10.15.5                                     |

| Ruby                        | 2.6.5                                       |

| Bundler?                    | false                                       |

| Git                         | git version 2.24.3 (Apple Git-128)          |

| Installation Source         | ~/.rbenv/versions/2.6.5/bin/fastlane        |

| Host                        | Mac OS X 10.15.5 (19F96)                    |

| Ruby Lib Dir                | ~/.rbenv/versions/2.6.5/lib                 |

| OpenSSL Version             | OpenSSL 1.1.1d  10 Sep 2019                 |

| Is contained                | false                                       |

| Is homebrew                 | false                                       |

| Is installed via Fabric.app | false                                       |

| Xcode Path                  | /Applications/Xcode.app/Contents/Developer/ |

| Xcode Version               | 12.0                                        |

System Locale

| Variable | Value       |   |

| -------- | ----------- | - |

| LANG     | en_GB.UTF-8 | ✅ |

| LC_ALL   |             |   |

| LANGUAGE |             |   |

fastlane files:

./.fastlane/Fastfile


platform :ios do
  desc "Runs tests"
  lane :unit_test do |options|  
    result = multi_scan(
      buildlog_path: "./build/logs",
      collate_reports: true,
      invocation_based_tests: true,
      output_directory: "./build/output",
      result_bundle: true,
      output_types: 'junit,xcresult',
      output_files: 'report.junit,Debug.xcresult',
      scheme: "MutliScanDemo",
      try_count: 3,
            fail_build: true,
            reset_simulator: true,
            suppress_xcode_output: false
    )

    if result[:failed_testcount] > 0
      UI.test_failure! "There are #{result[:failed_testcount]} failing tests :("
    end
  end
end




No Appfile found

fastlane gems

| Gem      | Version | Update-Status |

| -------- | ------- | ------------- |

| fastlane | 2.168.0 | ✅ Up-To-Date  |

Loaded fastlane plugins:

| Plugin                      | Version | Update-Status |

| --------------------------- | ------- | ------------- |

| fastlane-plugin-test_center | 3.14.8  | ✅ Up-To-Date  |

Loaded gems


| Gem                         | Version      |

| --------------------------- | ------------ |

| did_you_mean                | 1.3.0        |

| slack-notifier              | 2.3.2        |

| rouge                       | 2.0.7        |

| xcpretty                    | 0.3.0        |

| terminal-notifier           | 2.0.0        |

| terminal-table              | 1.8.0        |

| plist                       | 3.5.0        |

| CFPropertyList              | 3.0.2        |

| addressable                 | 2.7.0        |

| multipart-post              | 2.0.0        |

| word_wrap                   | 1.0.0        |

| colored                     | 1.2          |

| highline                    | 1.7.10       |

| commander-fastlane          | 4.4.6        |

| ruby2_keywords              | 0.0.2        |

| faraday                     | 1.1.0        |

| faraday_middleware          | 1.0.0        |

| gh_inspector                | 1.1.3        |

| rubyzip                     | 2.3.0        |

| security                    | 0.1.3        |

| xcpretty-travis-formatter   | 1.0.0        |

| bundler                     | 2.1.4        |

| uber                        | 0.1.0        |

| declarative-option          | 0.1.0        |

| representable               | 3.0.4        |

| retriable                   | 3.1.2        |

| mini_mime                   | 1.0.2        |

| httpclient                  | 2.8.3        |

| google-api-client           | 0.38.0       |

| nanaimo                     | 0.3.0        |

| colored2                    | 3.1.2        |

| claide                      | 1.0.3        |

| atomos                      | 0.1.3        |

| xcodeproj                   | 1.19.0       |

| unicode-display_width       | 1.7.0        |

| public_suffix               | 4.0.6        |

| tty-screen                  | 0.8.1        |

| tty-cursor                  | 0.7.1        |

| tty-spinner                 | 0.9.3        |

| babosa                      | 1.0.4        |

| excon                       | 0.78.0       |

| unf_ext                     | 0.0.7.7      |

| unf                         | 0.1.4        |

| domain_name                 | 0.5.20190701 |

| http-cookie                 | 1.0.3        |

| faraday-cookie_jar          | 0.0.7        |

| fastimage                   | 2.2.0        |

| json                        | 2.3.1        |

| mini_magick                 | 4.11.0       |

| dotenv                      | 2.7.6        |

| naturally                   | 2.2.0        |

| simctl                      | 1.6.8        |

| jwt                         | 2.2.2        |

| declarative                 | 0.0.20       |

| multi_json                  | 1.15.0       |

| signet                      | 0.14.0       |

| os                          | 1.1.1        |

| memoist                     | 0.16.2       |

| googleauth                  | 0.14.0       |

| rake                        | 13.0.1       |

| digest-crc                  | 0.6.1        |

| google-cloud-errors         | 1.0.1        |

| google-cloud-env            | 1.4.0        |

| google-cloud-core           | 1.5.0        |

| google-cloud-storage        | 1.29.1       |

| emoji_regex                 | 3.2.1        |

| aws-eventstream             | 1.1.0        |

| aws-sigv4                   | 1.2.2        |

| aws-partitions              | 1.395.0      |

| jmespath                    | 1.4.0        |

| aws-sdk-core                | 3.109.3      |

| aws-sdk-kms                 | 1.39.0       |

| aws-sdk-s3                  | 1.84.1       |

| forwardable                 | 1.2.0        |

| logger                      | 1.3.0        |

| stringio                    | 0.0.2        |

| ipaddr                      | 1.2.2        |

| openssl                     | 2.1.2        |

| ostruct                     | 0.1.0        |

| strscan                     | 1.0.0        |

| date                        | 2.0.0        |

| fileutils                   | 1.1.0        |

| etc                         | 1.0.1        |

| io-console                  | 0.4.7        |

| zlib                        | 1.0.0        |

| mini_portile2               | 2.4.0        |

| nokogiri                    | 1.10.7       |

| rexml                       | 3.2.4        |

| psych                       | 3.1.0        |

| mutex_m                     | 0.1.0        |

| oj                          | 3.10.6       |

| bigdecimal                  | 1.4.1        |

| webrick                     | 1.4.2        |

| trainer                     | 0.9.1        |

| xctest_list                 | 1.2.1        |

| colorize                    | 0.8.1        |

| fastlane-plugin-test_center | 3.14.8       |



generated on: 2020-11-19

[lyndsey-ferguson]

Thanks for attaching the sample project. I'm sure that will help me track down and fix the problem faster.

[lyndsey-ferguson]

In the meantime, I suggest running multi_scan once for each target with the only_testing parameter set to the name of the test target. Sorry for the inconvenience.

[asos-chrisblackmore]

Thanks for the suggestion @lyndsey-ferguson! Unfortunately one of the targets in question has a substantial amount of unit tests in it, and providing the target to only_testing results in all test functions being passed along, which unfortunately generates an error: Argument list too long - fork failed. I've had to disable retrying until the issue is resolved.

Please reach out if you'd like any help to test any potential fixes available! :)

[lyndsey-ferguson]

@asos-chrisblackmore if you have tons of tests, then I suggest running them in batches. Try with a batch_count: 10 to see how that works for you (or even more, I have no idea how many tests are in your targets).

[lyndsey-ferguson]

There is a problem with retrying tests with invocation_based_tests (#281). The sample you provided isn't an invocation test based project. Are you certain that you want to use that option?

    result = multi_scan(
      buildlog_path: "./build/logs",
      collate_reports: true,
      invocation_based_tests: true,
      output_directory: "./build/output",
      result_bundle: true,
      output_types: 'junit,xcresult',
      output_files: 'report.junit,Debug.xcresult',
      scheme: "MutliScanDemo",
      try_count: 3,
      fail_build: true,
      reset_simulator: true,
      suppress_xcode_output: false
    )

If I remove that option from the sample, everything seems to work fine, except for the final report of the passing/failing tests.

[lyndsey-ferguson]

I've created issue #333 for the failed tests summary.

[lyndsey-ferguson]

@asos-chrisblackmore please let me know if removing the option :invocation_based_tests works for you. If you _do_ have invocation tests, I apologize but I do not have a workaround to properly retry those tests at the moment.

[asos-chrisblackmore]

@lyndsey-ferguson apologies for the delay in getting back to you!

Disabling invocation_based_tests does indeed solve this particular issue, thank you for the suggestion!

The junit (report.junit) looks correct, but there's now xcresult files per-target, which isn't ideal. I'd expect them to be collated like they would previously. I'm assuming that's a separate issue?

[iammike]

Just a note, invocation based tests also worked for getting me from 0 tests to all my tests. Unfortunately, I'm on Circle and it seems to ignore parallel_testrun_count so end of the line for me :(

[lyndsey-ferguson]

@asos-chrisblackmore yes, the failure to collate xcresult files for separate targets would be a separate issue, in fact, it would have to be a feature request as I didn't build the collation to handle separate targets. You _could_ achieve the same result with the xcresulttool and the merge command. This Ruby code may help guide you if you wanted to use it in your fastlane.

@iammike I do think that CircleCI does not let you spin up additional simulators. I am not familiar with CircleCI (I tried to sign up, but they required access to my employer's private repos and I wasn't comfortable giving them that), so I cannot tell you an exact workaround, but if you can, I suggest:

Create a stage that uses gets the tests (maybe using tests_from_xctestrun) and then slices that list of tests into batches. For each batch, your pipeline schedules a job to test just the tests from its batch.

Profit!

[iammike]

Yeah, working with Circle has been a nightmare. They do so much weird stuff regarding Xcode/don't support much of anything "fun" or useful w/ XCUITest. Thinking we'll migrate off it soon. Thanks for the idea, unfortunately out of time for this spike right now so will file it away and try again later.

[lyndsey-ferguson]

@iammike Ug, sorry to hear that you're having problems with CircleCI 😢

[lyndsey-ferguson]

@marcomorain I thought you may want to bubble @iammike's bad experience up to the CircleCI product managers.

[marcomorain]

Thanks I’ll pass this thread on 👍

[iammike]

Thanks I’ll pass this thread on 👍

@marcomorain Ticket 86853 🤗

[asos-chrisblackmore]

@lyndsey-ferguson using your collate_xcresults lane would solve that particular problem no? Using something like Dir.glob("../build/output/**/*.xcresult") to make sure all xcresult files (regardless of the file structure) are pulled in.

EDIT: I'm referring to myself adding the above as an extra step after invoking multi_scan - not suggesting on how to fix it within multi_scan itself :)

[lyndsey-ferguson]

@asos-chrisblackmore I _think_ that should work as it is basically the same thing

[asos-chrisblackmore]

Yup, tried it, works perfectly. If you're happy for this issue to continue under https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/281 (albeit manifesting differently) then feel free to close this. Appreciate your investigation and suggestions :)

[lyndsey-ferguson]

Wait, are you saying that there isn't anything more to do here?
🤯

[asos-chrisblackmore]

Removing invocation_based_tests is a good enough workaround for my use case, test retrying correctly fails for multiple targets when doing so!

There still is an issue, but like I said, if you're happy for that to continue under https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/281 then go for it. I'm not too if a fix would address both problems or not. I'll leave it with you!

[lyndsey-ferguson]

My plan for #281 is to fix it so that the failing test class is retested for invocation based tests. I'm closing this ticket for now.

Thanks!

[asos-chrisblackmore]

Hey @lyndsey-ferguson - jus an FYI that although removing invocation_based_tests does solve the problem, supplying a batch_count drastically slows down the test run. It batches per target - so with 10 targets, and a batch count of 5, would result in 50 batches. There seems to be a little overhead with batches, and when you consider 1000's of tests, it does become an issue in terms of speed/performance.

[lyndsey-ferguson]

I moved this to a discussion