Fastlane-plugin-test_center: Feature Idea: Run tests in parallel
Hello!
I found this plugin very useful. But it misses one of the XCode 9 features: parallel run of the tests. I wonder, if it is possible to add such feature to the plugin? It will reduce tests' run time a lot.
Many thanks!
kisialiu
All 143 comments
Thanks for the feedback!
Running tests in parallel on my list. Once I am finished working on the current feature, I'll start that.
lyndsey-ferguson
on 22 Jan 2018
I'm excited for this mysterious feature :P
zacoid55
on 22 Jan 2018
this code seems to indicate that the same tests can be run on multiple different simulators. Not that I can split up the tests. There must be a way to specify batches of tests to run on multiple simulators.
lyndsey-ferguson
on 2 Aug 2018
This article writes that "From Xcode 9 and later we are able not only to run the same test on different devices simultaneously, but also different set of tests on different devices"
This article takes the painful approach of creating a new test scheme for each "batch" and enabling and disabling tests to select which tests run in each scheme.
This article indicates that one has to make multiple calls to xcodebuild with the scheme and destination and send the process to the background with &.
_Interesting note:_ it seems that if one does not start up the simulators prior to running the xcodebuild command, the iOS Simulator will run in _headless_ mode.
lyndsey-ferguson
on 2 Aug 2018
It seems we have to duplicate the iOS Simulator that the user selected.
lyndsey-ferguson
on 2 Aug 2018
Ok, it looks like I'll have to spin up separate processes. I would have to shard the tests (using the batch option) then create a separate process for each batch to call the correcting_scan_helper for each batch. That way we get the re-try power for each batch of tests. Wooo, sounds exciting 🎉 .
lyndsey-ferguson
on 2 Aug 2018
Fastlane::FastFile.new.parse("lane :batched_multi_scan do
multi_scan(
...
)
end").runner.execute(:batched_multi_scan)
I believe that I have it basically working, but there are problems with scan failing due to the Simulator never finishing the bootstrap. There _may_ be some people that have dealt with this who tried to run tests in parallel. Will look at this next time I work on this feature.
lyndsey-ferguson
on 9 Aug 2018
Here is the log so that I do not have to re-run it just to review it every day when I get time to work on this:
bef parallel
[✔] 🚀
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
| Used plugins |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
| Plugin | Version | Action |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
| fastlane-plugin-test_center | 3.5.3 | suppressed_tests suppress_tests suppress_tests_from_junit tests_from_xctestrun collate_test_result_bundles |
| | | tests_from_junit multi_scan collate_html_reports collate_junit_reports collate_json_reports |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
[09:30:14]: You have require'd a gem, if this is a third party gem, please use `fastlane_require 'pry-byebug'` to ensure the gem is installed locally.
[09:30:14]: Driving the lane 'parallel' 🚀
[09:30:14]: ------------------------
[09:30:14]: --- Step: multi_scan ---
[09:30:14]: ------------------------
+----------------------+----------------------+
| Summary for multi_scan (test_center v3.5.3) |
+----------------------+----------------------+
| try_count | 3 |
| batch_count | 2 |
| fail_build | false |
+----------------------+----------------------+
[09:30:14]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:14]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:17]: Found simulator "iPhone 5s (12.0)"
[09:30:17]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:17]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:19]: Found simulator "iPhone 5s (12.0)"
+----------------------------+----------------------------------------------------------------------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+----------------------------------------------------------------------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj |
| scheme | AtomicBoy |
| fail_build | false |
| build_for_testing | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr |
| clean | true |
| skip_build | false |
| output_directory | ./fastlane/test_output |
| output_types | html,junit |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+----------------------------------------------------------------------------------------------------+
[09:30:19]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=67C5D0A4-A1A8-4407-8152-EEB28BE02C27' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' clean build-for-testing | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-14852-18ahzgo'
[09:30:19]: ▸ Loading...
[09:30:20]: ▸ Clean Succeeded
[09:30:21]: ▸ Compiling GameScene.m
[09:30:21]: ▸ Compiling GameViewController.m
[09:30:21]: ▸ Compiling main.m
[09:30:21]: ▸ Compiling AppDelegate.m
[09:30:21]: ▸ Linking AtomicBoy
[09:30:21]: ▸ Compiling TestObject.swift
[09:30:22]: ▸ Compiling SwiftAtomicBoyUITests.swift
[09:30:22]: ▸ Copying /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy/GameScene.sks
[09:30:22]: ▸ Copying /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy/Actions.sks
[09:30:23]: ▸ Compiling Main.storyboard
[09:30:25]: ▸ Compiling LaunchScreen.storyboard
[09:30:25]: ▸ Compiling AtomicBoyUITests.m
[09:30:25]: ▸ Processing Info.plist
[09:30:25]: Running Tests: ▸ Touching AtomicBoy.app (in target: AtomicBoy)
[09:30:25]: ▸ Processing Info.plist
[09:30:25]: ▸ Linking AtomicBoyUITests
[09:30:25]: ▸ Processing Info.plist
[09:30:25]: ▸ Compiling AtomicBoyTests.m
[09:30:25]: ▸ Linking AtomicBoyTests
[09:30:26]: ▸ Copying /Applications/Xcode-10.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks/XCTest.framework
[09:30:26]: ▸ Copying /Applications/Xcode-10.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/lib/libXCTestBundleInject.dylib
[09:30:26]: Running Tests: ▸ Touching AtomicBoyTests.xctest (in target: AtomicBoyTests)
[09:30:27]: ▸ Copying /Applications/Xcode-10.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/Frameworks/XCTest.framework
[09:30:27]: ▸ Copying /Applications/Xcode-10.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Library/PrivateFrameworks/XCTAutomationSupport.framework
[09:30:27]: Running Tests: ▸ Touching AtomicBoyUITests.xctest (in target: AtomicBoyUITests)
[09:30:27]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+
[09:30:27]: ------------------------------------------------------
[09:30:27]: --- Starting test run on testable 'AtomicBoyTests' ---
[09:30:27]: ------------------------------------------------------
[09:30:27]: ------------------------------------------------------
[09:30:27]: --- Starting test run on testable 'AtomicBoyTests' ---
[09:30:27]: ------------------------------------------------------
================================================================================
--------------------------------------------------------------------------------
[09:30:27]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:27]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:30]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:30]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-1 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyTests/AtomicBoyTests/tes |
| | tExample"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyTests |
| output_types | html,junit |
| output_files | report.html,report.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:32]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=59E8A537-CBAD-470D-9480-6780EFE16934' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyTests/AtomicBoyTests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15058-1i9z6s5'
[09:30:32]: ▸ Loading...
[09:30:34]: ▸ 2018-08-10 09:30:34.116 xcodebuild[15115:2869368] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:34]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-34--0400.xcresult/1_Test/Diagnostics/AtomicBoyTests-622AE983-B489-46F3-8794-A0A2DF29DE1C/AtomicBoyTests-F79EC9BF-D312-4768-A637-FF108C02B260/Session-AtomicBoyTests-2018-08-10_093034-DDEvVP.log
[09:30:34]: ▸ 2018-08-10 09:30:34.117 xcodebuild[15115:2869360] [MT] IDETestOperationsObserverDebug: (C893BDC4-6E4B-4061-B553-45A6184A77E5) Beginning test session AtomicBoyTests-C893BDC4-6E4B-4061-B553-45A6184A77E5 at 2018-08-10 09:30:34.117 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7fce817d1f70> {
[09:30:34]: ▸ SimDevice: iPhone 5s-1 (59E8A537-CBAD-470D-9480-6780EFE16934, iOS 11.1, Booted)
[09:30:34]: ▸ } (11.1 (15B87))
[09:30:36]: ▸ Selected tests
[09:30:36]: ▸ Test Suite AtomicBoyTests.xctest started
[09:30:36]: ▸ AtomicBoyTests
[09:30:36]: ▸ ✓ testExample (0.001 seconds)
[09:30:36]: ▸ Executed 1 test, with 0 failures (0 unexpected) in 0.001 (0.003) seconds
[09:30:36]: ▸
[09:30:36]: ▸ 2018-08-10 09:30:36.793 xcodebuild[15115:2869360] [MT] IDETestOperationsObserverDebug: 2.752 elapsed -- Testing started completed.
[09:30:36]: ▸ 2018-08-10 09:30:36.793 xcodebuild[15115:2869360] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:30:36]: ▸ 2018-08-10 09:30:36.793 xcodebuild[15115:2869360] [MT] IDETestOperationsObserverDebug: 2.752 sec, +2.752 sec -- end
[09:30:36]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 0 |
+--------------------+---+
--------------------------------------------------------------------------------
[09:30:27]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:27]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:30]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:30]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-2 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyTests/AtomicBoyTests/tes |
| | tPerformanceExample"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyTests |
| output_types | html,junit |
| output_files | report-2.html,report-2.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:32]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=612BA9D1-4929-493A-BE0E-FA034F022B4F' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyTests/AtomicBoyTests/testPerformanceExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report-2.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report-2.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15059-5r8oh7'
[09:30:32]: ▸ Loading...
[09:30:34]: ▸ 2018-08-10 09:30:34.110 xcodebuild[15112:2869371] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:34]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-34--0400.xcresult/1_Test/Diagnostics/AtomicBoyTests-8AF10262-9308-4911-9D30-F7C9F21691C8/AtomicBoyTests-37E4C659-E69E-46D4-9197-75B94E111D84/Session-AtomicBoyTests-2018-08-10_093034-bNaEqS.log
[09:30:34]: ▸ 2018-08-10 09:30:34.110 xcodebuild[15112:2869359] [MT] IDETestOperationsObserverDebug: (6934DBAA-6715-46FC-BCEA-49FEB0BF609F) Beginning test session AtomicBoyTests-6934DBAA-6715-46FC-BCEA-49FEB0BF609F at 2018-08-10 09:30:34.110 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7f812a340fd0> {
[09:30:34]: ▸ SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
[09:30:34]: ▸ } (11.1 (15B87))
[09:30:36]: ▸ Selected tests
[09:30:36]: ▸ Test Suite AtomicBoyTests.xctest started
[09:30:36]: ▸ AtomicBoyTests
[09:30:36]: ▸ ◷ testPerformanceExample measured (0.000 seconds)
[09:30:36]: ▸ ✓ testPerformanceExample (0.262 seconds)
[09:30:36]: ▸ Executed 1 test, with 0 failures (0 unexpected) in 0.262 (0.264) seconds
[09:30:36]: ▸
[09:30:37]: ▸ 2018-08-10 09:30:37.154 xcodebuild[15112:2869359] [MT] IDETestOperationsObserverDebug: 3.066 elapsed -- Testing started completed.
[09:30:37]: ▸ 2018-08-10 09:30:37.155 xcodebuild[15112:2869359] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:30:37]: ▸ 2018-08-10 09:30:37.155 xcodebuild[15112:2869359] [MT] IDETestOperationsObserverDebug: 3.066 sec, +3.066 sec -- end
[09:30:37]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 0 |
+--------------------+---+
================================================================================
[09:30:37]: --------------------------------------------------------
[09:30:37]: --------------------------------------------------------
[09:30:37]: --- Starting test run on testable 'AtomicBoyUITests' ---
[09:30:37]: --- Starting test run on testable 'AtomicBoyUITests' ---
[09:30:37]: --------------------------------------------------------
[09:30:37]: --------------------------------------------------------
================================================================================
--------------------------------------------------------------------------------
[09:30:37]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:37]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:39]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:39]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-1 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample2"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyUITests |
| output_types | html,junit |
| output_files | report.html,report.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:42]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=59E8A537-CBAD-470D-9480-6780EFE16934' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample2 test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15138-df90hm'
[09:30:42]: ▸ Loading...
[09:30:43]: ▸ 2018-08-10 09:30:43.754 xcodebuild[15196:2870883] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:43]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-43--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-44B38688-85F5-4F8A-A0FD-BE2D35C4557C/AtomicBoyUITests-30BD73CB-D56E-4D50-BEDD-0382E9E5C06F/Session-AtomicBoyUITests-2018-08-10_093043-OCA5LO.log
[09:30:43]: ▸ 2018-08-10 09:30:43.754 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: (4A88164B-1D2F-44F9-B31C-FDD2BAB16E13) Beginning test session AtomicBoyUITests-4A88164B-1D2F-44F9-B31C-FDD2BAB16E13 at 2018-08-10 09:30:43.754 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7fa438c461a0> {
[09:30:43]: ▸ SimDevice: iPhone 5s-1 (59E8A537-CBAD-470D-9480-6780EFE16934, iOS 11.1, Booted)
[09:30:43]: ▸ } (11.1 (15B87))
[09:30:47]: ▸ Selected tests
[09:30:47]: ▸ Test Suite AtomicBoyUITests.xctest started
[09:30:47]: ▸ AtomicBoyUITests
[09:30:51]: ▸ ✗ testExample2, ((false) is true) failed
[09:30:51]: ▸ AtomicBoyUITests
[09:30:51]: ▸ testExample2, ((false) is true) failed
[09:30:51]: ▸ AtomicBoyUITests.m:48
[09:30:51]: ▸ Executed 1 test, with 1 failure (0 unexpected) in 4.936 (4.938) seconds
[09:30:51]: ▸
[09:30:52]: ▸ 2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 8.505 elapsed -- Testing started completed.
[09:30:52]: ▸ 2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:30:52]: ▸ 2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 8.505 sec, +8.505 sec -- end
[09:30:52]: ▸ Failing tests:
[09:30:52]: ▸ -[AtomicBoyUITests testExample2]
[09:30:52]: ▸ ** TEST EXECUTE FAILED **
2018-08-10 09:30:43.754 xcodebuild[15196:2870883] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-43--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-44B38688-85F5-4F8A-A0FD-BE2D35C4557C/AtomicBoyUITests-30BD73CB-D56E-4D50-BEDD-0382E9E5C06F/Session-AtomicBoyUITests-2018-08-10_093043-OCA5LO.log
2018-08-10 09:30:43.754 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: (4A88164B-1D2F-44F9-B31C-FDD2BAB16E13) Beginning test session AtomicBoyUITests-4A88164B-1D2F-44F9-B31C-FDD2BAB16E13 at 2018-08-10 09:30:43.754 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7fa438c461a0> {
SimDevice: iPhone 5s-1 (59E8A537-CBAD-470D-9480-6780EFE16934, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyUITests.xctest started
AtomicBoyUITests
✗ testExample2, ((false) is true) failed
AtomicBoyUITests
testExample2, ((false) is true) failed
AtomicBoyUITests.m:48
Executed 1 test, with 1 failure (0 unexpected) in 4.936 (4.938) seconds
2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 8.505 elapsed -- Testing started completed.
2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2018-08-10 09:30:52.241 xcodebuild[15196:2870703] [MT] IDETestOperationsObserverDebug: 8.505 sec, +8.505 sec -- end
Failing tests:
-[AtomicBoyUITests testExample2]
** TEST EXECUTE FAILED **
[09:30:52]: Exit status: 65
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 1 |
+--------------------+---+
[09:30:52]: Re-running scan on only failed tests
[09:30:52]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:52]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:55]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:55]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-1 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample2"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyUITests |
| output_types | html,junit |
| output_files | report-2.html,report-2.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:57]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=59E8A537-CBAD-470D-9480-6780EFE16934' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample2 test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15138-lgsc51'
[09:30:57]: ▸ Loading...
[09:30:58]: ▸ 2018-08-10 09:30:58.708 xcodebuild[15277:2872059] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:58]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-58--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-D5F70378-9DFB-44A4-B4D6-17B5211F9765/AtomicBoyUITests-562F6E41-5D09-4BD6-8831-2E42877F62DF/Session-AtomicBoyUITests-2018-08-10_093058-E7Y6TH.log
[09:30:58]: ▸ 2018-08-10 09:30:58.709 xcodebuild[15277:2872039] [MT] IDETestOperationsObserverDebug: (A9F055D1-50B6-493C-A1F2-59534DEFB738) Beginning test session AtomicBoyUITests-A9F055D1-50B6-493C-A1F2-59534DEFB738 at 2018-08-10 09:30:58.709 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7fa8a3f98ac0> {
[09:30:58]: ▸ SimDevice: iPhone 5s-1 (59E8A537-CBAD-470D-9480-6780EFE16934, iOS 11.1, Booted)
[09:30:58]: ▸ } (11.1 (15B87))
[09:31:01]: ▸ Selected tests
[09:31:01]: ▸ Test Suite AtomicBoyUITests.xctest started
[09:31:01]: ▸ AtomicBoyUITests
[09:31:06]: ▸ ✓ testExample2 (4.801 seconds)
[09:31:06]: ▸ Executed 1 test, with 0 failures (0 unexpected) in 4.801 (4.803) seconds
[09:31:06]: ▸
[09:31:06]: ▸ 2018-08-10 09:31:06.983 xcodebuild[15277:2872039] [MT] IDETestOperationsObserverDebug: 8.295 elapsed -- Testing started completed.
[09:31:06]: ▸ 2018-08-10 09:31:06.983 xcodebuild[15277:2872039] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:31:06]: ▸ 2018-08-10 09:31:06.983 xcodebuild[15277:2872039] [MT] IDETestOperationsObserverDebug: 8.295 sec, +8.295 sec -- end
[09:31:06]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 0 |
+--------------------+---+
--------------------------------------------------------------------------------
[09:30:37]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:37]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:39]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:39]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-2 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyUITests |
| output_types | html,junit |
| output_files | report-2.html,report-2.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:42]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=612BA9D1-4929-493A-BE0E-FA034F022B4F' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-2.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15139-x2du60'
[09:30:42]: ▸ Loading...
[09:30:43]: ▸ 2018-08-10 09:30:43.766 xcodebuild[15193:2870719] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:43]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-43--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-94CE1DBE-5C8C-4ACF-A838-1521DC92886A/AtomicBoyUITests-FD561C9C-27A4-4BDA-8494-EE7F3910FCDF/Session-AtomicBoyUITests-2018-08-10_093043-GmV1t4.log
[09:30:43]: ▸ 2018-08-10 09:30:43.766 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: (DB943392-CDCF-4B3A-877F-3B3B64B48270) Beginning test session AtomicBoyUITests-DB943392-CDCF-4B3A-877F-3B3B64B48270 at 2018-08-10 09:30:43.766 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7faca3ec3f70> {
[09:30:43]: ▸ SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
[09:30:43]: ▸ } (11.1 (15B87))
[09:30:47]: ▸ Selected tests
[09:30:47]: ▸ Test Suite AtomicBoyUITests.xctest started
[09:30:47]: ▸ AtomicBoyUITests
[09:30:51]: ▸ ✗ testExample, ((false) is true) failed
[09:30:51]: ▸ AtomicBoyUITests
[09:30:51]: ▸ testExample, ((false) is true) failed
[09:30:51]: ▸ AtomicBoyUITests.m:40
[09:30:51]: ▸ Executed 1 test, with 1 failure (0 unexpected) in 4.940 (4.942) seconds
[09:30:51]: ▸
[09:30:52]: ▸ 2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 8.542 elapsed -- Testing started completed.
[09:30:52]: ▸ 2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:30:52]: ▸ 2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 8.542 sec, +8.542 sec -- end
[09:30:52]: ▸ Failing tests:
[09:30:52]: ▸ -[AtomicBoyUITests testExample]
[09:30:52]: ▸ ** TEST EXECUTE FAILED **
2018-08-10 09:30:43.766 xcodebuild[15193:2870719] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-43--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-94CE1DBE-5C8C-4ACF-A838-1521DC92886A/AtomicBoyUITests-FD561C9C-27A4-4BDA-8494-EE7F3910FCDF/Session-AtomicBoyUITests-2018-08-10_093043-GmV1t4.log
2018-08-10 09:30:43.766 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: (DB943392-CDCF-4B3A-877F-3B3B64B48270) Beginning test session AtomicBoyUITests-DB943392-CDCF-4B3A-877F-3B3B64B48270 at 2018-08-10 09:30:43.766 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7faca3ec3f70> {
SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyUITests.xctest started
AtomicBoyUITests
✗ testExample, ((false) is true) failed
AtomicBoyUITests
testExample, ((false) is true) failed
AtomicBoyUITests.m:40
Executed 1 test, with 1 failure (0 unexpected) in 4.940 (4.942) seconds
2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 8.542 elapsed -- Testing started completed.
2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2018-08-10 09:30:52.240 xcodebuild[15193:2870702] [MT] IDETestOperationsObserverDebug: 8.542 sec, +8.542 sec -- end
Failing tests:
-[AtomicBoyUITests testExample]
** TEST EXECUTE FAILED **
[09:30:52]: Exit status: 65
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 1 |
+--------------------+---+
[09:30:52]: Re-running scan on only failed tests
[09:30:52]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:52]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:30:55]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:30:55]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-2 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyUITests |
| output_types | html,junit |
| output_files | report-3.html,report-3.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:30:57]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=612BA9D1-4929-493A-BE0E-FA034F022B4F' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-3.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-3.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15139-wji90u'
[09:30:57]: ▸ Loading...
[09:30:58]: ▸ 2018-08-10 09:30:58.699 xcodebuild[15276:2872047] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:30:58]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-58--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-16EFE1CB-CEC6-4734-B018-2F81D2CC666A/AtomicBoyUITests-AF710FA6-9D64-4DB8-A3C0-A28044DEAA60/Session-AtomicBoyUITests-2018-08-10_093058-aBX7Ma.log
[09:30:58]: ▸ 2018-08-10 09:30:58.699 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: (C5A9DED9-C6D5-4E3B-810A-5F4EBC5922A4) Beginning test session AtomicBoyUITests-C5A9DED9-C6D5-4E3B-810A-5F4EBC5922A4 at 2018-08-10 09:30:58.699 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7f8f149e62b0> {
[09:30:58]: ▸ SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
[09:30:58]: ▸ } (11.1 (15B87))
[09:31:01]: ▸ Selected tests
[09:31:01]: ▸ Test Suite AtomicBoyUITests.xctest started
[09:31:01]: ▸ AtomicBoyUITests
[09:31:06]: ▸ ✗ testExample, ((false) is true) failed
[09:31:06]: ▸ AtomicBoyUITests
[09:31:06]: ▸ testExample, ((false) is true) failed
[09:31:06]: ▸ AtomicBoyUITests.m:40
[09:31:06]: ▸ Executed 1 test, with 1 failure (0 unexpected) in 4.603 (4.605) seconds
[09:31:06]: ▸
[09:31:06]: ▸ 2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 8.106 elapsed -- Testing started completed.
[09:31:06]: ▸ 2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:31:06]: ▸ 2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 8.106 sec, +8.106 sec -- end
[09:31:06]: ▸ Failing tests:
[09:31:06]: ▸ -[AtomicBoyUITests testExample]
[09:31:06]: ▸ ** TEST EXECUTE FAILED **
2018-08-10 09:30:58.699 xcodebuild[15276:2872047] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-30-58--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-16EFE1CB-CEC6-4734-B018-2F81D2CC666A/AtomicBoyUITests-AF710FA6-9D64-4DB8-A3C0-A28044DEAA60/Session-AtomicBoyUITests-2018-08-10_093058-aBX7Ma.log
2018-08-10 09:30:58.699 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: (C5A9DED9-C6D5-4E3B-810A-5F4EBC5922A4) Beginning test session AtomicBoyUITests-C5A9DED9-C6D5-4E3B-810A-5F4EBC5922A4 at 2018-08-10 09:30:58.699 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7f8f149e62b0> {
SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
} (11.1 (15B87))
Selected tests
Test Suite AtomicBoyUITests.xctest started
AtomicBoyUITests
✗ testExample, ((false) is true) failed
AtomicBoyUITests
testExample, ((false) is true) failed
AtomicBoyUITests.m:40
Executed 1 test, with 1 failure (0 unexpected) in 4.603 (4.605) seconds
2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 8.106 elapsed -- Testing started completed.
2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2018-08-10 09:31:06.786 xcodebuild[15276:2872040] [MT] IDETestOperationsObserverDebug: 8.106 sec, +8.106 sec -- end
Failing tests:
-[AtomicBoyUITests testExample]
** TEST EXECUTE FAILED **
[09:31:06]: Exit status: 65
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 1 |
+--------------------+---+
[09:31:06]: Re-running scan on only failed tests
[09:31:06]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:31:06]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
[09:31:08]: Successfully loaded '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+
[09:31:08]: $ xcodebuild -showBuildSettings -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj
+----------------------------+--------------------------------------+
| Summary for scan 2.100.1 |
+----------------------------+--------------------------------------+
| project | /Users/lyndsey.ferguson/repo/fastla |
| | ne-plugin-test_center/AtomicBoy/Ato |
| | micBoy.xcodeproj |
| scheme | AtomicBoy |
| test_without_building | true |
| derived_data_path | /Users/lyndsey.ferguson/Library/Dev |
| | eloper/Xcode/DerivedData/AtomicBoy- |
| | flqqvvvzbouqymbyffgdbtjoiufr |
| device | iPhone 5s-2 (11.1) |
| skip_build | false |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| clean | false |
| only_testing | ["AtomicBoyUITests/AtomicBoyUITests |
| | /testExample"] |
| output_directory | ./fastlane/test_output/results-Atom |
| | icBoyUITests |
| output_types | html,junit |
| output_files | report-4.html,report-4.junit |
| fail_build | true |
| xcode_path | /Applications/Xcode-10.app |
+----------------------------+--------------------------------------+
[09:31:10]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=612BA9D1-4929-493A-BE0E-FA034F022B4F' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building | tee '/Users/lyndsey.ferguson/Library/Logs/scan/AtomicBoy-AtomicBoy.log' | xcpretty --report html --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-4.html' --report junit --output '/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report-4.junit' --report junit --output '/var/folders/b_/rglnmdbd2hn6pskh12w78y9ddb1yfp/T/junit_report20180810-15139-lbcnkr'
[09:31:10]: ▸ Loading...
[09:31:11]: ▸ 2018-08-10 09:31:11.829 xcodebuild[15333:2873208] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[09:31:11]: ▸ /Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr/Logs/Test/Test-AtomicBoy-2018.08.10_09-31-11--0400.xcresult/1_Test/Diagnostics/AtomicBoyUITests-0F8BB9EF-5218-4273-B9C3-301C9061BEB5/AtomicBoyUITests-7EA7A170-2597-4A4A-B9EA-6DF07B330915/Session-AtomicBoyUITests-2018-08-10_093111-kfmMc1.log
[09:31:11]: ▸ 2018-08-10 09:31:11.829 xcodebuild[15333:2873193] [MT] IDETestOperationsObserverDebug: (B67AD1F2-E226-4676-BBC0-9400252A2887) Beginning test session AtomicBoyUITests-B67AD1F2-E226-4676-BBC0-9400252A2887 at 2018-08-10 09:31:11.830 with Xcode 10L221o on target <DVTiPhoneSimulator: 0x7f998ce45ba0> {
[09:31:11]: ▸ SimDevice: iPhone 5s-2 (612BA9D1-4929-493A-BE0E-FA034F022B4F, iOS 11.1, Booted)
[09:31:11]: ▸ } (11.1 (15B87))
[09:31:14]: ▸ Selected tests
[09:31:14]: ▸ Test Suite AtomicBoyUITests.xctest started
[09:31:14]: ▸ AtomicBoyUITests
[09:31:18]: ▸ ✓ testExample (4.277 seconds)
[09:31:18]: ▸ Executed 1 test, with 0 failures (0 unexpected) in 4.277 (4.279) seconds
[09:31:18]: ▸
[09:31:19]: ▸ 2018-08-10 09:31:19.071 xcodebuild[15333:2873193] [MT] IDETestOperationsObserverDebug: 7.263 elapsed -- Testing started completed.
[09:31:19]: ▸ 2018-08-10 09:31:19.071 xcodebuild[15333:2873193] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[09:31:19]: ▸ 2018-08-10 09:31:19.071 xcodebuild[15333:2873193] [MT] IDETestOperationsObserverDebug: 7.263 sec, +7.263 sec -- end
[09:31:19]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 0 |
+--------------------+---+
================================================================================
+-------------------+-------------------------------------------------------------------------------------------------------------------------+
| multi_scan results |
+-------------------+-------------------------------------------------------------------------------------------------------------------------+
| result | true |
| total_tests | 4 |
| passing_testcount | 4 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 0 |
| report_files | ["/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.junit", |
| | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.junit", |
| | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.junit", |
| | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/report.html", |
| | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyUITests/report.html", |
| | "/Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/fastlane/test_output/results-AtomicBoyTests/report.html"] |
+-------------------+-------------------------------------------------------------------------------------------------------------------------+
+------+------------+-------------+
| fastlane summary |
+------+------------+-------------+
| Step | Action | Time (in s) |
+------+------------+-------------+
| 1 | multi_scan | 64 |
+------+------------+-------------+
[09:31:19]: fastlane.tools finished successfully 🎉
#######################################################################
# fastlane 2.101.1 is available. You are on 2.100.1.
# You should use the latest version.
# Please update using `bundle update fastlane`.
#######################################################################
2.101.1 Hotfix for supply and google_play_track_version_codes
* [supply] return empty array when aab not set (#12998) via jrnk
* [spaceship] fetch list of available bundle ids (#13001) via heartofagoof
* [action] google_play_track_version_codes needs :timeout (#13014) via joshdholtz
2.101.0 Improvements
* [supply] added new timeout option (#12981) via Josh Holtz
* [snapshot] fix window is not found (#12964) via Saúl Moreno Abril
* [spaceship] support subscription_pricing_target when creating IAP. (#11875) via Ben Yohay
* [fastlane] filter out non-xcodeproj files from the list returned by `project_paths`. (#12985) via Michał Kałużny
* [action]Add ability to specify Carthage execuatable (#12969) via Brentley Jones
* [spaceship] parametrize view_by in analytics API (#12967) via Pawel Rusin
* [action] added author_email to last_git_commit action (#12926) via wag-miles
* [supply] fix issue that uploading aab with mapping file (#12856) via Takao Chiba
* [supply] make Google API Client gemspec version looser (#12918) via Takeru Chuganji
Please update using `bundle update fastlane`
So, the log files that the two different processes are writing to are different. So, there should not be contention around that.
lyndsey-ferguson
on 10 Aug 2018
Two different simulators
lyndsey-ferguson
on 10 Aug 2018
tee is writing to the same file. While I don't think it should cause the bootstrap problem, it cannot be good.
lyndsey-ferguson
on 10 Aug 2018
I have read that people have to quit or reset the simulators when this problem occurs. Maybe I can reset the particular simulator between each run?
lyndsey-ferguson
on 10 Aug 2018
Running the two xcodebuild test commands in the background does not give me the same error:
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=59E8A537-CBAD-470D-9480-6780EFE16934' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample2 test-without-building &; set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme AtomicBoy -project /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/AtomicBoy/AtomicBoy.xcodeproj -destination 'platform=iOS Simulator,id=612BA9D1-4929-493A-BE0E-FA034F022B4F' -derivedDataPath '/Users/lyndsey.ferguson/Library/Developer/Xcode/DerivedData/AtomicBoy-flqqvvvzbouqymbyffgdbtjoiufr' -only-testing:AtomicBoyUITests/AtomicBoyUITests/testExample test-without-building &
Ug, upon reviewing the above log, I see that there was not an issue bootstrapping a simulator.
lyndsey-ferguson
on 13 Aug 2018
Odd, I have already written this here, but to repeat:
- Resetting the simulators for the run seems to allow the parallel use of
xcodebuild testto function without anybootstrapissues. - Next is to add a
:multi_scanoption to parallelize when running with batches or multiple targets. - Next is to refactor the
forkcode out to be optional depending on the parallelize option. - Next is to programmatically create duplicate simulators for each run and delete them when complete.
lyndsey-ferguson
on 13 Aug 2018
Here is the current status:
- Resetting the simulator does not seem to prevent the bootstrap issues. I just saw that repeat.
- I have added in functionality to conditionally parallelize the execution.
- I have solved the hanging issue. I have added a working indicator.
Remaining work:
- It is hanging when it tries to get the
tests_passedresult. That is because it is passing back a IO object. I need to serialize the result and pass it back to the parent process. - I still need to fix the bootstrap issues.
- I still need to programmatically create duplicate simulators for each run and delete them when done.
- I need to change the parallelization to scan testables in parallel too.
lyndsey-ferguson
on 17 Aug 2018
See the project for progress
lyndsey-ferguson
on 24 Aug 2018
@lyndsey-ferguson -- With Xcode 10, apple has introduced parallel testing without the need to create additional schemas and duplicating simulators. There is setting in the Target Scheme that you can enable testing on parallel simulators and also xcodebuild has new flags added to provide the number of simulators that can be launched etc. GIven all this new stuff, do you still need to work on implementing this issue?
Sandhu412-zz
on 8 Oct 2018
Oh, I didn't know this. I made something beautiful, but if Xcode has the ability to automatically split the tests up, that sounds great. Can you save me time and write out what needs to be done to take advantage of this?
lyndsey-ferguson
on 8 Oct 2018
I found this article. It seems that Xcode is not so smart and if you have a test class that has many tests, the tests will all run on one simulator. My approach is to split the tests up and run chunks on each simulator.
lyndsey-ferguson
on 8 Oct 2018
ah, I see. You are correct that Xcode will launch multiple simulators and distribute the tests by class. So if a class has 10 tests, all 10 tests will go to one simulator. But I did not realize you were trying to split the tests in a class across simulators as well. But unlike Xcode 9, we dont have to create n additional schemes etc, Xcode does everything for us is what I meant.
By the way, this is not related to multi_scan not being able to find any tests right?
Sandhu412-zz
on 8 Oct 2018
I'm not sure yet what the cause is for multi_scan not finding the tests. When you get a chance, please let me know the results of my questions. I need more information to proceed on that issue.
lyndsey-ferguson
on 8 Oct 2018
Oh, and when I'm done adding the feature, you will not need to create additional schemes to parallelize across simulators. You just have to provide the :batch_count, :parallelize: true, and _maybe_ :max_simulators: X.
lyndsey-ferguson
on 8 Oct 2018
Oh, and when I'm done adding the feature, you will not need to create additional schemes to parallelize across simulators. You just have to provide the
:batch_count,:parallelize: true, and _maybe_:max_simulators: X.
-- This will be awesome!
-- I will get back to you with answers to your questions.
Sandhu412-zz
on 8 Oct 2018
Does this mean that multi_scan does not support UI Tests when the scheme has the Xcode 10 option to run tests in parallel enabled?
We've got that enabled and the tests run in 3 cloned simulators, but multi_scan only seems to run the tests once despite having the try_count set to 3?
Jon889
on 31 Oct 2018
@Jon889 basically, yes, multi_scan has not been set up to use that option. multi_scan uses the results of the junit report to determine which tests to re-run. I believe that either fastlane or xcpretty had problems generating that report when the tests have been setup to run in parallel via the Scheme.
It may be that a newer version of fastlane or xcpretty has fixed that problem. My plan for now is to, when given the option to parallelize, disable the xcodebuild version of running tests in parallel to ensure that the best results are being generated.
lyndsey-ferguson
on 1 Nov 2018
I would be happy to help testing this feature if you like :)
quintuschen
on 31 Jan 2019
@quintuschen I thought I had wrapped up a lot of the main functionality, but it totally fails on Jenkins with more than 3 parallel streams. My branch is a mess with debugging and experiments, I can clean it up in upcoming weeks and then send you a branch to use for testing.
lyndsey-ferguson
on 31 Jan 2019
Thanks a lot. Kudos! :) Excited with this parallel running feature.
quintuschen
on 31 Jan 2019
If you would like to test the feature, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "test-parallel-multiscan"
If you run into issues as I suspect you will, please attach the console output as a text file to this issue (makes it easier for me to review).
This works fine for me on my local computer. However, when running this on CI (Jenkins), often the parallel tests will fail with log statements stating that a connection to testmanagerd failed. I believe that that issue is due to how I am parallelizing: I am forking the entire fastlane process. By doing that, I am probably coming up against a limit of the number of file descriptors that can be open.
In my tests, CI can run fine with 3 forks and the tests. If I go higher, the whole thing collapses. My next plan is to not fork the fastlane process, and just spawn off direct calls to xcodebuild directly and wait for those to finish.
lyndsey-ferguson
on 4 Feb 2019
Thanks @lyndsey-ferguson I'll give it a try within this week. :)
quintuschen
on 4 Feb 2019
I've been giving this a whirl today, and so far I've come across 2 small issues.
1) I had to manually put gem "pry-byebug"in my Gemfile and run bundle install. When I didn't, fastlane would fail to load the plugin saying something about not being able to find "pry-byebug" and calling multi_scan would fail. Installing it through my main Gemfile fixed it.
2) It seems to have an odd problem if there are ( )s in the simulator name. When I tried to run it on "iPad Pro (12.9-inch) (2nd generation)", it tried to run the tests on iPhone 4s instead. I was able to work around this by renaming the sim in Xcode and changing my Fastfile to take out the ( )s.
I've only done very limited testing so far, but after I worked out the pry-byebug thing it worked without a hitch when running our tests on iPhone X in 3 parallelized batches. We have a large number of time consuming UI tests we run regularly that normally takes around 2-2.5 hours. My parallelized iPhone X test run earlier completed in just over 1 hour. So this'll be a huge timesaver for us if all goes well. 👍🏻 We have a Jenkins setup with 3 build nodes, so I'll be keeping an eye on how it does on them.
jcharr1
on 7 Feb 2019
Thank you for testing @jcharr1. I have a check that removes pry-byebug before releases, and for a good reason as you found out. Thanks for figuring that out.
It is interesting that it is unable to find the simulators, I have code that clones the simulator(s) that you have chosen to ensure that each "batch" of tests is run on a separate set of simulators. I'll have look into that to see what is going on. Here's the project ticket.
Regarding your findings, I'd be interested to hear what happens when you try 4,5, or 6 batches. It _may_ work if you do not have a ton of tests or if you do not do what my big test case does, which is to open sockets between the test-runner and the app to transfer mock files.
lyndsey-ferguson
on 7 Feb 2019
I've removed the require 'pry-byebug'
lyndsey-ferguson
on 7 Feb 2019
I tried 6 batches, but it failed under Jenkins and didn't do too well running it directly either. I backed it down to 5, and it finished in Jenkins. But it only saved about 10 minutes over 3 batches and caused some very noticeable lag on the nodes. So I went back to 3.
Maybe it's not implemented yet, but I noticed that total_retry_count seems to always return 0 even though some tests in the run definitely failed and were reran.
I'm wondering how we might be able to gather code coverage in the parallel version. Before, I used testrun_completed_block to gather it after the first test run had finished so that it would have data for all the tests. But this seems to be ignored when running in parallel. I can gather it after multi_scan finishes, but that seems to only get coverage for part of the tests.
jcharr1
on 7 Feb 2019
total_retry_count may have been missed, yes.
I think I can handle the code-coverage case too.
I am not surprised by the failures that you encountered. Did it complain about testmanagerd connections?
lyndsey-ferguson
on 7 Feb 2019
Yep.
Testing failed:
InnovativeUITests-Runner.app encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying error: Lost connection to testmanagerd))
** TEST EXECUTE FAILED **
jcharr1
on 8 Feb 2019
I am also trying to test the build that Lyndsey provided. When I specify batch_count:3, I noticed that there are actually 4 simulators running, e.g. from "iPhone X-batchclone-1" to "iPhone X-batchclone-4", is this expected? or should we expect three simulators running?
quintuschen
on 18 Feb 2019
I've seen that happen too occasionally. I've noticed it mostly if I reduce the batch_count between runs. For example, I tried a 6 batch run. Then right after that one finished I tried it with a batch_count of 5 but it spawned 6 clones. I think one of them automatically closed after a little while. For me, it doesn't do this all the time.
I know this is a WIP, but I'm curious if failing tests are actually being rerun yet. I seem to end up with more test failures on average than with the old version even with my try_count set to 4 as usual. The test failures are usually sporadic crashes or other hiccups that could most likely have been resolved by running the tests again.
I've also noticed a minor issue with the test results in Blue Ocean. You can see how it says there were 2 Existing Failure (which is accurate) but the list of failed tests included both new and existing failures.
jcharr1
on 18 Feb 2019
I am able to run it locally (I have the same question as @jcharr1 that it seemed not retrying failed cases) but not able to run it on Jenkins agents. I've set batch_count to 3 and run the tests on our Jenkins build agents, all of the spawned simulators got stuck when running them.
One thing I noticed is that on our Jenkins, we have other iOS build jobs running too, not UI tests in the same folder but other tasks that could use simulators. We don't use the same simulator though but would this be related?
quintuschen
on 18 Feb 2019
I've turned off retry until I can get the simulators to run without the testmanagerd connection failures. So, yes, there will not be any retrying of failed tests and you'll have more failing tests.
Also, regarding other simulators, I clone the requested simulator(s) so as to avoid them stomping on each other (which will happen with the same simulator). The cloned simulators are deleted just before the process exits.
lyndsey-ferguson
on 18 Feb 2019
My latest attempt was to run the 6 xcodebuild test runs directly from multi_scan by spawning them. That resulted in the same testmanagerd issue on Jenkins.
I will confirm that spawning the xcodebuild test from my plugin locally does not result in a problem. Then I'll try directly on the mac mini that Jenkins used. Then I'll try from a Script Build Step via Jenkins, then I'll try banging my head on the wall.
Here's the project if you're interested.
lyndsey-ferguson
on 18 Feb 2019
Good news, I can make it fail locally. 😢 Now to figure out if there is anything I can do.
lyndsey-ferguson
on 18 Feb 2019
5 simulators are okay, 6 are not.
lyndsey-ferguson
on 18 Feb 2019
Oh, and I'll look into the number of sims per batch. I've noticed that too, but haven't worried about it for the moment.
lyndsey-ferguson
on 18 Feb 2019
@lyndsey-ferguson that's fine about retries being disabled for now. I just wanted to make sure I hand't borked something up on my end.
jcharr1
on 18 Feb 2019
Maybe the fact that I am re-using a simulator with that extra batch?
lyndsey-ferguson
on 18 Feb 2019
Hm. Running the command via bash on my local, ensuring that I have only 6 sims running, does not result in the testmanagerd disconnection issues.
lyndsey-ferguson
on 18 Feb 2019
@jcharr1 do you have direct access to the CI agent that Jenkins used to run multi_scan in parallel? If so, does the same testmanagerd disconnection issue happen there? Does it happen on your local computer?
lyndsey-ferguson
on 22 Feb 2019
Also, please try fetching a fresh version of this branch. I've done some work on it.
lyndsey-ferguson
on 22 Feb 2019
Yes, all my Jenkins nodes are here in the house. I never saw the testmanagerd disconnect when running it directly from the terminal. And I only saw it when trying to run 6 batches in Jenkins. I haven't seen it when running 5 or less. And here's some specs on my nodes if it helps:
- mid 2015 15" Macbok Pro w/2.5Ghz i7, 16GB RAM. Mojave 10.14.3. This is also my main computer.
- late 2012 Mac Mini w/2.6Ghz i7, 16GB RAM, and an OWC SSD. Also Mojave 10.14.3. This is the Jenkins server.
- early 2011 17" Macbook Pro w/2.2Ghz i7, 8GB RAM, a OWC SSD drive for macOS and a hybrid SSD/HD for data. High Sierra 10.13.6.
They're not bad but they're certainly not beasts either. So they do struggle with running lots of sims at the same time.
jcharr1
on 22 Feb 2019
So, on all the machines, there is no testmanagerd problem when running it from the Terminal? I get it sometimes. I logged into a Mac mini that we use for Jenkins via Screen Sharing and it happens every time.
lyndsey-ferguson
on 22 Feb 2019
And this is with the latest branch?
lyndsey-ferguson
on 22 Feb 2019
I've only tried running it directly from Terminal on my 2015 MBP. Yes, with the latest update to the branch.
jcharr1
on 22 Feb 2019
Can you run it directly from the Terminal for #3? That would be something similar to what I would do (even better if you could use Screen Sharing to access that Terminal).
I'm not sure if the Jenkins JVM slave.jar is the problem some how (seems far fetched).
lyndsey-ferguson
on 22 Feb 2019
Hmm. It won't run at all on my 2011 MBP today either with Jenkins or directly through Terminal. It keeps giving errors about the test runner not being able to be installed or launched. It would at least run before. It'll run if I set parallelize to false. It still works on my other 2 Macs.
[17:00:20]: $ xcodebuild -showBuildSettings -workspace ./Innovative.xcworkspace -scheme Innovative
[17:00:23]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '0' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=C7DB9E36-3444-4837-9CD3-29E5BF14710A' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-l9zqix' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/AudioPlayerControlsUITests/testShowHideAudioPlayerControls -only-testing:InnovativeUITests/AudioPlayerUITests/testAudioPlayerControls -only-testing:InnovativeUITests/AudioPlayerUITests/testAudioPlayerInNewestLessons -only-testing:InnovativeUITests/AudioPlayerUITests/testAutoPlayWithNextPrevButtonsTap -only-testing:InnovativeUITests/AudioPlayerUITests/testAutoPlay -only-testing:InnovativeUITests/AudioPlayerUITests/testRepeatWithNextPrevTrackTap -only-testing:InnovativeUITests/AudioPlayerUITests/testRepeat -only-testing:InnovativeUITests/CompletionUITests/testCompletionInAudio -only-testing:InnovativeUITests/CompletionUITests/testCompletionInVideo -only-testing:InnovativeUITests/DownloadManagerUITests/testDownloadLesson test-without-building | tee '/Users/jason/Library/Logs/scan-0/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-0/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-1ey0a6w'
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '1' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=68710351-3C68-4312-865D-AABF3078167A' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1dblj9l' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/DownloadManagerUITests/testDownloadLessonsWithCancel -only-testing:InnovativeUITests/DownloadManagerUITests/testDownloadsFromCategory -only-testing:InnovativeUITests/DownloadManagerUITests/testSelectUnselectAllDownloadsFromCategory -only-testing:InnovativeUITests/ExpansionTest/testExpansion -only-testing:InnovativeUITests/HomeScreenSwitchLanguageTest/testHomeScreenSwitchLanguage -only-testing:InnovativeUITests/HomeScreenUITests/testHomeScreen -only-testing:InnovativeUITests/InitialSwitchLanguageTest/testInitialSwitchLanguage -only-testing:InnovativeUITests/LessonScreenUITests/testLessonsScreen -only-testing:InnovativeUITests/LessonsBrowseSearchUITests/testBrowseLessonsAndSearch -only-testing:InnovativeUITests/LibraryUITests/testLibraryClear test-without-building | tee '/Users/jason/Library/Logs/scan-0-1/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-1/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-c72smn'
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '2' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=61B1AF22-A60F-4CC9-8C6D-F22B6AB4C643' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-156jsor' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/LibraryUITests/testLibraryFillWthCompletionTest -only-testing:InnovativeUITests/LineByLineAudioTest/testLineByLineAudio -only-testing:InnovativeUITests/LockedLessonsUITests/testLockedLessons -only-testing:InnovativeUITests/LoginLogoutUITests/testLogin -only-testing:InnovativeUITests/LoginLogoutUITests/testTransitionToRegistrationScreen -only-testing:InnovativeUITests/LoginSwitchLanguageTest/testLoginSwitchLanguage -only-testing:InnovativeUITests/MyTeacherTest/testMyTeacher -only-testing:InnovativeUITests/PDFViewerTest/testPDFViewer -only-testing:InnovativeUITests/PrivacyAndTermOfUseUITests/testPrivacyAndTermOfUse -only-testing:InnovativeUITests/RegistrationUITests/testRegistration test-without-building | tee '/Users/jason/Library/Logs/scan-0-1-2/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-2/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-c370r5'
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '3' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=085692ED-5962-4099-8BBF-877BF63EFD4E' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-7guh9' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/SeriesScreenUITests/testLessonSearch -only-testing:InnovativeUITests/SeriesScreenUITests/testSeriesSearch -only-testing:InnovativeUITests/SeriesScreenUITests/testTransitionToDownloadScreen -only-testing:InnovativeUITests/SettingsScreenSwitchLanguageTest/testSettingsScreenSwitchLanguage -only-testing:InnovativeUITests/SettingsTest/testSettings -only-testing:InnovativeUITests/SignUpInitScreenUITests/testAutoSlides -only-testing:InnovativeUITests/SignUpInitScreenUITests/testLayout -only-testing:InnovativeUITests/SignUpInitScreenUITests/testManualSlides -only-testing:InnovativeUITests/SignUpInitScreenUITests/testTransitionToLoginScreen -only-testing:InnovativeUITests/SignUpInitScreenUITests/testTransitionToSignUpScreen test-without-building | tee '/Users/jason/Library/Logs/scan-0-1-2-3/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-3/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-1h3u4i2'
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '4' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=CFA95E5F-00F2-41E9-8E38-795828205BD6' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1k3hmi2' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/SignUpInitScreenUITests/testTransitionToSwitchLanguageScreen -only-testing:InnovativeUITests/TabsTest/testTabs -only-testing:InnovativeUITests/TextSizeSettingsUITest/testTextSize -only-testing:InnovativeUITests/UpgradeScreenTest/testUpgradeScreen -only-testing:InnovativeUITests/VideoUITests/testVideos -only-testing:InnovativeUITests/VocabularyTest/testVocabulary -only-testing:InnovativeUITests/WordBankLabelUITests/testCopyWords -only-testing:InnovativeUITests/WordBankLabelUITests/testDeleteWords -only-testing:InnovativeUITests/WordBankLabelUITests/testMoveWords -only-testing:InnovativeUITests/WordBankLabelUITests/testRenameLabel test-without-building | tee '/Users/jason/Library/Logs/scan-0-1-2-3-4/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-4/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-1kfbgif'
[17:00:27]: --------------------------------------
[17:00:27]: --- Starting test run on batch '5' ---
[17:00:27]: --------------------------------------
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -destination 'platform=iOS Simulator,id=B829CA06-F082-4887-B611-04370E8062A4' -derivedDataPath '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1eihn5i' -disable-concurrent-testing -enableCodeCoverage YES -xctestrun '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-hwl8pk/Innovative_iphonesimulator12.1-x86_64.xctestrun' -only-testing:InnovativeUITests/WordBankLabelUITests/testUnlabelWords -only-testing:InnovativeUITests/WordBankUITests/testAddLabelFromMainScreen -only-testing:InnovativeUITests/WordBankUITests/testAddWordsToWordBankFromVocabulary -only-testing:InnovativeUITests/WordBankUITests/testDeleteLabels -only-testing:InnovativeUITests/WordBankUITests/testPlayWords -only-testing:InnovativeUITests/WordBankUITests/testWBMainScreenExpired -only-testing:InnovativeUITests/WordBankUITests/testWBMainScreen test-without-building | tee '/Users/jason/Library/Logs/scan-0-1-2-3-4-5/Innovative-Innovative.log' | xcpretty --report junit --output '/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone-tests/InnovativeLanguage.com/test_output/results-InnovativeUITests-batch-5/report.junit' --report junit --output '/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/junit_report20190222-15273-1i0ymv6'
2019-02-22 17:00:30.012 xcodebuild[17276:83675] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-l9zqix/Logs/Test/Test-Transient Testing-2019.02.22_17-00-29--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-B7A3F931-10FB-4A7E-B5AC-22391077B553/InnovativeUITests-3B75E519-F157-43B5-8289-3C1FE43C8120/Session-InnovativeUITests-2019-02-22_170030-K8BPep.log
2019-02-22 17:00:30.013 xcodebuild[17276:83583] [MT] IDETestOperationsObserverDebug: (F2EB4D1A-4B7E-4708-A320-0DA24C0FF055) Beginning test session InnovativeUITests-F2EB4D1A-4B7E-4708-A320-0DA24C0FF055 at 2019-02-22 17:00:30.013 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7fd9a9c04d80> {
SimDevice: iPhone X-batchclone-6 (C7DB9E36-3444-4837-9CD3-29E5BF14710A, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.083 xcodebuild[17280:83685] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1dblj9l/Logs/Test/Test-Transient Testing-2019.02.22_17-00-29--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-A6B70173-8FAF-47FB-8754-B563ADAE6B69/InnovativeUITests-3C5A81C6-1D58-4531-9FF3-8D50F490A374/Session-InnovativeUITests-2019-02-22_170030-tvmIKX.log
2019-02-22 17:00:30.084 xcodebuild[17280:83595] [MT] IDETestOperationsObserverDebug: (5C15C38F-79CE-4268-8238-696275A690A2) Beginning test session InnovativeUITests-5C15C38F-79CE-4268-8238-696275A690A2 at 2019-02-22 17:00:30.084 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7fac095c8840> {
SimDevice: iPhone X-batchclone-1 (68710351-3C68-4312-865D-AABF3078167A, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.336 xcodebuild[17284:83693] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-156jsor/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-17156A30-3DA4-4EBC-BCFD-0DD96CF142B5/InnovativeUITests-C6F4FB99-8A56-4DC0-B9A3-7BEC8EBA74DA/Session-InnovativeUITests-2019-02-22_170030-xO4NiX.log
2019-02-22 17:00:30.336 xcodebuild[17284:83620] [MT] IDETestOperationsObserverDebug: (8979B4AB-C698-48BD-BA7F-8E02101D3BE0) Beginning test session InnovativeUITests-8979B4AB-C698-48BD-BA7F-8E02101D3BE0 at 2019-02-22 17:00:30.336 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7faba14d0dc0> {
SimDevice: iPhone X-batchclone-2 (61B1AF22-A60F-4CC9-8C6D-F22B6AB4C643, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.362 xcodebuild[17292:83704] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1k3hmi2/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-910BE5A7-45F5-44CB-A3DB-F43A0AABD29E/InnovativeUITests-C79C295F-1BE8-468F-AA06-E4C019FA0F77/Session-InnovativeUITests-2019-02-22_170030-0kJtyk.log
2019-02-22 17:00:30.363 xcodebuild[17292:83646] [MT] IDETestOperationsObserverDebug: (C7A28D24-563F-4343-929F-805890AC6C65) Beginning test session InnovativeUITests-C7A28D24-563F-4343-929F-805890AC6C65 at 2019-02-22 17:00:30.363 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7fa6d81494c0> {
SimDevice: iPhone X-batchclone-4 (CFA95E5F-00F2-41E9-8E38-795828205BD6, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.379 xcodebuild[17288:83817] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-7guh9/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-5B2C050E-C33B-4417-8EEB-83D6337CADE3/InnovativeUITests-0B02B451-BBAF-4617-92D1-F1A6CFB6C717/Session-InnovativeUITests-2019-02-22_170030-VhVB4N.log
2019-02-22 17:00:30.380 xcodebuild[17288:83634] [MT] IDETestOperationsObserverDebug: (7A6BE41F-7E89-46C8-A8F7-C6E4C29F3A49) Beginning test session InnovativeUITests-7A6BE41F-7E89-46C8-A8F7-C6E4C29F3A49 at 2019-02-22 17:00:30.380 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7f93c75c98a0> {
SimDevice: iPhone X-batchclone-3 (085692ED-5962-4099-8BBF-877BF63EFD4E, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.391 xcodebuild[17296:83710] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1eihn5i/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult/1_Test/Diagnostics/InnovativeUITests-4BF5CA13-BBD3-4CA4-87BE-3555B8878BFD/InnovativeUITests-806F47C2-A7D2-4D99-9DD8-6F5CC35E299F/Session-InnovativeUITests-2019-02-22_170030-AYuOgT.log
2019-02-22 17:00:30.391 xcodebuild[17296:83660] [MT] IDETestOperationsObserverDebug: (C37D9661-FA00-4FC8-A4BA-57187BF27FF4) Beginning test session InnovativeUITests-C37D9661-FA00-4FC8-A4BA-57187BF27FF4 at 2019-02-22 17:00:30.391 with Xcode 10B61 on target <DVTiPhoneSimulator: 0x7fdab3169500> {
SimDevice: iPhone X-batchclone-5 (B829CA06-F082-4887-B611-04370E8062A4, iOS 12.1, Shutdown)
} (12.1 (16B91))
2019-02-22 17:00:30.561 xcodebuild[17292:83803] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.561 xcodebuild[17288:83666] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.564 xcodebuild[17276:83599] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.566 xcodebuild[17296:83823] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.567 xcodebuild[17284:83861] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.569 xcodebuild[17280:83804] CoreSimulatorService connection interrupted. Resubscribing to notifications.
2019-02-22 17:00:30.597 xcodebuild[17276:83583] [MT] IDETestOperationsObserverDebug: 0.884 elapsed -- Testing started completed.
2019-02-22 17:00:30.597 xcodebuild[17276:83583] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.597 xcodebuild[17276:83583] [MT] IDETestOperationsObserverDebug: 0.884 sec, +0.884 sec -- end
2019-02-22 17:00:30.597 xcodebuild[17280:83595] [MT] IDETestOperationsObserverDebug: 0.668 elapsed -- Testing started completed.
2019-02-22 17:00:30.598 xcodebuild[17280:83595] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.598 xcodebuild[17280:83595] [MT] IDETestOperationsObserverDebug: 0.668 sec, +0.668 sec -- end
2019-02-22 17:00:30.598 xcodebuild[17276:83583] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.598 xcodebuild[17276:83583] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-l9zqix/Logs/Test/Test-Transient Testing-2019.02.22_17-00-29--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7fd9a995ce00 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
2019-02-22 17:00:30.598 xcodebuild[17280:83595] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.598 xcodebuild[17280:83595] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1dblj9l/Logs/Test/Test-Transient Testing-2019.02.22_17-00-29--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7fac095f9c80 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
2019-02-22 17:00:30.599 xcodebuild[17284:83620] [MT] IDETestOperationsObserverDebug: 0.372 elapsed -- Testing started completed.
2019-02-22 17:00:30.599 xcodebuild[17284:83620] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.599 xcodebuild[17284:83620] [MT] IDETestOperationsObserverDebug: 0.372 sec, +0.372 sec -- end
2019-02-22 17:00:30.601 xcodebuild[17284:83620] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.601 xcodebuild[17284:83620] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-156jsor/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7faba3c15a30 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
2019-02-22 17:00:30.627 xcodebuild[17288:83634] [MT] IDETestOperationsObserverDebug: 0.332 elapsed -- Testing started completed.
2019-02-22 17:00:30.627 xcodebuild[17288:83634] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.627 xcodebuild[17288:83634] [MT] IDETestOperationsObserverDebug: 0.332 sec, +0.332 sec -- end
2019-02-22 17:00:30.627 xcodebuild[17296:83660] [MT] IDETestOperationsObserverDebug: 0.331 elapsed -- Testing started completed.
2019-02-22 17:00:30.627 xcodebuild[17296:83660] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.627 xcodebuild[17296:83660] [MT] IDETestOperationsObserverDebug: 0.331 sec, +0.331 sec -- end
2019-02-22 17:00:30.628 xcodebuild[17292:83646] [MT] IDETestOperationsObserverDebug: 0.349 elapsed -- Testing started completed.
2019-02-22 17:00:30.628 xcodebuild[17292:83646] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-02-22 17:00:30.628 xcodebuild[17292:83646] [MT] IDETestOperationsObserverDebug: 0.349 sec, +0.349 sec -- end
2019-02-22 17:00:30.628 xcodebuild[17296:83660] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.628 xcodebuild[17296:83660] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1eihn5i/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7fdab072c960 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
2019-02-22 17:00:30.628 xcodebuild[17292:83646] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.628 xcodebuild[17292:83646] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-1k3hmi2/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7fa6d83039b0 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
2019-02-22 17:00:30.630 xcodebuild[17288:83634] Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}
2019-02-22 17:00:30.630 xcodebuild[17288:83634] Error Domain=IDETestOperationsObserverErrorDomain Code=4 "Failed to install or launch the test runner" UserInfo={NSLocalizedRecoverySuggestion=If you believe this error represents a bug, please attach the result bundle at /var/folders/dk/z57xpvj52nx4wdtb5zzw_q7c0000gr/T/d20190222-15273-7guh9/Logs/Test/Test-Transient Testing-2019.02.22_17-00-30--0600.xcresult, NSLocalizedDescription=Failed to install or launch the test runner, NSUnderlyingError=0x7f93ca0098f0 {Error Domain=NSPOSIXErrorDomain Code=53 "Software caused connection abort" UserInfo={NSLocalizedDescription=Error returned in reply: Connection interrupted}}}
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
Testing failed:
InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Error returned in reply: Connection interrupted. Software caused connection abort))
** TEST EXECUTE FAILED **
[17:00:30]: Deleting iPhone X-batchclone-1
[17:00:32]: Deleting iPhone X-batchclone-2
[17:00:33]: Deleting iPhone X-batchclone-3
[17:00:34]: Deleting iPhone X-batchclone-4
[17:00:35]: Deleting iPhone X-batchclone-5
[17:00:35]: Deleting iPhone X-batchclone-6
+-------------------+---------------------------------------------------------------------+
| multi_scan results |
+-------------------+---------------------------------------------------------------------+
| result | false |
| total_tests | 0 |
| passing_testcount | 0 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 0 |
| report_files | ["/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone |
| | -tests/InnovativeLanguage.com/test_output/report-InnovativeUITests |
| | .junit", |
| | "/Volumes/DATA/jenkins_slave/workspace/for_iOS_Multibranch_iphone- |
| | tests/InnovativeLanguage.com/test_output/report.junit"] |
+-------------------+---------------------------------------------------------------------+
[17:00:36]: -------------------
[17:00:36]: --- Step: is_ci ---
[17:00:36]: -------------------
+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | hostname | 0 |
| 2 | Verifying fastlane version | 0 |
| 3 | default_platform | 0 |
| 4 | import_from_git | 1 |
| 5 | is_ci | 0 |
| 6 | hostname | 0 |
| 7 | cocoapods | 6 |
| 8 | is_ci | 0 |
| 9 | is_ci | 0 |
| 10 | match | 4 |
| 11 | multi_scan | 89 |
| 12 | is_ci | 0 |
+------+----------------------------+-------------+
[17:00:36]: fastlane.tools finished successfully 🎉
Using the Xcode feature of parallel testing, multi-scan reruns the complete set of tests again instead of just running the failed tests. @lyndsey-ferguson is there any way we can use the Xcode parallel testing feature with this?
vksmaini
on 25 Feb 2019
@jcharr1 thanks for the log, this may help, this is odd that this will fail.
@vksmaini I'm not using the parallel part of xcode because it will not balance tests across the simulators; it parallelizes based on the testsuites. That means if one testsuite has 4 tests, and another has 200, and the rest have twenty, you'll not have an efficient use of your machine's capabilities because it will stick all 200 on one sim (this is what I've read and have been told -- no testing on my end).
If you find that multi_scan works with parallel testing, stick with it.
lyndsey-ferguson
on 25 Feb 2019
@jcharr1 it could be that my attempt to fix overwriting derived data paths by giving it a random path for this run is the problem. Mine may be working because I am explicitly giving it something that it needs, but for your case what it needs is not there.
lyndsey-ferguson
on 25 Feb 2019
@jcharr1 I added a bit of code to copy the derived data directory contents to the temp folders I wrote about above. Can you try again?
lyndsey-ferguson
on 25 Feb 2019
Just a note for posterity. I ran into the testmanagerd issue locally. I ran the parallelized multi_scan, and my local's webpage responsiveness slowed down to a crawl. Perhaps the demand on resources is causing socket network issues with testmanagerd?
lyndsey-ferguson
on 25 Feb 2019
@lyndsey-ferguson it's working on my 2011 MBP again now. On all my nodes, I have a custom Xcode directory where it stores all the archives and derived data for that machine in (it makes it easy to wipe with a script periodically). I don't know if that has something to do with why it wouldn't run before. It's weird that my 2 Mojave nodes didn't have any problems. Though on my Mojave nodes, I'm running tests through the Xcode 12.2 beta to avoid this problem which causes some of our tests to stall out and fail: https://stackoverflow.com/questions/53432769/repeated-request-for-microphone-permission-on-ios-simulator
jcharr1
on 25 Feb 2019
If it is running again, it probably is because with your custom setup for storing derived data is not playing well with my hacking. At least it is running. What happens when you run it from the Terminal on that machine with 6 batches? I imagine the testmanagerd problem will happen again: I am seeing it on my local again. I think I am going past the number of file descriptors available.
lyndsey-ferguson
on 25 Feb 2019
With 6 it only spawned 4 saying InnovativeUITests-Runner.app encountered an error (Failed to install or launch the test runner. (Underlying error: Unable to boot device due to insufficient system resources. Please see Simulator Help for information on adjusting system settings to allow more simulated devices to boot at once.
jcharr1
on 25 Feb 2019
Whooo!
lyndsey-ferguson
on 25 Feb 2019
@jcharr1 does that failure also include something like:
maxUserProcs: 709, runningUserProcs: 569, enforcedProcBuffer: 144
? I wonder why I don't see that.
lyndsey-ferguson
on 25 Feb 2019
Yep. maxUserProcs: 709, runningUserProcs: 571, enforcedProcBuffer: 144
jcharr1
on 25 Feb 2019
Ok, I know why I don't see that on the jenkins mini, we've actually increased resource limits:
/Users/jenkins/: ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 524288
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 5000
virtual memory (kbytes, -v) unlimited
Ok, since it seems one what resources the machine actually has, I am going to revert back to forking the code and calling scan from each sub-process. The savings that I get from just spawning xcodebuild test does not have a measurable effect on the testmanagerd disconnection issue.
lyndsey-ferguson
on 27 Feb 2019
@jcharr1 @vksmaini @quintuschen @Sandhu412 @zacoid55 @kisialiu I have just successfully run the tests in parallel on Jenkins. Tests, spread out in 6 batches, that would take a combined time of 3 hours took 1 hour 9 mins.
Now, I did not yet review the results to make sure it ran all the tests, but it looks like it could have. At least I did not get testmanagerd connection failures.
Do you realize how monumental this is for me? What a break through!!!!! And a relief, I had been losing my drive on this puzzle, but I got it!
Can you test?
If so, modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-10-parellel-multi_scan"
Please attach the console output as a text file to this issue (makes it easier for me to review).
lyndsey-ferguson
on 15 Mar 2019
🚀
lyndsey-ferguson
on 15 Mar 2019
and the second run is still working!
lyndsey-ferguson
on 15 Mar 2019
Hmm I'm getting .rvm/gems/ruby-2.4.1/bundler/gems/fastlane-plugin-test_center-55605d3e8e18/lib/fastlane/plugin/test_center/actions/multi_scan.rb:22:inrun': [!] uninitialized constant TestCenter::Helper::RetryingScan::Runner (NameError)` for some reason from that branch!
kdthomas2121
on 18 Mar 2019
@kdthomas2121 try now
lyndsey-ferguson
on 18 Mar 2019
@lyndsey-ferguson no still getting the same error
kdthomas2121
on 18 Mar 2019
What SHA is listed in your Gemfile.lock file for the `fastlane-plugin-test_center' gem?
lyndsey-ferguson
on 18 Mar 2019
getting the same error as @kdthomas2121 uninitialized constant TestCenter::Helper::RetryingScan::Runner
content related to test_center in Gemlock looks like this
remote: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git
revision: 7e7f12e6a96e2a6cbd482e7437d18a425ee28fe5
branch: issue-10-parellel-multi_scan
specs:
fastlane-plugin-test_center (3.6.3)
json
plist
xcodeproj
xctest_list (>= 1.1.7)
DEPENDENCIES
fastlane (~> 2.116.0)
fastlane-plugin-test_center!
ishammoho
on 19 Mar 2019
Yea I have the same as above in my gemfile.lock
On Tue, 19 Mar 2019 at 01:35, ishammoho notifications@github.com wrote:
getting the same error as @kdthomas2121 https://github.com/kdthomas2121 uninitialized
constant TestCenter::Helper::RetryingScan::Runnercontent related to test_center in Gemlock looks like this
GIT
remote: https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git
revision: 7e7f12e6a96e2a6cbd482e7437d18a425ee28fe5
branch: issue-10-parellel-multi_scan
specs:
fastlane-plugin-test_center (3.6.3)
json
plist
xcodeproj
xctest_list (>= 1.1.7)DEPENDENCIES
fastlane (~> 2.116.0)
fastlane-plugin-test_center!—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/10#issuecomment-474163871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AerVy2e90tKwEl3X-CK6yEETl6hGXkdRks5vYD71gaJpZM4RoU01
.
--
KYLE THOMAS
SENIOR CONSULTANT
Infinity Works
Mobile: 07891098771
Email: kyle.[email protected] louise.featherstone@infinityworks.com
Infinity Works Consulting is a limited company registered in England and
Wales.
Company Registration No. 08189469
Registered offices: 3rd Floor, Apsley House, 78 Wellington St, Leeds, LS1
2EQ
https://maps.google.com/?q=78+Wellington+St,+Leeds,+LS1+2EQ&entry=gmail&source=g
--
Infinity Works Consulting Limited is a limited company registered in
England with registered number 08189469 and VAT registered number 179
6938 34. Our registered office is at Apsley House, 78 Wellington Street,
Leeds, LS1 2EQ.
This email is intended solely for the addressee and may
contain confidential information. If you have received this message in
error, please send it back to us, and immediately and permanently delete
it. Do not use, copy or disclose the information contained in this message
or in any attachment.
kdthomas2121
on 19 Mar 2019
Can you try now?
lyndsey-ferguson
on 19 Mar 2019
Now getting
such file -- pry-byebug
the documentation for more details: https://docs.fastlane.tools/actions
On Tue, 19 Mar 2019 at 14:17, Lyndsey Ferguson notifications@github.com
wrote:
Can you try now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/10#issuecomment-474390237,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AerVy6ik__P0vISzIfVlS1t5Hi0Io4Mpks5vYPFsgaJpZM4RoU01
.
--
KYLE THOMAS
SENIOR CONSULTANT
Infinity Works
Mobile: 07891098771
Email: [email protected] louise.featherstone@infinityworks.com
www.infinityworks.com
Infinity Works Consulting is a limited company registered in England and
Wales.
Company Registration No. 08189469
Registered offices: 3rd Floor, Apsley House, 78 Wellington St, Leeds, LS1
2EQ
https://maps.google.com/?q=78+Wellington+St,+Leeds,+LS1+2EQ&entry=gmail&source=g
--
Infinity Works Consulting Limited is a limited company registered in
England with registered number 08189469 and VAT registered number 179
6938 34. Our registered office is at Apsley House, 78 Wellington Street,
Leeds, LS1 2EQ.
This email is intended solely for the addressee and may
contain confidential information. If you have received this message in
error, please send it back to us, and immediately and permanently delete
it. Do not use, copy or disclose the information contained in this message
or in any attachment.
kdthomas2121
on 19 Mar 2019
whoops, left that in
lyndsey-ferguson
on 19 Mar 2019
Fixed!
lyndsey-ferguson
on 19 Mar 2019
Now I've got
such file -- colorize
On Tue, 19 Mar 2019 at 15:53, Lyndsey Ferguson notifications@github.com
wrote:
Fixed!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/10#issuecomment-474438514,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AerVy6de6LRp8sjz4bxmFgNL75QvKlUtks5vYQfvgaJpZM4RoU01
.
--
KYLE THOMAS
SENIOR CONSULTANT
Infinity Works
Mobile: 07891098771
Email: kyle.[email protected] louise.featherstone@infinityworks.com
Infinity Works Consulting is a limited company registered in England and
Wales.
Company Registration No. 08189469
Registered offices: 3rd Floor, Apsley House, 78 Wellington St, Leeds, LS1
2EQ
https://maps.google.com/?q=78+Wellington+St,+Leeds,+LS1+2EQ&entry=gmail&source=g
--
Infinity Works Consulting Limited is a limited company registered in
England with registered number 08189469 and VAT registered number 179
6938 34. Our registered office is at Apsley House, 78 Wellington Street,
Leeds, LS1 2EQ.
This email is intended solely for the addressee and may
contain confidential information. If you have received this message in
error, please send it back to us, and immediately and permanently delete
it. Do not use, copy or disclose the information contained in this message
or in any attachment.
kdthomas2121
on 19 Mar 2019
sorry about that. I fixed that one now.
lyndsey-ferguson
on 19 Mar 2019
you'll need to bundle install again
lyndsey-ferguson
on 19 Mar 2019
I'm really hopeful that this comes together! It's going to save my CI agents LOADS of test running time! Thanks for all your hard work @lyndsey-ferguson !
amelendezSGY
on 24 Mar 2019
I _believe_ that I have overcome the last major obstacle. This has been hard work, at some points I didn't know what to try next.
Now, all that remains is some major clean up, some automated tests and the refactoring that comes with that.
lyndsey-ferguson
on 25 Mar 2019
From my perspective, the most critical pieces of this parallelized testing system that need to come together for this all to work in my own CI setup are:
• the failed tests need to be rerun as many times as allowed by my multi_scan configuration
• trainer (https://github.com/xcpretty/trainer/tree/master/fastlane-plugin-trainer) can still convert the Xcode plist log to a JUnit report
Bonus points if:
• My XCTestHTMLReport can still generate an HTML report of the tests from the JUnit report properly (https://github.com/TitouanVanBelle/XCTestHTMLReport).
My assumption is that you're hard at work on making sure those first few bullets are addressed (thanks again!). The bonus bullet should work if the Xcode plist log is healthy.
amelendezSGY
on 25 Mar 2019
I think that this current test branch has this covered:
the failed tests need to be rerun as many times as allowed by my multi_scan configuration
I don't currently support trainer, I believe that there is some incompatibility with how multi_scan collates the test_result bundles and trainer. I do not currently support XCTestHTMLReports either. There are some open tickets on that with guidance that I have given, but these last points are not in my current plan for parallelizing multi_scan. The focus is on keeping the existing functionality with the added bonus of re-running tests in parallel.
lyndsey-ferguson
on 25 Mar 2019
Thanks for the heads up. I'll give it all a try and see what happens once you push a release containing the work on your current branch. Best of luck in the home stretch!
amelendezSGY
on 25 Mar 2019
Been trying this out, but it doesn't seem to run the tests in parallel when using fastlane. It runs them on a sim one at a time, I've got disable_concurrent_testing set to false
kdthomas2121
on 28 Mar 2019
@kdthomas2121 I haven't provided much documentation on how to make this happen, so if you didn't already know what I am about to tell you, that's understandable: have you provided a batch_count option to multi_scan? It should the number of sims you want.
Do not set it to more than 6 unless you want everything to fall apart.
If you did do that, I'd love to learn more.
lyndsey-ferguson
on 28 Mar 2019
I've fixed the colorize issue mentioned. Revisit my instructions above, yet change:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-10-parellel-multi_scan"
to
gem 'fastlane-plugin-test_center', '3.6.3.parallelizing'
I tried it by setting batch_count: 2, however didn't see test running in parallel. it splits the tests in to 2 batches and only one batch runs.
ishammoho
on 15 May 2019
did you add parallelize: true to the options passed to multi_scan?
lyndsey-ferguson
on 15 May 2019
Thanks. I didn't have parallelize: true, its running on parallel now. However rerun doesn't seems to be working well. When simulator launches to rerun it seems to be in a switched off state and also it's producing report only for 1 batch (I have asked to run in 3 batches).
ishammoho
on 16 May 2019
@ishammoho would you be able to provide a log?
lyndsey-ferguson
on 16 May 2019
Hi @lyndsey-ferguson !
I can't thank you enough for this project.
at Instabug our tests (unit and UI) took 25m and 1h respectively and it was required to run our release script but as with any sufficiently large test suit we had many flaky tests so we re-ran that 1h tests multiple times, before we can release which wasted a lot of time from multiple members on the team and multi_scan, did help a ton with that!
Now this too, which I just tried it successfully cut that 1h run to 30m on CircleCI and now trying even more aggressive parallelization, this is just mind-blowing 🤯🤯🤯🤯!
Many Many thanks 🙏🙏🙏🙏🙏
yousefhamza
on 22 May 2019
There's a minor issue, which is when it a batch fails it doesn't fail the build.
we are setting fail_build: true
yousefhamza
on 22 May 2019
@yousefhamza thank you for sharing how this helped you. I really value hearing that this plugin _is_ helping.
To confirm: are you saying that you're now saving 30 mins on CircleCI with the parallelization?
Also, is this fail_build: true not working with the parallelization feature work in progress?
lyndsey-ferguson
on 22 May 2019
@yousefhamza That's awesome news. Will you be able to share your lane with us pls? and also wondering whether you setup Simulator on CI at runtime? Thanks.
ishammoho
on 22 May 2019
@lyndsey-ferguson Exactly 👍
@ishammoho here it is.
desc "Run UI tests"
lane :ui_tests do
result = multi_scan(
workspace: "Instabug/Instabug.xcworkspace",
try_count: 3,
fail_build: true,
device: 'iPhone 8 (11.4)',
batch_count: 3,
parallelize: true,
scheme: 'InstabugDemoUITests'
)
unless result[:failed_testcount].zero?
UI.message("There are #{result[:failed_testcount]} legitimate failing tests")
end
end
Not sure what you mean with "setup the simulator in Ci at runtime" but the only interaction we have with the simulator on CI is launching it before running the tests because that sometimes caused the specs to fail with:
xcrun instruments -w "iPhone 8 (11.4) [" || true
yousefhamza
on 22 May 2019
Thanks @yousefhamza , we use https://github.com/plu/simctl this to setup simulator.
Thanks @lyndsey-ferguson for your awesome work. Many thanks. Will try this out again and let you know how it goes.
ishammoho
on 22 May 2019
Also, is this fail_build: true not working with the parallelization feature work in progress?
@lyndsey-ferguson Exactly 👍
@yousefhamza I think that the current code I've made available does not collate the junit report files correctly and thus the code that examines the results does not find the failing tests. I will make sure that functions before I alert you to a fully functional parallelized alpha for testing.
FYI, I have to clean up the code and have just finished refactoring out the retrying portion that works on a batch of tests. Now that I have completed that, I have to port the code that runs the batches of test in parallel to use my new code.
lyndsey-ferguson
on 22 May 2019
Not sure what you mean with "setup the simulator in Ci at runtime" but the only interaction we have with the simulator on CI is launching it before running the tests because that sometimes caused the specs to fail with...
That may be a problem with the CoreSimulatorService hanging. I am putting in something that will quit that service so that it can start up in a functional state. I am considering adding a check to see if it is in that state and doing that automatically (or with a option from you, the user, to indicate that you're okay with that).
lyndsey-ferguson
on 22 May 2019
Thanks @lyndsey-ferguson for your awesome work. Many thanks. Will try this out again and let you know how it goes.
It is my pleasure, truly.
lyndsey-ferguson
on 22 May 2019
Oh, and one other thing, multi_scan creates a clone of the simulator that you've chosen for each 'batch'. Thus, if you ask for a batch count of 3, it will create 3 clones, and delete them when it is done.
lyndsey-ferguson
on 22 May 2019
@lyndsey-ferguson Our pre-start simulator thing isn't related to multi_scan specifically we had this issue since we started on CircleCI
Can't wait for the release 🙏
yousefhamza
on 23 May 2019
@lyndsey-ferguson -- I have been busy with other work and could not get a chance to try the latest improvements although I am following the thread. I am testing this today in my multi_scan. One question I had from earlier is the feature of collating the multiple HTML reports into one report.
For ex: If I run as 3 batches then the output has 3 HTML reports. Do you know if the capability to collate all 3 HTML into 1 HTML report is now available? If yes, could you please point me towards an examples if you dont mind.
Sandhu412-zz
on 24 May 2019
@Sandhu412 No, currently this alpha does not collate any reports. It will, but it does not yet.
lyndsey-ferguson
on 24 May 2019
Hi @lyndsey-ferguson, do you have any ETA for this feature?
I love this tool by the way :)
edias
on 31 May 2019
@edias maybe the fall (September). I know that that is far away, but I only get around 30 mins a day on this ;)
And thanks for the ❤️, tell your friends 😄
lyndsey-ferguson
on 31 May 2019
This seems to be working pretty great!
One question, how can you get a test_result bundle out, while using the parallelize option?
Jon889
on 11 Jun 2019
@Jon889 are you seeing a problem getting the test_result bundle out? I haven't gotten to testing everything to make sure it works, but my "mental model" of what _I do have_ suggests that you should be able to pass true in :result_bundle option and it _should_ place the bundle in your output directory.
lyndsey-ferguson
on 14 Jun 2019
I am working on cleaning up a beta of the feature in this PR
lyndsey-ferguson
on 18 Jun 2019
Everyone this is now in beta. Please test by following the instructions here
lyndsey-ferguson
on 20 Jun 2019
getting 'Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs' when trying bundle install. Plugin file is updated to be
gem 'fastlane-plugin-test_center', '3.8.0.parallelizing.beta.1'
ishammoho
on 20 Jun 2019
I got this issue but it continued to install:
Retrying dependency api due to error (2/4): Bundler::HTTPError Net::HTTPBadGateway:
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>502 Bad Gateway</title>
</head>
<body>
<h1>Error 502 Bad Gateway</h1>
<p>Bad Gateway</p>
<h3>Guru Mediation:</h3>
<p>Details: cache-dca17760-DCA 1560988676 826515299</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
Did yours not install?
lyndsey-ferguson
on 20 Jun 2019
No it didn't
ishammoho
on 20 Jun 2019
what version of bundler are you using?
lyndsey-ferguson
on 20 Jun 2019
Also, did you delete your Gemfile.lock file?
lyndsey-ferguson
on 20 Jun 2019
Bundler version 2.0.1
No didn't delete the gem.lock
ishammoho
on 20 Jun 2019
Also, try the options here: https://stackoverflow.com/a/38410872/71877
lyndsey-ferguson
on 20 Jun 2019
Ok, try following the stack overflow steps. I'm using the same bundler version. I'll try running bundle install again to see if I continue having the same error.
lyndsey-ferguson
on 20 Jun 2019
Cool. will try that. Thanks.
ishammoho
on 20 Jun 2019
The second and third run after deleting the Gemfile.lock succeeded without any error each time.
lyndsey-ferguson
on 20 Jun 2019
getting this error
| Summary for multi_scan (test_center v3.8.0.parallelizing.beta.1) |
+-------------------------+--------------------------------------------------+
| try_count | 3 |
| parallel_testrun_count | 3 |
| testrun_completed_block | #<Proc:0x00007f888daeff20@Fastfile:343 (lambda)> |
+-------------------------+--------------------------------------------------+
[15:42:03]: Could not find option 'force_quit_simulator' in the list of available options: workspace, project, scheme, device, devices, skip_detect_devices, reset_simulator, reinstall_app, app_identifier, only_testing, skip_testing, xctestrun, toolchain, clean, code_coverage, address_sanitizer, thread_sanitizer, open_report, output_directory, output_style, output_files, buildlog_path, include_simulator_logs, suppress_xcode_output, formatter, xcpretty_args, derived_data_path, should_zip_build_products, result_bundle, use_clang_report_name, max_concurrent_simulators, disable_concurrent_testing, skip_build, test_without_building, build_for_testing, sdk, configuration, xcargs, xcconfig, slack_url, slack_channel, slack_message, slack_use_webhook_configured_username_and_icon, slack_username, slack_icon_url, skip_slack, slack_only_on_failure, destination, custom_report_file_name, fail_build, try_count, batch_count, invocation_based_tests, quit_simulators, output_types, parallel_testrun_count, testrun_completed_block
tried with parallelization = true, initially, howevever it didn't work and showed multi_scan has no option of parallelization, then tried with parallel_testrun_count: 3 then got the above error.
ishammoho
on 20 Jun 2019
Right, I've changed the option. I'll update the PR to include the instructions. Can you post an example of how you are calling multi_scan? Also, is there a stack crawl of the exception that occurs for force_quit_simulator?
lyndsey-ferguson
on 20 Jun 2019
@ishammoho okay, I think I know what the problem is. :force_quit_simulator was added to fastlane 22 days ago. As I deleted my Gemfile, _and_, I do not specify the version of fastlane, you probably are using a version of fastlane greater than 22 days old. I'll update this.
lyndsey-ferguson
on 20 Jun 2019
@ishammoho I've removed the possibility of calling multi_scan with :force_quit_simulator as I already have that option. I've released a new beta and have updated the instructions in the PR.
lyndsey-ferguson
on 20 Jun 2019
Thanks @lyndsey-ferguson , now getting this error
[!] You must pass at least one result_bundle_path
Can you post a sample of the Fastfile where you call multi_scan? I need the log too (not just the one error 😖 😄 )
lyndsey-ferguson
on 20 Jun 2019
:)
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+
[08:45:49]: Error: cannot find xctestrun file ''
[08:45:49]: --------------------------
[08:45:49]: --- Step: xchtmlreport ---
[08:45:49]: --------------------------
+------------------+--------------------------+
| Lane Context |
+------------------+--------------------------+
| DEFAULT_PLATFORM | ios |
| PLATFORM_NAME | ios |
| LANE_NAME | ios run_automation_tests |
+------------------+--------------------------+
[08:45:49]: You must pass at least one result_bundle_path
+------+------------------------------------+-------------+
| fastlane summary |
+------+------------------------------------+-------------+
| Step | Action | Time (in s) |
+------+------------------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | opt_out_usage | 0 |
| 3 | default_platform | 0 |
| 4 | Switch to ios setup_simulator lane | 0 |
| 5 | set_scan_xcargs | 0 |
| 💥 | multi_scan | 15 |
| 💥 | xchtmlreport | 0 |
+------+------------------------------------+-------------+
[08:45:49]: fastlane finished with errors
[!] You must pass at least one result_bundle_path
Fastlane looks like this
test_run_block = lambda do |testrun_info|
failed_test_count = testrun_info[:failed].size
if failed_test_count > 0
UI.important('The run of tests would finish with failures due to fragile tests here.')
try_attempt = testrun_info[:try_count]
if try_attempt < 3
UI.header('Since we are using :multi_scan, we can re-run just those failing tests!')
end
end
end
result = multi_scan(
workspace: File.absolute_path('../Com.xcworkspace'),
scheme: 'Automation',
try_count: 3,
parallel_testrun_count: 4,
fail_build: false,
# testrun_completed_block: test_run_block,
# output_files: 'report.html',
# output_types: 'html'
)
unless result[:failed_testcount].zero?
UI.message("There are #{result[:failed_testcount]} legitimate failing tests")
end
rescue => ex
# Logs error instead of failing build
UI.error(ex)
ensure
xchtmlreport(enable_junit: true)
end
end
@ishammoho Here's the problem:
Error: cannot find xctestrun file ''
You can see that immediately after that error, it is trying to run the xchtmlreport action and failing because there is no result_bundle_path.
I need to add some logging to try and determine what the correct xctestrun file path is. Currently I am looking in the derived data path/Build/Products/<scheme name>*.xctestrun. Until I put the logging in, can you go to your Xcode project, right click on the app that is being tested, and show in Finder? One level up should be where the xctestrun file is.
Project:
Right Click:
Finder Location:
Can you perform the same steps and show me the _complete_ name of the xctestrun file?
lyndsey-ferguson
on 20 Jun 2019
If you call multi_scan with the full path to the xctestrun, it will not fail to find it 😉
I still have to fix how I find it though.
lyndsey-ferguson
on 20 Jun 2019
@ishammoho, I have created the following Parallelization Bug: Unable to find built xctestrun file, please use that issue to continue this investigation.
lyndsey-ferguson
on 20 Jun 2019
@ishammoho I've moved your comments to the Issue
lyndsey-ferguson
on 21 Jun 2019
@ishammoho just in case you are not getting notifications from the Issue itself, I have a fix that I'd like you to test. Please see the Issue Parallelization Bug: Unable to find built xctestrun file for instructions.
lyndsey-ferguson
on 21 Jun 2019
Fixed in v3.8.0
lyndsey-ferguson
on 7 Aug 2019
I was reading a blog entry that linked to this Issue and ended up wasting time reading everyone's comments; I just wanted to say thank you to everyone who helped test this and provided me with encouragement and excitement along the way
🤗
lyndsey-ferguson
on 21 Sep 2020
Related issues
BohdanOrlov
·
9Comments
mykola-mokhnach
·
10Comments
Ravi61
·
8Comments
jordan-romeroporter-earnin
·
8Comments
ryanpato
·
4Comments
Most helpful comment
I _believe_ that I have overcome the last major obstacle. This has been hard work, at some points I didn't know what to try next.
Now, all that remains is some major clean up, some automated tests and the refactoring that comes with that.