Fastlane-plugin-test_center: Build times are slower when using multi_scan to parallelize tests
New Issue Checklist
- [x] Updated
fastlane-plugin-test_centerto the latest version - [x] I read the README.md
- [x] I reviewed the example(s) for the action(s) I am using
- [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson. 😍
Issue Description
Unit Tests multi_scan setup:
multi_scan(
scheme: options[:scheme],
buildlog_path: PIPELINE_LOGS_DIR,
code_coverage: true,
collate_reports: true,
derived_data_path: PIPELINE_DERIVED_DATA_DIR,
output_directory: PIPELINE_OUTPUT_DIR,
output_types: "junit,xcresult",
output_files: options[:output_files],
device: PIPELINE_BUILD_DEVICE,
try_count: 3,
batch_count: 1,
parallel_testrun_count: 3
)
UI Tests multi_scan setup:
multi_scan(
scheme: options[:scheme],
buildlog_path: PIPELINE_LOGS_DIR,
code_coverage: false,
collate_reports: true,
derived_data_path: PIPELINE_DERIVED_DATA_DIR,
output_directory: PIPELINE_OUTPUT_DIR,
output_types: "junit",
output_files: options[:output_files],
device: PIPELINE_BUILD_DEVICE,
try_count: 3,
batch_count: 1,
parallel_testrun_count: 1
)
Right now we are on version 3.11.5 of fastlane-plugin-test_center. I was looking to parallelize our UI tests, but I also wanted to check different configurations of multi_scan to see if I could optimize build times. So I spent a lot of time last week running close to a hundred builds across different multi_scan configurations on both the latest version of fastlane-plugin-test_center (3.14.7) and the current version we are consuming (3.11.5). What I found surprised me and I think may be an indication of a bug with how multi_scan is parallelizing tests. Here are the average build times I'm seeing...
Unit tests on version 3.11.5:
[~38m] batch_count: 1, parallel_testrun_count: 1
[~41m] batch_count: 1, parallel_testrun_count: 2
[~45m] batch_count: 1, parallel_testrun_count: 3
[~36m] batch_count: 2, parallel_testrun_count: 2
[~36m] batch_count: 3, parallel_testrun_count: 3
Unit tests on version 3.14.7 (latest version):
[~42m] batch_count: 1, parallel_testrun_count: 1
[~50m] batch_count: 1, parallel_testrun_count: 2
[~58m] batch_count: 1, parallel_testrun_count: 3
[~48m] batch_count: 2, parallel_testrun_count: 2
[~56m] batch_count: 3, parallel_testrun_count: 3
UI tests on version 3.11.5:
[~37m] batch_count: 1, parallel_testrun_count: 1
[~31m] batch_count: 1, parallel_testrun_count: 2
[~35m] batch_count: 1, parallel_testrun_count: 3
[~33m] batch_count: 2, parallel_testrun_count: 2
[~32m] batch_count: 3, parallel_testrun_count: 3
UI tests on version 3.14.7 (latest version):
[~34m] batch_count: 1, parallel_testrun_count: 1
[~38m] batch_count: 1, parallel_testrun_count: 2
[~43m] batch_count: 1, parallel_testrun_count: 3
[~37m] batch_count: 2, parallel_testrun_count: 2
[~40m] batch_count: 3, parallel_testrun_count: 3
You can see that on the older 3.11.5 version, build times are almost always faster when tests are parallelized in some way. However, on the latest version, 3.14.7, build times are always slower when tests are parallelized and build times are longer in general outside of running UI tests serially. I've spent a little time looking at the logs and comparing them, and I don't see anything standing out. It still looks like tests are being parallelized on the latest version, they're just taking longer to run. I don't know what version in between 3.11.5 and 3.14.7 may have caused this issue, but until it's resolved I can't upgrade to the latest version of this plugin without taking a big hit to build times.
Complete output when running fastlane, including the stack trace and command used
The output is too long for me to embed it in this issue since we have close to 1000 unit tests, and I don't want to expose details about the app I work on. I'm hoping the data I have provided is enough to explain the issue, but I realize this output could be key in debugging it. I can flesh out a dummy project with some unit tests and gather fastlane output that way, so let me know. It wouldn't line up with the rest of the data I've included in this issue, but it would give us some output to look at.
Environment
Unit Tests on version 3.11.5 with batch_count: 3, parallel_testrun_count: 3:
[15:57:05]: fastlane detected a Gemfile in the current directory
🚫 fastlane environment 🚫
Stack
| Key | Value |
| --------------------------- | ------------------------------------------- |
| OS | 10.15.7 |
| Ruby | 2.6.5 |
| Bundler? | false |
| Git | git version 2.29.1 |
| Installation Source | ~/.rbenv/versions/2.6.5/bin/fastlane |
| Host | Mac OS X 10.15.7 (19H2) |
| Ruby Lib Dir | ~/.rbenv/versions/2.6.5/lib |
| OpenSSL Version | OpenSSL 1.1.1g 21 Apr 2020 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 12.1 |
System Locale
| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |
fastlane files:
redacted
fastlane gems
| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.162.0 | 🚫 Update available |
Loaded fastlane plugins:
| Plugin | Version | Update-Status |
| -------------------------------- | ------- | ------------------ |
| fastlane-plugin-test_center | 3.11.5 | 🚫 Update available |
| fastlane-plugin-semantic_release | 1.12.0 | ✅ Up-To-Date |
Loaded gems
| Gem | Version |
| -------------------------------- | ------------ |
| did_you_mean | 1.3.0 |
| slack-notifier | 2.3.2 |
| rouge | 2.0.7 |
| xcpretty | 0.3.0 |
| terminal-notifier | 2.0.0 |
| unicode-display_width | 1.7.0 |
| terminal-table | 1.8.0 |
| plist | 3.5.0 |
| CFPropertyList | 3.0.2 |
| addressable | 2.7.0 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-screen | 0.8.1 |
| tty-cursor | 0.7.1 |
| tty-spinner | 0.9.3 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| unf_ext | 0.0.7.7 |
| unf | 0.1.4 |
| domain_name | 0.5.20190701 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.7 |
| faraday_middleware | 1.0.0 |
| fastimage | 2.2.0 |
| gh_inspector | 1.1.3 |
| mini_magick | 4.10.1 |
| rubyzip | 2.3.0 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.7.6 |
| bundler | 2.1.4 |
| naturally | 2.2.0 |
| simctl | 1.6.8 |
| uber | 0.1.0 |
| declarative | 0.0.20 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.2 |
| mini_mime | 1.0.2 |
| multi_json | 1.15.0 |
| signet | 0.14.0 |
| httpclient | 2.8.3 |
| google-api-client | 0.38.0 |
| nanaimo | 0.3.0 |
| colored2 | 3.1.2 |
| claide | 1.0.3 |
| atomos | 0.1.3 |
| xcodeproj | 1.19.0 |
| public_suffix | 4.0.6 |
| babosa | 1.0.4 |
| excon | 0.78.0 |
| ruby2_keywords | 0.0.2 |
| faraday | 1.1.0 |
| json | 2.3.1 |
| jwt | 2.2.2 |
| os | 1.1.1 |
| memoist | 0.16.2 |
| googleauth | 0.14.0 |
| rake | 13.0.1 |
| digest-crc | 0.6.1 |
| google-cloud-errors | 1.0.1 |
| google-cloud-env | 1.4.0 |
| google-cloud-core | 1.5.0 |
| google-cloud-storage | 1.29.1 |
| emoji_regex | 3.0.0 |
| aws-eventstream | 1.1.0 |
| aws-sigv4 | 1.2.2 |
| aws-partitions | 1.385.0 |
| jmespath | 1.4.0 |
| aws-sdk-core | 3.109.1 |
| aws-sdk-kms | 1.39.0 |
| aws-sdk-s3 | 1.83.1 |
| forwardable | 1.2.0 |
| logger | 1.3.0 |
| stringio | 0.0.2 |
| ipaddr | 1.2.2 |
| openssl | 2.1.2 |
| ostruct | 0.1.0 |
| strscan | 1.0.0 |
| date | 2.0.0 |
| fileutils | 1.1.0 |
| etc | 1.0.1 |
| zlib | 1.0.0 |
| mini_portile2 | 2.4.0 |
| nokogiri | 1.10.10 |
| rexml | 3.1.9 |
| psych | 3.1.0 |
| mutex_m | 0.1.0 |
| webrick | 1.4.2 |
| xctest_list | 1.2.1 |
| colorize | 0.8.1 |
| fastlane-plugin-test_center | 3.11.5 |
| fastlane-plugin-semantic_release | 1.12.0 |
generated on: 2020-11-03
Unit Tests on version 3.14.7 with batch_count: 3, parallel_testrun_count: 3:
[15:57:45]: fastlane detected a Gemfile in the current directory
🚫 fastlane environment 🚫
Stack
| Key | Value |
| --------------------------- | ------------------------------------------- |
| OS | 10.15.7 |
| Ruby | 2.6.5 |
| Bundler? | false |
| Git | git version 2.29.1 |
| Installation Source | ~/.rbenv/versions/2.6.5/bin/fastlane |
| Host | Mac OS X 10.15.7 (19H2) |
| Ruby Lib Dir | ~/.rbenv/versions/2.6.5/lib |
| OpenSSL Version | OpenSSL 1.1.1g 21 Apr 2020 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 12.1 |
System Locale
| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |
fastlane files:
redacted
fastlane gems
| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.165.0 | 🚫 Update available |
Loaded fastlane plugins:
| Plugin | Version | Update-Status |
| -------------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.14.7 | ✅ Up-To-Date |
| fastlane-plugin-semantic_release | 1.12.0 | ✅ Up-To-Date |
Loaded gems
| Gem | Version |
| -------------------------------- | ------------ |
| did_you_mean | 1.3.0 |
| slack-notifier | 2.3.2 |
| atomos | 0.1.3 |
| CFPropertyList | 3.0.2 |
| claide | 1.0.3 |
| colored2 | 3.1.2 |
| nanaimo | 0.3.0 |
| xcodeproj | 1.19.0 |
| rouge | 2.0.7 |
| xcpretty | 0.3.0 |
| terminal-notifier | 2.0.0 |
| unicode-display_width | 1.7.0 |
| terminal-table | 1.8.0 |
| plist | 3.5.0 |
| addressable | 2.7.0 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-screen | 0.8.1 |
| tty-cursor | 0.7.1 |
| tty-spinner | 0.9.3 |
| babosa | 1.0.4 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| excon | 0.78.0 |
| ruby2_keywords | 0.0.2 |
| faraday | 1.1.0 |
| unf_ext | 0.0.7.7 |
| unf | 0.1.4 |
| domain_name | 0.5.20190701 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.7 |
| faraday_middleware | 1.0.0 |
| fastimage | 2.2.0 |
| gh_inspector | 1.1.3 |
| mini_magick | 4.10.1 |
| rubyzip | 2.3.0 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.7.6 |
| bundler | 2.1.4 |
| naturally | 2.2.0 |
| simctl | 1.6.8 |
| uber | 0.1.0 |
| declarative | 0.0.20 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.2 |
| mini_mime | 1.0.2 |
| multi_json | 1.15.0 |
| signet | 0.14.0 |
| memoist | 0.16.2 |
| os | 1.1.1 |
| googleauth | 0.14.0 |
| httpclient | 2.8.3 |
| google-api-client | 0.38.0 |
| google-cloud-env | 1.4.0 |
| google-cloud-errors | 1.0.1 |
| google-cloud-core | 1.5.0 |
| rake | 13.0.1 |
| digest-crc | 0.6.1 |
| google-cloud-storage | 1.29.1 |
| jmespath | 1.4.0 |
| aws-eventstream | 1.1.0 |
| aws-sigv4 | 1.2.2 |
| aws-sdk-core | 3.109.1 |
| aws-sdk-kms | 1.39.0 |
| aws-sdk-s3 | 1.83.1 |
| public_suffix | 4.0.6 |
| json | 2.3.1 |
| jwt | 2.2.2 |
| emoji_regex | 3.0.0 |
| aws-partitions | 1.387.0 |
| forwardable | 1.2.0 |
| logger | 1.3.0 |
| stringio | 0.0.2 |
| ipaddr | 1.2.2 |
| openssl | 2.1.2 |
| ostruct | 0.1.0 |
| strscan | 1.0.0 |
| date | 2.0.0 |
| fileutils | 1.1.0 |
| etc | 1.0.1 |
| zlib | 1.0.0 |
| mini_portile2 | 2.4.0 |
| nokogiri | 1.10.10 |
| rexml | 3.1.9 |
| psych | 3.1.0 |
| mutex_m | 0.1.0 |
| webrick | 1.4.2 |
| trainer | 0.9.1 |
| xctest_list | 1.2.1 |
| colorize | 0.8.1 |
| fastlane-plugin-test_center | 3.14.7 |
| fastlane-plugin-semantic_release | 1.12.0 |
generated on: 2020-11-03
UI Tests on version 3.11.5 with batch_count: 1, parallel_testrun_count: 2:
[15:57:40]: fastlane detected a Gemfile in the current directory
🚫 fastlane environment 🚫
Stack
| Key | Value |
| --------------------------- | ------------------------------------------- |
| OS | 10.15.7 |
| Ruby | 2.6.5 |
| Bundler? | false |
| Git | git version 2.29.1 |
| Installation Source | ~/.rbenv/versions/2.6.5/bin/fastlane |
| Host | Mac OS X 10.15.7 (19H2) |
| Ruby Lib Dir | ~/.rbenv/versions/2.6.5/lib |
| OpenSSL Version | OpenSSL 1.1.1g 21 Apr 2020 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 12.1 |
System Locale
| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |
fastlane files:
redacted
fastlane gems
| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.162.0 | 🚫 Update available |
Loaded fastlane plugins:
| Plugin | Version | Update-Status |
| -------------------------------- | ------- | ------------------ |
| fastlane-plugin-test_center | 3.11.5 | 🚫 Update available |
| fastlane-plugin-semantic_release | 1.12.0 | ✅ Up-To-Date |
Loaded gems
| Gem | Version |
| -------------------------------- | ------------ |
| did_you_mean | 1.3.0 |
| slack-notifier | 2.3.2 |
| rouge | 2.0.7 |
| xcpretty | 0.3.0 |
| terminal-notifier | 2.0.0 |
| unicode-display_width | 1.7.0 |
| terminal-table | 1.8.0 |
| plist | 3.5.0 |
| CFPropertyList | 3.0.2 |
| addressable | 2.7.0 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-screen | 0.8.1 |
| tty-cursor | 0.7.1 |
| tty-spinner | 0.9.3 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| unf_ext | 0.0.7.7 |
| unf | 0.1.4 |
| domain_name | 0.5.20190701 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.7 |
| faraday_middleware | 1.0.0 |
| fastimage | 2.2.0 |
| gh_inspector | 1.1.3 |
| mini_magick | 4.10.1 |
| rubyzip | 2.3.0 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.7.6 |
| bundler | 2.1.4 |
| naturally | 2.2.0 |
| simctl | 1.6.8 |
| uber | 0.1.0 |
| declarative | 0.0.20 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.2 |
| mini_mime | 1.0.2 |
| multi_json | 1.15.0 |
| signet | 0.14.0 |
| httpclient | 2.8.3 |
| google-api-client | 0.38.0 |
| nanaimo | 0.3.0 |
| colored2 | 3.1.2 |
| claide | 1.0.3 |
| atomos | 0.1.3 |
| xcodeproj | 1.19.0 |
| public_suffix | 4.0.6 |
| babosa | 1.0.4 |
| excon | 0.78.0 |
| ruby2_keywords | 0.0.2 |
| faraday | 1.1.0 |
| json | 2.3.1 |
| jwt | 2.2.2 |
| os | 1.1.1 |
| memoist | 0.16.2 |
| googleauth | 0.14.0 |
| rake | 13.0.1 |
| digest-crc | 0.6.1 |
| google-cloud-errors | 1.0.1 |
| google-cloud-env | 1.4.0 |
| google-cloud-core | 1.5.0 |
| google-cloud-storage | 1.29.1 |
| emoji_regex | 3.0.0 |
| aws-eventstream | 1.1.0 |
| aws-sigv4 | 1.2.2 |
| aws-partitions | 1.385.0 |
| jmespath | 1.4.0 |
| aws-sdk-core | 3.109.1 |
| aws-sdk-kms | 1.39.0 |
| aws-sdk-s3 | 1.83.1 |
| forwardable | 1.2.0 |
| logger | 1.3.0 |
| stringio | 0.0.2 |
| ipaddr | 1.2.2 |
| openssl | 2.1.2 |
| ostruct | 0.1.0 |
| strscan | 1.0.0 |
| date | 2.0.0 |
| fileutils | 1.1.0 |
| etc | 1.0.1 |
| zlib | 1.0.0 |
| mini_portile2 | 2.4.0 |
| nokogiri | 1.10.10 |
| rexml | 3.1.9 |
| psych | 3.1.0 |
| mutex_m | 0.1.0 |
| webrick | 1.4.2 |
| xctest_list | 1.2.1 |
| colorize | 0.8.1 |
| fastlane-plugin-test_center | 3.11.5 |
| fastlane-plugin-semantic_release | 1.12.0 |
generated on: 2020-11-03
UI Tests on version 3.14.7 with batch_count: 1, parallel_testrun_count: 2:
[15:57:33]: fastlane detected a Gemfile in the current directory
🚫 fastlane environment 🚫
Stack
| Key | Value |
| --------------------------- | ------------------------------------------- |
| OS | 10.15.7 |
| Ruby | 2.6.5 |
| Bundler? | false |
| Git | git version 2.29.1 |
| Installation Source | ~/.rbenv/versions/2.6.5/bin/fastlane |
| Host | Mac OS X 10.15.7 (19H2) |
| Ruby Lib Dir | ~/.rbenv/versions/2.6.5/lib |
| OpenSSL Version | OpenSSL 1.1.1g 21 Apr 2020 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 12.1 |
System Locale
| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |
fastlane files:
redacted
fastlane gems
| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.165.0 | 🚫 Update available |
Loaded fastlane plugins:
| Plugin | Version | Update-Status |
| -------------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.14.7 | ✅ Up-To-Date |
| fastlane-plugin-semantic_release | 1.12.0 | ✅ Up-To-Date |
Loaded gems
| Gem | Version |
| -------------------------------- | ------------ |
| did_you_mean | 1.3.0 |
| slack-notifier | 2.3.2 |
| atomos | 0.1.3 |
| CFPropertyList | 3.0.2 |
| claide | 1.0.3 |
| colored2 | 3.1.2 |
| nanaimo | 0.3.0 |
| xcodeproj | 1.19.0 |
| rouge | 2.0.7 |
| xcpretty | 0.3.0 |
| terminal-notifier | 2.0.0 |
| unicode-display_width | 1.7.0 |
| terminal-table | 1.8.0 |
| plist | 3.5.0 |
| addressable | 2.7.0 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-screen | 0.8.1 |
| tty-cursor | 0.7.1 |
| tty-spinner | 0.9.3 |
| babosa | 1.0.4 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| excon | 0.78.0 |
| ruby2_keywords | 0.0.2 |
| faraday | 1.1.0 |
| unf_ext | 0.0.7.7 |
| unf | 0.1.4 |
| domain_name | 0.5.20190701 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.7 |
| faraday_middleware | 1.0.0 |
| fastimage | 2.2.0 |
| gh_inspector | 1.1.3 |
| mini_magick | 4.10.1 |
| rubyzip | 2.3.0 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.7.6 |
| bundler | 2.1.4 |
| naturally | 2.2.0 |
| simctl | 1.6.8 |
| uber | 0.1.0 |
| declarative | 0.0.20 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.2 |
| mini_mime | 1.0.2 |
| multi_json | 1.15.0 |
| signet | 0.14.0 |
| memoist | 0.16.2 |
| os | 1.1.1 |
| googleauth | 0.14.0 |
| httpclient | 2.8.3 |
| google-api-client | 0.38.0 |
| google-cloud-env | 1.4.0 |
| google-cloud-errors | 1.0.1 |
| google-cloud-core | 1.5.0 |
| rake | 13.0.1 |
| digest-crc | 0.6.1 |
| google-cloud-storage | 1.29.1 |
| jmespath | 1.4.0 |
| aws-eventstream | 1.1.0 |
| aws-sigv4 | 1.2.2 |
| aws-sdk-core | 3.109.1 |
| aws-sdk-kms | 1.39.0 |
| aws-sdk-s3 | 1.83.1 |
| public_suffix | 4.0.6 |
| json | 2.3.1 |
| jwt | 2.2.2 |
| emoji_regex | 3.0.0 |
| aws-partitions | 1.387.0 |
| forwardable | 1.2.0 |
| logger | 1.3.0 |
| stringio | 0.0.2 |
| ipaddr | 1.2.2 |
| openssl | 2.1.2 |
| ostruct | 0.1.0 |
| strscan | 1.0.0 |
| date | 2.0.0 |
| fileutils | 1.1.0 |
| etc | 1.0.1 |
| zlib | 1.0.0 |
| mini_portile2 | 2.4.0 |
| nokogiri | 1.10.10 |
| rexml | 3.1.9 |
| psych | 3.1.0 |
| mutex_m | 0.1.0 |
| webrick | 1.4.2 |
| trainer | 0.9.1 |
| xctest_list | 1.2.1 |
| colorize | 0.8.1 |
| fastlane-plugin-test_center | 3.14.7 |
| fastlane-plugin-semantic_release | 1.12.0 |
generated on: 2020-11-03
CorbinMontague
All 7 comments
So, this is just a random brainstorm of what could be the problem.
- If Simulator logs are being included (via a Scanfile or an environment variable), that could account for the slowdown as the new simulators are being spun up prior to running tests and all the logs are being pulled out.
- Additionally, if the simulator has a _lot_ of logs, that will now take a lot of time to pull as code was fixed to not pull empty simulator logs, but all the logs. There _should_ be code to _only_ pull the logs from when the simulator was started (try updating fastlane).
- More log collation is done for multiple target projects. So, that could slow things down. To test, you can turn off collation to see if that makes a difference.
collate_reports: false.
Hmmm...well, that's all I can think of right now. Let's talk about those options and then take this issue from what we learn from each other.
@CorbinMontague can you share your thoughts on the above?
lyndsey-ferguson
on 14 Nov 2020
@CorbinMontague are you including simulator logs? Have you been able to turn off collation to see if that makes a difference?
lyndsey-ferguson
on 29 Nov 2020
@lyndsey-ferguson Hey, sorry about the slow response here. I will carve out some time this week to turn off collation and kick off a few test runs.
CorbinMontague
on 30 Nov 2020
@lyndsey-ferguson Simulator logs are not being included. At least I assume not since include_simulator_logs defaults to false and I'm not setting it in my multi_scan call. I'm gathering some data now around collation times to see how they compare in 3.11.5 vs 3.14.8. On a related note, I noticed there were some bug fixes between my current version and the latest in regards to log collation. I no longer have to manually call the collate API for junit files since collate_reports: true now properly collates those, however it does not work for xcresult files so I still have to do those myself.
CorbinMontague
on 1 Dec 2020
@CorbinMontague one thing about the include simulator logs, even though you don't set it, it can be set by a Scanfile or by your CI system sometimes (CircleCI as one example).
If you run multi_scan with the --verbose flag, and you see the scan summary of options before it tests, you will be able to see if include_simulator_logs is true or false.
lyndsey-ferguson
on 1 Dec 2020
@lyndsey-ferguson I don't see a Scanfile anywhere and include_simulator_logs is false as you can see below:
+------------------------------------------------+-----------------------------------------------------+
| Summary for scan 2.168.0 |
+------------------------------------------------+-----------------------------------------------------+
| scheme | [redacted] |
| buildlog_path | ./CICD/Logs |
| code_coverage | true |
| derived_data_path | ./CICD/DerivedData |
| output_directory | ./CICD/Build |
| output_types | junit |
| output_files | unitTests.junit |
| device | iPhone 11 |
| workspace | ./[redacted].xcworkspace |
| skip_detect_devices | false |
| force_quit_simulator | false |
| reset_simulator | false |
| disable_slide_to_type | true |
| reinstall_app | false |
| app_identifier | [redacted] |
| clean | false |
| open_report | false |
| should_zip_build_products | false |
| use_clang_report_name | false |
| disable_concurrent_testing | false |
| build_for_testing | true |
| slack_use_webhook_configured_username_and_icon | false |
| slack_username | fastlane |
| slack_icon_url | https://fastlane.tools/assets/img/fastlane_icon.png |
| skip_slack | false |
| slack_only_on_failure | false |
| xcodebuild_command | env NSUnbufferedIO=YES xcodebuild |
| fail_build | true |
| include_simulator_logs | false |
| skip_build | false |
| xcode_path | /Applications/Xcode.app |
+------------------------------------------------+-----------------------------------------------------+
I also had a look at my build logs when I gathered all this data originally and it looks like in the case of both 3.11.5 and 3.14.7, collating the logs took around 2-3 minutes. So the additional time seems to be happening during the test runs from what I can tell.
I'm probably not going to have time to push this issue forward any more for a while, so if you aren't sure what's going on I guess we can close this issue until I'm able to reproduce this with a sample project so I can include full logs for us to debug.
CorbinMontague
on 3 Dec 2020
Sounds good. I'll close this for now. You could try using this sampler to see where the time is going:
https://github.com/rbspy/rbspy
I used it when I was developing the parallelized version of multi_scan and saw that it was taking a lot of time (I found out that each fork was buffering the console output and had to turn that off to send the output to the console right away -- which I had piped to a file on disk).
lyndsey-ferguson
on 12 Dec 2020
Related issues
ericwengelking
·
10Comments
lyndsey-ferguson
·
6Comments
ishammoho
·
9Comments
lucasecf
·
11Comments
dvataksi16
·
9Comments