Fastlane-plugin-test_center: Pipeline timeout with 3.11.6 version

[ssharma0312]

fastlane-plugin-test_center
Version: 3.11.6
Xcode Version: 11.4.1
Pipelines: GitLab Pipelines

With latest version 3.11.6, I noticed the pipeline is times out. After completion of test execution for all tests, the Simulators never shutdowns, and pipeline never finishes. After rolling back to 3.11.5 version, this issues seems to be resolved. Unfortunately, I don't have any logs.

[lyndsey-ferguson]

Hello @ssharma0312. Thank you for reporting a problem. However, I do need more information in order to provide assistance.

1. You didn't state it, but looks like an issue with the multi_scan action. Can you confirm?
2. Please re-run your pipeline with the --verbose flag and collect the logs. Then, remove any sensitive data (passwords, tokens, private company data, etc.), and paste the logs here.
3. Please run the command fastlane env and copy and paste that information here. There shouldn't be any sensitive data in that block of text, but review it and remove any data is. This should include what gems you are using, and how you are calling the action.

[ssharma0312]

1

Yes, issue in multi_scan

2

Can you please provide an example where to pass this flag "--verbose"

3

Stack

| Key | Value |
| --------------------------- | -------------------------------------------------------------- |
| OS | 10.15.5 |
| Ruby | 2.6.3 |
| Bundler? | false |
| Git | git version 2.24.2 (Apple Git-127) |
| Installation Source | /usr/local/bin/fastlane |
| Host | Mac OS X 10.15.5 (19F101) |
| Ruby Lib Dir | /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib |
| OpenSSL Version | LibreSSL 2.8.3 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 11.4.1 |

System Locale

| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_CA.UTF-8 | ✅ |
| LC_ALL | | |
| LANGUAGE | | |

fastlane gems

| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.154.0 | 🚫 Update available |

Loaded fastlane plugins:

| Plugin | Version | Update-Status |
| --------------------------- | ------- | ------------------ |
| fastlane-plugin-test_center | 3.11.5 | 🚫 Update available |

Loaded gems

| Gem | Version |
| --------------------------- | ------------ |
| did_you_mean | 1.3.0 |
| slack-notifier | 2.3.2 |
| atomos | 0.1.3 |
| CFPropertyList | 2.3.6 |
| claide | 1.0.3 |
| colored2 | 3.1.2 |
| nanaimo | 0.3.0 |
| xcodeproj | 1.17.1 |
| rouge | 2.0.7 |
| xcpretty | 0.3.0 |
| terminal-notifier | 2.0.0 |
| unicode-display_width | 1.7.0 |
| terminal-table | 1.8.0 |
| plist | 3.5.0 |
| public_suffix | 4.0.5 |
| addressable | 2.7.0 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-screen | 0.8.1 |
| tty-cursor | 0.7.1 |
| tty-spinner | 0.9.3 |
| babosa | 1.0.3 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| excon | 0.76.0 |
| faraday | 1.0.1 |
| unf_ext | 0.0.7.7 |
| unf | 0.1.4 |
| domain_name | 0.5.20190701 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.6 |
| faraday_middleware | 1.0.0 |
| fastimage | 2.2.0 |
| gh_inspector | 1.1.3 |
| json | 2.1.0 |
| mini_magick | 4.10.1 |
| rubyzip | 2.3.0 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.7.6 |
| bundler | 2.1.4 |
| naturally | 2.2.0 |
| simctl | 1.6.8 |
| jwt | 2.2.1 |
| uber | 0.1.0 |
| declarative | 0.0.20 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.2 |
| mini_mime | 1.0.2 |
| multi_json | 1.15.0 |
| signet | 0.14.0 |
| memoist | 0.16.2 |
| os | 1.1.0 |
| googleauth | 0.13.0 |
| httpclient | 2.8.3 |
| google-api-client | 0.38.0 |
| google-cloud-env | 1.3.3 |
| google-cloud-errors | 1.0.1 |
| google-cloud-core | 1.5.0 |
| rake | 13.0.1 |
| digest-crc | 0.6.1 |
| google-cloud-storage | 1.27.0 |
| emoji_regex | 3.0.0 |
| jmespath | 1.4.0 |
| aws-partitions | 1.347.0 |
| aws-eventstream | 1.1.0 |
| aws-sigv4 | 1.2.1 |
| aws-sdk-core | 3.104.3 |
| aws-sdk-kms | 1.36.0 |
| aws-sdk-s3 | 1.75.0 |
| forwardable | 1.2.0 |
| logger | 1.3.0 |
| stringio | 0.0.2 |
| ipaddr | 1.2.2 |
| openssl | 2.1.2 |
| ostruct | 0.1.0 |
| strscan | 1.0.0 |
| date | 2.0.0 |
| fileutils | 1.1.0 |
| etc | 1.0.1 |
| io-console | 0.4.7 |
| zlib | 1.0.0 |
| libxml-ruby | 3.1.0 |
| rexml | 3.1.9 |
| psych | 3.1.0 |
| mutex_m | 0.1.0 |
| webrick | 1.4.2 |
| xctest_list | 1.2.1 |
| colorize | 0.8.1 |
| fastlane-plugin-test_center | 3.11.5 |
`

Also, I just updated to 3.13.0, the pipeline took 3 hours to finish vs what it usually takes on v3.11.5
I suspect it's duplicating & re-running tests, once you clarify on --verbose flag details, I can get the logs on both versions.

[lyndsey-ferguson]

Thank you for the response. For #2, when you run your fastlane, for example:

bundle exec fastlane mylane --verbose

[lyndsey-ferguson]

Closed due to lack of logs or more info required to debug.

[ssharma0312]

@lyndsey-ferguson apologies for late response, so I made two branches to carefully compare the difference. One branch is one version 3.11.5 which completes around 55 minutes where as another branch I updated to latest version of this plugin at 3.14.3. Both times the time taken by branch with latest version i.e. 3.14.3 is almost double. I run using --verbose flag so attached complete raw logs. For privacy reasons, some info in the log is masked.
Uploading CompleteRawLog.txt…

[ssharma0312]

[ryanpato]

I'll be interested to see the log too but I can't open your link.

I've only just recently started testing multi-scan, but I see a similar sounding issue myself when working with multiple test-plan configurations and multiple devices in one run, with a try count of 2, and forcefully killing/erasing simulators.

Originally I wanted to do something like this: (where testplan has two enabled configurations)

def runthetests 
  multi_scan(device: device_01, device_02, testplan: testplan)
end

But something expected to take 40 minutes (even running tests sequentially and concurrently) took over 2 hours before it timed out.

Anyway, i'll figure it out and raise a ticket properly when I get time - but mentioning the following incase your issue is similar - i'm currently trying something like this:

def runthetests 
  multi_scan(device: device_01, only_testing_config: config01)
  multi_scan(device: device_01, only_testing_config: config02)
  multi_scan(device: device_02, only_testing_config: config01)
  multi_scan(device: device_02, only_testing_config: config02)
end

Which runs within the expected time, but means I will have to collate the reports and do a bit of tweaking

[lyndsey-ferguson]

@ssharma0312 The log file link in your comment is an empty link: [Uploading CompleteRawLog.txt…]()

Can you try again?

@ryanpato you're facing a problem that I am currently working on. I have limited time that I can work on this plugin, so you'll have to be patient 😃

[ssharma0312]

@lyndsey-ferguson sorry about that, try now:
CompleteRawLog.txt

[lyndsey-ferguson]

Thank you, that worked

[lyndsey-ferguson]

@ssharma0312 could you provide both logs: for the slow and the normal? I'd like to do a diff of both (or you can do that yourself and provide me with the diff). Again, please use the --verbose flag for both runs.

[ssharma0312]

@lyndsey-ferguson I start to suspect it could be slow network at that specific time. I spent more time running pipeline on the latest version 3.14.3, noticed now the execution time has dropped down to pretty much similar time as old version 3.11.5. I don't it's any issue. Closing Issue.