Fastlane-plugin-test_center: multi-scan parallel run with 1+ devices will only run on one of those devices

[bpoplauschi]

New Issue Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example(s) for the action(s) I am using
• [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

The issue is this:

• parallel running multi_scan with several devices doesn't function as expected
  
  
  
  • if we run multi_scan with parallel off (parallel_testrun_count = 1 and batch_count = 1) multiple devices and multiple targets, then fastlane builds each target once and then runs it on every device
  
  
  • but if we enable parallelization with the same settings, multi_scan only runs on one of the devices:
  
  

  multi_scan(
    scheme: scheme, 
    devices: ["iPhone 8 (13.3)", "iPhone 8 (12.4)", "iPhone 8 (11.4)"],
    code_coverage: false, 
    derived_data_path: "./DerivedData", 
    output_directory: "./reports", 
    formatter: "xcpretty-json-formatter", 
    try_count: @retries_count,
    only_testing: ["TestsTarget1", "TestTarget2", "TestsTarget3"], 
    testrun_completed_block: test_run_block, 
    parallel_testrun_count: 3, 
    batch_count: 3
  )

- in this case, all the targets will be tested against "iPhone 8 (13.3)". Do you have any idea why this happens or can use any more info from me? ### Environment

| Key                         | Value                                       |
| --------------------------- | ------------------------------------------- |
| OS                          | 10.15.4                                     |
| Ruby                        | 2.6.3                                       |
| Bundler?                    | true                                        |
| Git                         | git version 2.21.1 (Apple Git-122.3)        |
| Installation Source         | ~/.rvm/gems/ruby-2.6.3/bin/fastlane         |
| Host                        | Mac OS X 10.15.4 (19E287)                   |
| Ruby Lib Dir                | ~/.rvm/rubies/ruby-2.6.3/lib                |
| OpenSSL Version             | OpenSSL 1.1.1d  10 Sep 2019                 |
| Is contained                | false                                       |
| Is homebrew                 | false                                       |
| Is installed via Fabric.app | false                                       |
| Xcode Path                  | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version               | 11.3.1                                      |

| Gem      | Version | Update-Status      |
| -------- | ------- | ------------------ |
| fastlane | 2.144.0 | 🚫 Update available |

| Plugin                                    | Version | Update-Status      |
| ----------------------------------------- | ------- | ------------------ |
| fastlane-plugin-test_center               | 3.10.1  | 🚫 Update available |
| fastlane-plugin-firebase_app_distribution | 0.1.4   | ✅ Up-To-Date       |

PS: Very nice work, this project is very helpful! Keep up the good and constant work!

[lyndsey-ferguson]

Hello @bpoplauschi, this seems like a bug to me. It _should_ create 3 clones of each of the devices that you provided and split the tests on each device. I don't have a lot of time to look at it right now, but I would hazard a guess that what is happening is that I am not providing the :destination correctly (only specifying one simulator) where the clone simulators are being set up

If you'd like to take a crack at it, I'm open to PRs can provide guidance. It may a while before I can get to this.

[lyndsey-ferguson]

@bpoplauschi
Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-216-multiple-devices-doesnt-work"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[lyndsey-ferguson]

Fixed in v3.11.4

[bpoplauschi]

@lyndsey-ferguson I really appreciate you looking into this issue and fixing it. I no longer work on that project, so I can't try out the fix, but maybe one of my former teammates can look into this: @nfagen1 @jleyrer @akoller-vs (Hi guys!).

[lyndsey-ferguson]

@BohdanOrlov thank you for connecting me with your former team members -- hopefully when they test, they will see that it is fixed.

I hope that you're enjoying your current project!

[akoller-vs]

@lyndsey-ferguson We finally got around to testing this newer version of the plugin and parallel runs with multiple devices appears to be fixed. However, we're seeing that failed tests are only reran when they fail on _all_ selected devices. If a test fails on only 1 device when using multiple devices, a retry test run correctly starts but doesn't actually rerun the failed tests.

[lyndsey-ferguson]

@akoller-vs thanks for testing and the info! Would you be able to open up a new issue with any logs/env details that you're allowed to provide?

🙏

[akoller-vs]

@lyndsey-ferguson Sure, here it is https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/263.

[lyndsey-ferguson]

Thanks, prioritized!