Fastlane-plugin-test_center: multi_scan picks wrong testplan when there are multiple test runs

[0rtm]

New Issue Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example(s) for the action(s) I am using
• [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

If you love this fastlane plugin, consider sponsoring it or asking your company to sponsor it. I would really appreciate any
gesture: https://github.com/sponsors/lyndsey-ferguson.

Issue Description

I am trying to use multi_scan for UI tests. I have 3 test plans for one target.
For simplicity let's call them testPlanA, testPlanB, TestPlanC

I am invoking multi scan as follows:

multi_scan(
...
testplan: "testPlanB"
)

When pipeline runs I am getting output of this form:

After building, found testrun files ['.../testPlanA_simulator_..._.xctestrun','.../testPlanB_simulator_..._.xctestrun','.../testPlanC_simulator_..._.xctestrun', ]
choosing 1st

Problem that first testrun does not contain tests from testPlanB, so when testPlanB is executed multi_scan is unable to find its tests so it runs testPlanA instead.

So I think, multi_scan should not pick testrun without checking which test plan is going to be used

Complete output when running fastlane, including the stack trace and command used

Environment

[lyndsey-ferguson]

@0rtm thanks for the report!

When developing this feature, the person helping me had this setup and I put in this logic to detect that. It _should_ modify which test runs it is looking at.

Can you provide the details that were included in the bug template?

<!-- Please run `fastlane env` and copy the output below. This will help us help you :+1:
If you used the `--capture_output` option, please remove this block as it is already included there. -->

[lyndsey-ferguson]

Oh, and @0rtm , great avatar!

[0rtm]

@0rtm thanks for the report!

When developing this feature, the person helping me had this setup and I put in this logic to detect that. It _should_ modify which test runs it is looking at.

Can you provide the details that were included in the bug template?

<!-- Please run `fastlane env` and copy the output below. This will help us help you :+1:
If you used the `--capture_output` option, please remove this block as it is already included there. -->

Unfortunately I cannot provide you with the details of the original run.
However, I setup an example project https://github.com/0rtm/multi_scan_test with a configuration very similar to the original.
Problem is I cannot make it produce multiple testrun files. While my original project has it happen every time multi_scan is run

[lyndsey-ferguson]

fastlane env will only show the versions of tools you are using.

From my experience, I was able to get multiple xctestrun files when the Scheme had multiple test plans. If your Scheme has multiple testplans, do you get multiple xctestrun files when you press Command-Shift-U?

[0rtm]

fastlane env will only show the versions of tools you are using.

From my experience, I was able to get multiple xctestrun files when the Scheme had multiple test plans. If your Scheme has multiple testplans, do you get multiple xctestrun files when you press Command-Shift-U?

I am getting just one

[0rtm]

fastlane env will only show the versions of tools you are using.
From my experience, I was able to get multiple xctestrun files when the Scheme had multiple test plans. If your Scheme has multiple testplans, do you get multiple xctestrun files when you press Command-Shift-U?

I am getting just one

I figured out what was wrong. My test plans are named UITests and Localization_UITests.
So the regex #{Scan.config[:derived_data_path]}/Build/Products/*#{scan_options[:testplan]}*.xctestrun" gets both of them, and picks just the first one

[lyndsey-ferguson]

So, how Xcode generates the name of the xctestrun files is a bit of a mystery to me. Can you provide the names of both of the xctestrun files? I want to see if there is a better pattern, maybe an _ is used to separate the rest of the name? Plus, I need to see if there are suffixes and prefixes that it can look for, it cannot just be a glob that looks for anything_TESTPLAN_ because that would catch your usecase.

[0rtm]

So, how Xcode generates the name of the xctestrun files is a bit of a mystery to me. Can you provide the names of both of the xctestrun files? I want to see if there is a better pattern, maybe an _ is used to separate the rest of the name? Plus, I need to see if there are suffixes and prefixes that it can look for, it cannot just be a glob that looks for anything_TESTPLAN_ because that would catch your usecase.

So here is the test runs Xcode generated.

multi_scan invocation:

multi_scan(
  clean: true,
  code_coverage: true,
  output_types: 'junit',
  workspace:Dir.entries('..').select {|f| f=~ /.*\.xcworkspace/ }[0],
  output_directory: './reports',
  output_files: 'UITests.junit',
  scheme:'SomeOtherScheme',
  try_count: 3,
  xcargs: "ONLY_ACTIVE_ARCH=YES",
  parallel_testrun_count: 2,
  reset_simulator: true,
  prelaunch_simulator: true,
  testplan: 'UITests',
  fail_build: true)

terminal output:

 After building, found xctestrun files ["./DerivedData/DemoApp/Build/Products/SomeOtherScheme_LocalizationUITests_iphonesimulator13.5-x86_64.xctestrun", "./DerivedData/DemoApp/Build/Products/SomeOtherScheme_UITests_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)

So It chooses LocalizationUITests over UITests as it is precedes alphabetically

[vdavydovHH]

This may sound silly, but try double quotes for the test plan parameter

[lyndsey-ferguson]

@0rtm, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-232-wrong-testplan-selected"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[0rtm]

@0rtm, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-232-wrong-testplan-selected"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

It works correctly now!

After building, found xctestrun files ["./DerivedData/DemoApp/Build/Products/SomeOtherScheme_UITests_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-06-17 18:54:38.06]: Removing report files generated by the build
DEBUG [2020-06-17 18:54:38.06]:   ./reports/UITests.junit
DEBUG [2020-06-17 18:54:38.07]: Looking in Scheme /DemoApp/DemoApp.xcodeproj/xcshareddata/xcschemes/SomeOtherScheme.xcscheme' for any testplans
DEBUG [2020-06-17 18:54:38.08]: TestCollector found testplans: []
DEBUG [2020-06-17 18:54:38.08]:   using :testplan option, UITests, using found one: 
DEBUG [2020-06-17 18:54:38.08]: Getting tests from xctestrun file at '/DemoApp/DerivedData/DemoApp/Build/Products/SomeOtherScheme_UITests_iphonesimulator13.5-x86_64.xctestrun'

[lyndsey-ferguson]

Great, thanks for the quick response!!!!

[lyndsey-ferguson]

Fixed in v3.11.2