Fastlane-plugin-test_center: Feature Request: capability of splitting the test cases as batch

[karthick-nagendran]

Feature Request

Motivation Behind Feature

I have the test cases where it's interlinked if test cases A is logged in then test B should be ideal if test case B is logged in then test case A should be ideal, but.
Currently, the number of test cases is splitting equally by the random manor and distributing to the simulators.
It will be greater if I have control of splitting the batches by using --only--testing flag so I can distribute the test cases which are all capable of independently running in parallel.

Feature Description

Need a mechanism of splitting the test into batches manually.
The test cases are able to split by only_testing flag so the user should split the batches by only_testing flag then pass into the multi_scan as batches so that the simulators can be started based on these batches.

This going to be a new enactment won't break anything existing functionalities

Alternatives or Workarounds

The alternate way I could think off is can initiate the parallel multi_scan lanes trigger at the same time so the user can split the cases themself by providing only_testing tag finally by using the collate_html_reports lane can merge all the HTML report together

I think internally the test cases are splitting randomly if we decide to split it by ourself then

[lyndsey-ferguson]

So, basically, you want to provide the batches to be tested on the different iOS Simulators yourself, yes? To override the automated splitting/batching?

[karthick-nagendran]

Yes, exactly that's the point.

[lyndsey-ferguson]

First @karthick-nagendran, thank you for the suggestion on how to make the plugin better. I welcome every opportunity to improve my fellow developer's lives :smile:. Your suggestion is a good one and I can imagine other people would also appreciate being able to control which tests are grouped together.

Unfortunately, my focus cannot be on adding new features to the plugin at this time. Would you be up to the task? I can provide guidance on adding such a feature.

Rather than waiting for your to jump to volunteer, I'll post my initial guidance here. If you cannot pick it up, perhaps someone else will come upon the guidance and take care of adding the new feature themselves.

Guidance:

1. Fork the test center plugin repo
2. Add a new FastlaneCore::ConfigItem for a batches option under the batch_count option. It would be an array of an array of strings: i.e.

[ 
  ["TestTarget1/TestSuite1/test1", "TestTarget1/TestSuite1/test2", "TestTarget1/TestSuite1/test3"],
  ["TestTarget1/TestSuite2/test1", "TestTarget1/TestSuite2/test2", "TestTarget1/TestSuite2/test3"],
  ["TestTarget99/TestSuite1/test1", "TestTarget99/TestSuite1/test2", "TestTarget99/TestSuite1/test3"]
]

1. You'll also need to check to see if :batch_count matches batches.size to ensure we're not getting wacky data. You can put the check at the beginning of the run method. It should call Fastlane.user_error! if these do not match.
2. Modify the TestCollector class so that if the batches option is given, the initialize method only sets the @batch_count and @batches instance variables for the class and then exits early. Since you're in there, could you also make all other methods private except for the test_batches method? It would help future readers.
3. Please update the existing tests and add a new test for this new functionality for the multi_scan_spec and the test_collector_spec.

And that's it, 5 steps! There actually may be more, but that is what I can think of at this moment. Thanks if you can do it, and if you cannot, I understand as I cannot either! :tada:

[lyndsey-ferguson]

Project started

[lyndsey-ferguson]

Released in Sponsors-first version of v3.15.0. Available to the general public on October 5th, 2020!

See the docs for more information.

[BytesGuy]

Would this feature help to split test cases into batches that can be run on separate physical Macs? For example, say we have 1000 tests - even running these in parallel on a single machine may take some time (especially if we are running older hardware). Ideally in this scenario I would want to split the tests into batches of 200 and run them on 5 machines at once (later having a job to collate the test results). The coordination of this would be out of scope for the plugin, it would be more down to the CI system, but it is obviously vital to have that ability to split them (preferably automatically) in that way.

[lyndsey-ferguson]

Hi @BytesGuy, no this feature won't help with your use case, but all of the actions in this plugin would. My first thought would be to have a job that runs tests_from_xctestrun to get the all the tests you want to run.

Then, I would split those tests into batches based on the number of other machines (5 in your case) (see this as an example). Using those batches, I would then schedule multi_scan jobs with the only_testing: parameters consisting of the tests in each batch.

Then, a final job would run to use the desired collation-* action you want to get the final report.

[lyndsey-ferguson]

Released in v.3.14.4

[BytesGuy]

Hi @BytesGuy, no this feature won't help with your use case, but all of the actions in this plugin would. My first thought would be to have a job that runs tests_from_xctestrun to get the all the tests you want to run.

Then, I would split those tests into batches based on the number of other machines (5 in your case) (see this as an example). Using those batches, I would then schedule multi_scan jobs with the only_testing: parameters consisting of the tests in each batch.

Then, a final job would run to use the desired collation-* action you want to get the final report.

Thanks, going to give that a try :) Splitting tests in this way would be perfect for speeding up large test suites in CircleCI - if I can get something working we may be able to include this in the docs!

[lyndsey-ferguson]

@BytesGuy whoop, that would be amazing!