Fastlane-plugin-test_center: Feature Request: option to not pre-delete MultiScanManager Simulator Clones

[slacklab]

Feature Request

Motivation Behind Feature

This was a question (see below) about an issue that was not foreseen with using multiple instances of a parallelized version of multi_scan. When run in tandem, the second running multi_scan will, as part of a pre-cleanup, delete cloned simulators that are specific to the parallel version of multi_scan.

Unfortunately, this means that the cloned simulators that the first instance of multi_scan created will be deleted by the second instance.

We can avoid this by offering users the option to _not_ delete these simulators.

Feature Description

1. Add an option to delete the simulators as part of a clean up. The default value should be true.
2. Enhanced the check before delete_multi_scan_cloned_simulators is called to check for this new option
3. Add a test to ensure that delete_multi_scan_cloned_simulators is called when the option is true, and it is not called when the option is false.

Alternatives or Workarounds

You can override the code that deletes the cloned simulators. There is a problem with this in that the simulators will fill up your hard drive. Instead, you should override the setup method.

Question Checklist

• [+] Updated fastlane-plugin-test_center to the latest version
• [+] I read the README.md
• [+] I reviewed the example(s) for the action(s) I am using
• [+] I searched for existing GitHub issues

This ### Question Subject

Question Description

Hello, I m glad to use yours plugin.
But faced with problem, don't know how to resolve this:

I run tests at gitlab-runner

One pipeline - is ok.
But when I start two pipelines at the same time
I got

[!] Tests have failed
[14:05:48]: Deleting iPhone SE Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70261072388660>
Invalid device: 33744459-2348-4D68-A2C0-6D98889F5106
[14:05:48]: Deleting iPhone SE Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70261072388660>
Invalid device: 24BED18D-C731-4ADE-886D-6DD87ACA57B1

Cos parallell pipeline before start multi_scan has delete this simulator above.

My config is:

result = multi_scan(
    project: MY_XCODE_PROJECT_FILEPATH,
    try_count: RETRIES,
    fail_build: true,
    scheme: 'ProjectUITests',
    testrun_completed_block: test_run_block,
    parallel_testrun_count: 2,
    result_bundle: true,
    collate_reports: false,
    reset_simulator: false,
    quit_simulators: false,
    prelaunch_simulator: true,
    #skip_detect_devices: true,
    force_quit_simulator: false,
    devices: ["iPhone SE"]
  )

Second pipeline delete simulators from previous running pipeline.
Before start I saw this :

[16:06:34]: Deleting iPhone SE Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70274943379540>
[16:06:51]: Deleting iPhone SE Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70274943379540>
[16:07:09]: Starting test run 1
[16:07:09]: Starting test run 2

This simulators are from previous pipeline and first pipeline has fail with some error:

UITests-Runner.app encountered an error (Failed to prepare device 'iPhone SE Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70274943379540>' for impending launch. (Underlying error: No matching device (29E80744-B134-4175-8BE5-3B3A2BE8F1A6) in set at /Users/new-ci/Library/Developer/CoreSimulator/Devices))

I use plugin 3.8.5 and 3.8.4
Please , can you give me any advice how to disable deleting simulators from current test run for previous test run? (Now its kill simulators)
Or how to fix this logic?

I saw this issue https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/88
but can't to apply this.

[lyndsey-ferguson]

If I understand correctly, you are running multi_scan twice on the same computer. Both calls are asking for parallelly running simulators. Correct?

I purposely delete pre-existing simulator clones before running multi_scan so as to not leave cloned simulators consuming space on your computer.

You _can_ "override" the method that deletes the simulators, but then it is up to you to clean up after yourself. In your Fastfile, I would add this code:

module TestCenter
  module Helper
    module MultiScanManager
      class SimulatorHelper
        def delete_multi_scan_cloned_simulators
          return
        end
      end
    end
  end
end

However, be careful. The more iOS Simulators you run in parallel, the more unstable everything becomes. You may find that your Simulators start crashing, fail to boot, disconnect midway in a testrun, etc.

[slacklab]

Thanks a lot. You are correct, I want to run two job ( 2 simulators at each job with multi_scan) at the same time at one computer.

I try use your code above:
Put this code at start of Fastfile:

fastlane_version "2.60.0"

default_platform :ios

module TestCenter
  module Helper
    module MultiScanManager
      class SimulatorHelper
        def delete_multi_scan_cloned_simulators
          return
        end
      end
    end
  end
end

platform :ios do
  before_all do
    # carthage
  end

MY_XCODE_PROJECT_FILEPATH = File.absolute_path('../Project.xcodeproj')
RETRIES = 2

lane :RunningMultiUITests do
  test_run_block = lambda do |testrun_info|
    failed_test_count = testrun_info[:failed].size

    if failed_test_count > 0
      UI.important('The run of tests would finish with failures due to fragile tests here.')

      try_attempt = testrun_info[:try_count]
      if try_attempt < RETRIES
        UI.header('Since we are using :multi_scan, we can re-run just those failing tests!')
      end
    end
  end

  result = multi_scan(
    project: MY_XCODE_PROJECT_FILEPATH,
    try_count: RETRIES,
    fail_build: true,
    scheme: 'ProjectUITests',
    testrun_completed_block: test_run_block,
    parallel_testrun_count: 2,
    result_bundle: true,
    collate_reports: false,
    reset_simulator: false,
    quit_simulators: false,
    prelaunch_simulator: true,
    force_quit_simulator: false,
    devices: ["iPhone SE"]
  )

end

But I get same old logic: (this log from third job - this job delete simulators on the two previous jobs)

[18:10:29]: Deleting iPhone SE Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70187903132060>
[18:10:37]: Deleting iPhone SE Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70203287293120>
[18:10:45]: Deleting iPhone SE Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70187903132060>
[18:10:53]: Deleting iPhone SE Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70203287293120>
[18:11:09]: Starting test run 1
[18:11:09]: Starting test run 2
[worker 1] [18:11:09]: Starting scan #1 with 4 tests for batch #1.

May, I wrong override function delete_multi_scan_cloned_simulators at Fastfile?
Please, can you get some advice how to disable deleting simulators before run tests?

[lyndsey-ferguson]

My theory is that the TestCenter module is being placed inside the Fastlane module.

Can you put this code in a separate file, simulator_helper_override.rb

module TestCenter
  module Helper
    module MultiScanManager
      class SimulatorHelper
        def delete_multi_scan_cloned_simulators
          return
        end
      end
    end
  end
end

and then in your Fastfile:

require_relative 'path/to/simulator_helper_override.rb'

[lyndsey-ferguson]

It could be that the multi_scan plugin is being loaded after your code override (I'm not sure).

If you put it in your lane, that _could_ work, maybe.

[lyndsey-ferguson]

@slacklab did you suggestion above help? let me know. If other people need this, I will consider adding an option to _not_ delete "old" simulators

[slacklab]

@lyndsey-ferguson Thanks a lot. I've done as you say:

Created external file with override content above. (simulator_helper_override.rb)
And call this from my lane for run tests:

lane :RunningMultiUITests do
  require_relative 'simulator_helper_override.rb'
  ...

It helps me , thanks. Now I can run many parallell tests at same computer without troubles.
Cloned simulators also deleting when tests done. And other run not delete old simulators.

Think, that this option for not delete old simulators would be useful for run at CI.
Cannot understand how to implement run many pipelines for tests at the same time without not delete old simulators

[lyndsey-ferguson]

Thanks for the confirmation. I'm glad that this works.

I take your suggestion that this could be helpful for other people: I'll convert this question to a Feature Request soon.

[lyndsey-ferguson]

@slacklab can you test a fix? If so, please modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "fix-issue-148+146"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[lyndsey-ferguson]

Also, remove your override 😄

[slacklab]

Thanks, I will try this fix some later. Now we are work by our release

[lyndsey-ferguson]

Fixed in v3.8.6