Fastlane-plugin-test_center: All the tests are retried instead of just the failed one when -quiet passed in the :xcargs

[Ravi61]

Question Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example(s) for the action(s) I am using
• [x] I searched for existing GitHub issues

Question Subject

I am using multi_scan to retry ui tests but it seems to be retrying all the tests instead of retrying just the failed ones.

Question Description

I am using the below command.

multi_scan(
      project: 'ios/Project.xcodeproj',
      scheme: 'ProjectUITests',
      destination: "name=#{deviceName}",
      clean: false,
      derived_data_path: './d',
      only_testing: ['ProjectUITests/ProjectPerformanceTests'],
      output_types: 'junit',
      output_files: 'report.junit',
      build_for_testing: true,
      result_bundle: true,
      fail_build: false,
      skip_build: false,
      output_directory: 'reports',
      configuration: 'Debug',
      prelaunch_simulator: false,
      xcargs: xcargs,
      disable_concurrent_testing: true,
      retry_test_runner_failures: true,
      try_count: 3,
      testrun_completed_block: test_run_block
)

It retries all the tests in the suit instead of just the failed one. When I checked the junit file, it seems to have no logs, even though the parallel testing option is disabled.

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='0' failures='0'/>

Logs reports 0 number of tests which should have been 2 and 0 number of failures which should have been 1.

** TEST EXECUTE FAILED **
[19:11:15]: Exit status: 65
[19:11:16]: Successfully sent Slack notification
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

[19:11:16]: Everything is fine, let's continue to try 2 for batch 1
[19:11:18]: Starting scan #2 with 2 tests for batch #1.

Scan summary:

+------------------------------------------------+---------------------------------------------------------------------------------+
|                                                     Summary for scan 2.134.0                                                     |
+------------------------------------------------+---------------------------------------------------------------------------------+
| project                                        | ios/Project.xcodeproj                                                           |
| scheme                                         | ProjectUITests                                                                  |
| clean                                          | false                                                                           |
| derived_data_path                              | ./d                                                                             |
| only_testing                                   | ["ProjectUITests/ProjectPerformanceTests/testHomePLT",                          |
|                                                | "ProjectUITests/ProjectPerformanceTests/testOfficialStorePLT"]                  |
| output_types                                   | junit                                                                           |
| output_files                                   | report.junit                                                                    |
| fail_build                                     | false                                                                           |
| skip_build                                     | false                                                                           |
| output_directory                               | /Users/test/Projects/ios-project/reports/ProjectUITests-batch-1                 |
| configuration                                  | Debug                                                                           |
| prelaunch_simulator                            | false                                                                           |
| xcargs                                         | -quiet -UseNewBuildSystem=YES OTHER_SWIFT_FLAGS='-DPERFORMANCE_TESTING'         |
|                                                | GCC_PREPROCESSOR_DEFINITIONS='$GCC_PREPROCESSOR_DEFINITIONS                     |
|                                                | PERFORMANCE_TESTING=1'  -parallel-testing-enabled NO -resultBundlePath          |
|                                                | '/Users/test/Projects/ios-project/reports/ProjectUITests-batch-1/repo           |
|                                                | rt.xcresult'                                                                  |
| disable_concurrent_testing                     | true                                                                            |
| skip_detect_devices                            | false                                                                           |
| force_quit_simulator                           | false                                                                           |
| reset_simulator                                | false                                                                           |
| reinstall_app                                  | false                                                                           |
| app_identifier                                 | com.project.Project                                                             |
| xctestrun                                      | ./d/Build/Products/ProjectUITests_iphoneos13.5-arm64.xctestrun                  |
| open_report                                    | false                                                                           |
| buildlog_path                                  | ~/Library/Logs/scan                                                             |
| include_simulator_logs                         | false                                                                           |
| should_zip_build_products                      | false                                                                           |
| use_clang_report_name                          | false                                                                           |
| slack_use_webhook_configured_username_and_icon | false                                                                           |
| slack_username                                 | fastlane                                                                        |
| slack_icon_url                                 | https://s3-eu-west-1.amazonaws.com/fastlane.tools/fastlane.png                  |
| skip_slack                                     | false                                                                           |
| slack_only_on_failure                          | false                                                                           |
| xcodebuild_command                             | env NSUnbufferedIO=YES xcodebuild                                               |
| xcode_path                                     | /Applications/Xcode.app                                                         |
| build_for_testing                              | false                                                                           |
| result_bundle                                  | false                                                                           |
+------------------------------------------------+---------------------------------------------------------------------------------+

PS: I am running this in an iOS 13.5, iPhone 7 device and not the simulator.

[lyndsey-ferguson]

Hi @Ravi61, thanks for the report. In the template for creating issues, there are instructions for providing more info (which I need).

I need the full logs with the --verbose flag, and I need the output of fastlane env.

AND, if you _really_ want me to solve this faster, a sample with a project and Fastfile will help alot.

[intoxicated]

This happened to me too. FYI, I use # of test cases with only-testing option. In addition, if you set try_count to 0, test will not be executed at all. try_count is a bit misleading here.

[lyndsey-ferguson]

@intoxicated what would you expect when you tell multi_scan to try to test zero times? This is not "retry" but "try".

[intoxicated]

Based on your documentation, its The number of times to retry running tests via scan

[Ravi61]

Hey, sorry I could not update details here due to personal commitments, but I tried this in a sample app and it worked perfectly fine. Upon checking the diff, I found that we were passing -quiet argument in xcargs parameter, which somehow was causing this issue. Removing that argument worked. If you'd still like anymore details, let me know.

@intoxicated maybe if you are facing the same issue, you can try the above thing or can share the details as @lyndsey-ferguson asked?

[lyndsey-ferguson]

@intoxicated excellent point, I'll look into either making this truly "retry" or "try": either in the docs or the code. I'll admit, I'm not completely focused these days with what's going on in the world and didn't look deeply into this...would you create a separate ticket for the try_count of 0 bug to keep Issues focused and keep me focused too?

@Ravi61 I think that multi_scan should at least issue a very big warning if someone else passes the -quiet option. Actually, I think it should issue the warning that quiet breaks the retry mechanism and then turn it off. Thougths?

[lyndsey-ferguson]

@intoxicated I think that the way that I've been using this and promoting this is that "try" is the number of tries. It seems to me that it would make sense to update the docs, add code to throw in a warning that "0" (zero) is an invalid :try_count and reset to to 1. Thoughts?

[lyndsey-ferguson]

Fixed in v3.10.3