Fastlane-plugin-test_center: Missing tests if the batch fails to start

[lyndsey-ferguson]

Hi @Cyberbeni, thank you for the Issue, and I apologize for the delay in responding.

In order to debug this, I need a few things.

I do not know if this is failing because multi_scan fails to continue testing, or if collate_junit_reports is failing to collect the tests properly.

If you provide the full logs, without the color codes, that will help me answer the first concern as I will be able to see all the tests that multi_scan is trying to test.

Try this:

FASTLANE_DISABLE_COLORS=1 bundle exec fastlane <insert_your_lane_name_here> --verbose

Secondly, if you turn off report collating, and attach the junit reports (feel free to anonymize the test names), that will help me review the results to see if the problem is collation. I will also be able to run the collation on my computer to see if there is a problem there.

Pass collate_reports: false to multi_scan to turn off collation.

Thanks for your collaboration!

[Cyberbeni]

Easy reproduction of "fails to start" tests is if you try parallel testing with Xcode 11.2.1 on 10.15.2, pretty much all of the batches will fail to start and none of them will be retired.

[lyndsey-ferguson]

It will be a while before I am able to test with that same setup. If you can provide me with the logs and sample files, it will potentially help me resolve this faster.

[lyndsey-ferguson]

I was able to download Xcode 11.2.1 and run a parallel test run. I cannot upgrade to 10.15.2 for various reasons right now.

Still, I was hoping that I could reproduce this issue: I cannot.

I'd like to repeat my request from above:

If you provide the full logs, without the color codes, that will help me answer the first concern as I will be able to see all the tests that multi_scan is trying to test.

Try this:

FASTLANE_DISABLE_COLORS=1 bundle exec fastlane <insert_your_lane_name_here> --verbose

Secondly, if you turn off report collating, and attach the junit reports (feel free to anonymize the test names), that will help me review the results to see if the problem is collation. I will also be able to run the collation on my computer to see if there is a problem there.

Pass collate_reports: false to multi_scan to turn off collation.

If you're not able to provide this information, no worries. I can close this Issue and re-open it when someone else can reproduce the issue.

[Cyberbeni]

Ah, it was probably 10.15.1 where the issue happened 100% of the time, like here: https://github.com/fastlane/fastlane/issues/15640

I can't reproduce the issue on my personal Mac. But I also remember having similar issue with iOS9 simulator where some of the tests were just not ran the first time, so they weren't retried.

https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/0c0a827bf6f56adc6f6cde2ab7c4d4381695eb75/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb#L234

Subtracting the passed tests from the previous tests instead of rerunning the failed tests might be able to solve the issue.

[lyndsey-ferguson]

So, you are saying that some of the tests are not run by xcodebuild test the first time? In your logs that you have not posted, do you see the same tests listed for -only-testing as are in the scan summary table?

For example:

+----------------------------+------------------------------------------------------------------------------------------+

|                                                Summary for scan 2.99.1                                                |

+----------------------------+------------------------------------------------------------------------------------------+

| scheme                     | Foo                                                                           |

| result_bundle              | true                                                                                     |

| test_without_building      | true                                                                                     |

| derived_data_path          | /Users/fresh/Library/Developer/Xcode/Derived/Foo-fa8a8a  |

| workspace                  | ./Frenchfie.xcworkspace                                                                 |

| devices                    | ["iPhone 7 (11.4)"]                                                                      |

| code_coverage              | true                                                                                     |

| skip_build                 | false                                                                                    |

| buildlog_path              | ~/Library/Logs/scan                                                                      |

| include_simulator_logs     | false                                                                                    |

| disable_concurrent_testing | false                                                                                    |

| should_zip_build_products  | false                                                                                    |

| open_report                | false                                                                                    |

| skip_slack                 | false                                                                                    |

| slack_only_on_failure      | false                                                                                    |

| use_clang_report_name      | false                                                                                    |

| clean                      | false                                                                                    |

| output_directory           | fastlane/test_output/multi_scan                                                          |

| only_testing               | ["Foo/abc/test1",                                   |

|                            | "Foo/abc/test2",                               |

| output_types               | junit,html                                                                               |

| output_files               | multi_scan_report.junit,multi_scan_report.html                                           |

| fail_build                 | true                                                                                     |

| xcode_path                 | /Applications/Xcode.app                                                                  |

+----------------------------+------------------------------------------------------------------------------------------+

And:

INFO [2018-07-25 16:55:17.13]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace ./Foo.xcworkspace -scheme Frog -destination 'platform=iOS Simulator,id=44DEAC14-9578-4892-BFA9-56EE826DACD6' -destination 'platform=iOS Simulator,id=44DEAC14-9578-4892-BFA9-56EE826DACD6' -derivedDataPath '/Users/fresh/Library/Developer/Xcode/Derived/Foo-gwxgyqjedayhefahmjrbldnzhgtx' -resultBundlePath 'fastlane/test_output/multi_scan/Foo.test_result' -enableCodeCoverage YES -only-testing:Foo/abc/test1 -only-testing:Foo/abc/test2 test-without-building | tee '/Users/fresh/Library/Logs/scan/Foo.log' | xcpretty  --report junit --output '/Users/fresh/Projects/Foo/foo-ios/fastlane/test_output/multi_scan/multi_scan_report.junit' --report html --output '/Users/fresh/Projects/Foo/foo-ios/fastlane/test_output/multi_scan/multi_scan_report.html' --report junit --output '/var/folders/sy/641lm9ks5179hj5_cd_94bn17rrh6f/T/junit_report20180725-57722-lsy0dr'

[Cyberbeni]

Yes, we had 5 nightly runs since dec 20 (out of 5 os versions per day) where success+failure didn't add up to the number of test cases that was started.

18 batch count, and out of 156 they only ran 147, 136, 129, 147, 147

[lyndsey-ferguson]

@Cyberbeni please include the information that I need to diagnose this.

I was able to run this successfully with our CI system using Xcode 11.3, fastane 2.106.2, and test_center 3.8.16, with a parallel_testrun_count of 2. It tests all of the tests it needs to and reports reflect that.

I will close this Issue on Friday if you cannot provide the information I need.

[Cyberbeni]

out.zip

[Cyberbeni]

Or you can make one where not every batch fails:
out 2.zip

Just put this in the test target:

@implementation NSObject (Crash)

+(void)load {
    if (arc4random_uniform(2) < 1) {
        exit(1);
    }
}

@end

[lyndsey-ferguson]

Thanks @Cyberbeni, at the very least, this will provide me clues to start figuring this out. It may take me a few days before I can start.

[Cyberbeni]

@lyndsey-ferguson is there any ETA on this? As I mentioned earlier, I think updating this one line to set the new only testing args based on the current ones minus the passed ones should probably solve the issue(if it happens rarely enough, as it is now):
https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/0c0a827bf6f56adc6f6cde2ab7c4d4381695eb75/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb#L234

[lyndsey-ferguson]

Hello @Cyberbeni , I'm sorry, some family issues came up and I forgot about this. I am not following what you're suggesting. Are you suggesting that the code look at what was originally sent in the only_testing option, and subtract the ones that passed from that list, and that becomes the next run's only_testing option?

I'm confused why the failing tests would not just equal that already?

BTW, I am open to [pull requests}(https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/master/docs/CONTRIBUTING.md) if you're up for it.

---

Now that I've slept, and I've reviewed your example, I think what you're saying is that if a test were to crash the test runner, that test is not included in the list of tests to re-run. It is not written to the list of failed tests in the junit file.

Is that what you're stating? If so, this should be an easy fix.

[Cyberbeni]

Hello @Cyberbeni , I'm sorry, some family issues came up and I forgot about this.

No worries, family should always come before work/hobby.

Now that I've slept, and I've reviewed your example, I think what you're saying is that if a test were to crash the test runner, that test is not included in the list of tests to re-run. It is not written to the list of failed tests in the junit file.
Is that what you're stating? If so, this should be an easy fix.

Yes, that is correct. We want to rerun tests that did not pass and those don't equal to failed ones in case the whole testrun fails to start.

[lyndsey-ferguson]

@Cyberbeni can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-180-crashed-tests-not-retried"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[Cyberbeni]

Ah, they are still not getting retried because they are seen as build errors (the log in the OP has the same error code):

[worker 2] ERROR [2020-03-01 14:17:30.24]: Exit status: 65
[worker 2] DEBUG [2020-03-01 14:17:30.24]: Batch #5 incrementing retry count to 1
[worker 2] DEBUG [2020-03-01 14:17:30.24]: Scan unable to test for batch #5

https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/blob/c4cf9483f5a3fe7278b76d270490e828e22d391e/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan_helper.rb#L123

[Cyberbeni]

Forgot to save the whole log the first time, reran it:

Archive.zip

[lyndsey-ferguson]

You set your try count to 2, can you set it higher? multi_scan will only retry that many times, and if you have a failing or crashing test, those are counted the same.

[Cyberbeni]

I've removed some stuff from handle_build_failure and now I'm seeing some Starting scan #2 messages:

https://github.com/Cyberbeni/fastlane-plugin-test_center/tree/no-error-on-build-failure

Archive 2.zip

[lyndsey-ferguson]

So, you're suggesting to not throw exceptions when I get these errors and to retry anyway?

[Cyberbeni]

Yep. I opened a PR.

I'm always confused on whether FastlaneCore::UI.error throws an exception or not. Cos it always seems like they do. But then they have code after them in a lot of places.

I remember missing tests for iOS9 simulator testruns with older Xcode versions where that other change would be also useful (But from my understanding they would only work for non invocation based tests properly) but I haven't experienced that in a while and iOS9 testing is getting removed with Xcode 11.4

[lyndsey-ferguson]

As you opened a new PR for the retry even with test runner failures, does this PR meet your needs?

[Cyberbeni]

Yes.

But one thing I still don't understand. The original behaviour should work the same way as now when "Test runner exited before starting test execution", there is even a test for it that it doesn't raise an exception. But then when I use the original behaviour, there is no Starting scan #2 anywhere in the log even though there are failures with "Test runner exited before starting test execution" text.

But I still want to retry on any "build failure" that might happen now or in the future. Like the one in our real project in the OP: MyAppUITests-Runner.app (17518) encountered an error (Failed to establish communication with the test runner. (Underlying error: The operation couldn’t be completed. (DTXProxyChannel error 1.)))

[Cyberbeni]

Now I get it, opened a new issue for it.

[lyndsey-ferguson]

I'll close this once I perform the release so I can link the two.

[lyndsey-ferguson]

Fixed in v3.10.0