Fastlane-plugin-test_center: Fastlane job succeeds even though multi_scan fails

[dionc]

I am using multi_scan with GitLab CI to run Xcode UI Tests. When a test genuinely fails (3 times with try_count: 3), the GitLab CI job still passes. If I use scan instead of multi_scan the job fails as it should.

```
Failing tests:
LoginUITests.testKeyboardShouldHideAfterUserTapsReturn()
* TEST EXECUTE FAILED *
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 1 |
| Number of failures | 1 |
+--------------------+---+

+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | multi_scan | 318 |
+------+----------------------------+-------------+

Job succeeded

[lyndsey-ferguson]

Hello @dionc, can you attach the junit file? Also, can you help debug by adding this block of code after your call to multi_scan?

test_result = tests_from_junit(junit: 'path/to/your/junit_report_file') # ⬅️ you have to replace that path with your real path
if test_result[:failed].size > 0
  raise FastlaneCore::Interface::FastlaneTestFailure, 'tests failed'
end

Then run your code in GitLab CI and let me know if it fails as expected?

I just tested it locally and I get a junit file with a failing case which is what Jenkins (what I use) has been configured for.

Perhaps the GitLab CI job is looking for a non-zero code and that is why scan works: it raises a FastlaneCore::Interface::FastlaneTestFailure which multi_scan catches to test for a retry, but it does not re-raise it as that causes the Build Step in Jenkins to fail and that prevents the display of test results.

[fishercraigj]

@lyndsey-ferguson - I feel like a "fail_build_after_mutli_scan" is kind of what @dionc might have been expecting. I say this because when you are looking at the actions for multi_scan, the fail_build variable from scan is present and set to true by default.

@dionc - I personally would always have a variable like that set to false because I don't want uber flaky UIAutomation to stop my build process. If you want your test suite to finish running all the tests in your suite while not failing the actual CI build because a keyboard fails to dismiss (in your case), you can do something similar to like what I do to get a summary. Obviously this could be easily expanded on.

Here I'm just logging it out to the terminal for demonstration purposes but I personally have private gems that take the results and send them out to Zulip (similar to Slack) and alert specific users in QA that the test have failed and then I also fire it out in an email to a wider audience.

result = tests_from_junit(junit: './fastlane/test_output/report.junit')
    failedCount = result[:failed].count
    passingCount = result[:passing].count
    testCount = failedCount + passingCount
    UI.message("<-------- UIAutomation testing has finished.  #{passingCount} of #{testCount} tests passed. \n\n Any failures were isolated and retried up to #{retryCount} times in attempt to pass.  List of failed tests: #{result[:failed]}")

[lyndsey-ferguson]

@fishercraigj which fail_build variable does scan have? I only see :slack_only_on_failure. It would be great if scan already had something for this that I can use in my code too.

[fishercraigj]

fastlane action multi_scan. Same thing shows up when running fastlane action scan. I'm on Fastlane 2.80.0.

[lyndsey-ferguson]

Oooh, that's great. It's in a different scan options file, that's why I didn't see it.

[fishercraigj]

Sweet! If you add that feature, my personal vote would be to have a "fail_build_after_mutli_scan" like option set to false by default. But it's your framework. I'll just adjust if I see a build failure in the logs because of it. :)

[lyndsey-ferguson]

Yeah, I'd like to not change it from the opposite of what scan has. I will update the version to a 3.0.0 to indicate breaking changes forthcoming.

[fishercraigj]

No worries, I'll leave myself a comment in my lane to adjust if/when I upgrade.

[lyndsey-ferguson]

Oh, you'll know 🔥

[dionc]

Hi @lyndsey-ferguson, by adding the debug block above it fails as expected.

Here's the contents of the junit report:
<?xml version='1.0' encoding='UTF-8'?> <testsuites name='NinetySecondsUITests.xctest' tests='31' failures='1'> <testsuite name='NinetySecondsUITests.BrandRegistrationUITests' tests='15' failures='0'> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testAboutYourBrandBlankBackTitle' time='35.471'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testAccountInfoBlankBackTitle' time='19.904'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testAccountInfoContinueDisabledAtFirstLaunch' time='20.164'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testAccountInfoTitle' time='19.039'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testBrandInfoCreateAccountDisabledAtFirstLaunch' time='33.076'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testCursorGoesToPasswordAfterUserTapsReturn' time='11.732'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testKeyboardShouldHideAfterUserTapsReturn' time='12.737'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testOpenAccountInfo' time='19.183'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testOpenBrandInfo' time='34.650'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testOpenSignUpSuccess' time='43.926'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testSelectAvatar' time='43.056'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testSignUpBlankBackTitle' time='10.067'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testSignUpContinueDisabledAtFirstLaunch' time='9.572'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testSignUpTitle' time='9.674'/> <testcase classname='NinetySecondsUITests.BrandRegistrationUITests' name='testSignUpViewHidePassword' time='14.895'/> </testsuite> <testsuite name='NinetySecondsUITests.ForgotPasswordUITests' tests='6' failures='1'> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testBlankLoginBackButton' time='10.474'/> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testDisabledResetButtonAtInitialLaunch' time='11.738'/> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testOpenResetPassword' time='25.607'/> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testOpenRestorePassword'> <failure message='XCTAssertTrue failed - '> ForgotPasswordUITests.swift:41 </failure> </testcase> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testShowHidePassword' time='23.596'/> <testcase classname='NinetySecondsUITests.ForgotPasswordUITests' name='testSuccessfulRestore' time='18.431'/> </testsuite> <testsuite name='NinetySecondsUITests.LoginUITests' tests='6' failures='0'> <testcase classname='NinetySecondsUITests.LoginUITests' name='testBlankLoginBackButton' time='8.500'/> <testcase classname='NinetySecondsUITests.LoginUITests' name='testCursorGoesToPasswordAfterUserTapsReturn' time='10.765'/> <testcase classname='NinetySecondsUITests.LoginUITests' name='testDisabledLoginButtonAtInitialLaunch' time='9.558'/> <testcase classname='NinetySecondsUITests.LoginUITests' name='testKeyboardShouldHideAfterUserTapsReturn' time='11.414'/> <testcase classname='NinetySecondsUITests.LoginUITests' name='testLoginButtonEnabledAfterEmailAndPasswordPopulation' time='17.258'/> <testcase classname='NinetySecondsUITests.LoginUITests' name='testOpenLogin' time='9.634'/> </testsuite> <testsuite name='NinetySecondsUITests.WelcomePageUITests' tests='4' failures='0'> <testcase classname='NinetySecondsUITests.WelcomePageUITests' name='testEnvironmentSheet' time='9.596'/> <testcase classname='NinetySecondsUITests.WelcomePageUITests' name='testOpenLogin' time='9.708'/> <testcase classname='NinetySecondsUITests.WelcomePageUITests' name='testOpenSignUp' time='9.301'/> <testcase classname='NinetySecondsUITests.WelcomePageUITests' name='testWelcomePages' time='18.037'/> </testsuite> </testsuites>

[lyndsey-ferguson]

Thanks, this fix should be in soon-ish.

[lyndsey-ferguson]

Fixed in v3.0.1

[dionc]

Tested, works now.