Fastlane-plugin-test_center: Best way to test multiple devices and configurations with retry capability?

[ryanpato]

Question Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example(s) for the action(s) I am using
• [x] I searched for existing GitHub issues

Question Subject

Multi-scan

Question Description

I'm trying to achieve:

• Multiple devices
• Multiple configurations
• Retry capability

*Example *

If I run the following, assuming a test-plan with 2 configurations 'left-to-right', 'right-to-left' containing launch arguments of the configuration name.

  multi_scan(
    workspace: File.join(ROOT_PATH, XCODE_WORKSPACE),
    scheme: UI_TESTS_SCHEME,
    testplan: UI_TESTS_PLAN,
    devices: ["iPhone", "iPad" , "iPod"],
    fail_build: fail_build,
    output_types: 'junit,xcresult',
    output_files: "ui_tests.junit,ui_tests.xcresult",
    result_bundle: true,
    skip_slack: true,
    try_count: 2,
    parallel_testrun_count: 1,
    force_quit_simulator: true,
    reset_simulator: true
  )

import XCTest

class ExampleTestSuite: XCTestCase {
    override func setUpWithError() throws {
        continueAfterFailure = false
        XCUIApplication().launch()
    }

    // Pass on all configs
    func testExample01() throws {
        XCTAssertTrue(true)
    }

    // Fail on left-to-right configuration
    func testExample02() throws {
        XCTAssertTrue(ProcessInfo.processInfo.arguments.contains("left-to-right"))
    }

    // Fail on right-to-left configuration
    func testExample03() throws {
        XCTAssertTrue(ProcessInfo.processInfo.arguments.contains("right-to-left"))
    }
}

This tested with one device runs even successful tests twice.

xcresult

Current workaround

To get around this I am currently splitting up my lane to do two multi-scans 🤷 :

lane:

• run multi-scan with only_testing: ["left-to-right"]
• collate junit and xcresults to temp directory (to not be overwritten)
• run multi-scan with only_testing: ["right-to-left"].
• collate junit and xcresults to temp directory (to not be overwritten)
• collate all junit and xcresults to my output folder

Because multi-scan seems to overwrite any xcresult, the second config run overwrites the first, so after each run I am collating the

xcresult

I'm wondering if there is perhaps a better/recommended way to achieve this? 😅

[lyndsey-ferguson]

Hello @ryanpato, thank you for reporting this. You'll have to bear with me as I work to understand your test scenario. I don't have much experience with testplans except in the multi_scan usecase to direct xcodebuild to run a given list of tests.

From what I understand, your testplan should run each test in the left-to-right mode, and then right-to-left mode, for each of the devices?

And, are you saying that you're using each test plan to control the launch arguments? Right now, multi_scan is not paying attention to anything but the selected tests in a testplan 😱 . I have created Feature Request #255, but as no one has asked for this before, and I have quite a few requests and some bugs of higher priority, I haven't looked at that issue since.

So, let's say that is what you're looking for, and it seems that multi_scan is almost working for you, but for some reason, it is overwriting the xcresult file (which it shouldn't). Is it overwriting the xcresult file produced by the left-to-right mode when it runs the right-to-left mode?

As a potential workaround, have you tried moving the xcresult and the junit file using a testrun_completed_block callback that you can code to be smart enough to know when to move it? See the examples to see how to do that.

Also, would you be able to post a copy of the console for your run of fastlane with the --verbose flag turned on?

Finally, if you can attach a sample project with a sample Fastfile, that would help me understand (and potentially solve) this issue.

[ryanpato]

From what I understand, your testplan should run each test in the left-to-right mode, and then right-to-left mode, for each of the devices?

That's correct!

So, let's say that is what you're looking for, and it seems that multi_scan is almost working for you, but for some reason, it is overwriting the xcresult file (which it shouldn't). Is it overwriting the xcresult file produced by the left-to-right mode when it runs the right-to-left mode?

Yeah after the left-to-right run, results files left-to-right.junit, left-to-right.xcresult are created. Then it will run the tests again with the right-to-left configuration, but this step is overwriting the left-to-right.xcresult and replacing it with right-to-left. Ideally i'd like one job that runs all devices and configs, and returns a simple uitest.xcresult and uitests.junit for example.

As a potential workaround, have you tried moving the xcresult and the junit file using a testrun_completed_block callback that you can code to be smart enough to know when to move it? See the examples to see how to do that.

I haven't actually tried using the callback 🤔 yeah this could be a much tidier workaround! I'll check the examples out for this thank you. Currently I do something like the following. But requires name changing of files and such as well 😅 it all works though! Just not the tidiest solution haha

lane: 
   multi-scan(left-to-right)
   collateMoveResults()
   multi-scan(right-to-left) 
   collateMoveResults()
   collateResultsAndMoveBackToOutput()

Also, would you be able to post a copy of the console for your run of fastlane with the --verbose flag turned on?
Finally, if you can attach a sample project with a sample Fastfile, that would help me understand (and potentially solve) this issue.

Yeah apologies, I will grab a copy of the logs and update here asap! Grabbing a project may take a slight longer but i'll get that done too 😃

I do think this ties into the ticket you have for configuration support though and I do appreciate it's perhaps not a common feature! We use test plans pretty extensively using them to test things like languages and accessibilites and so on.

Cheers again for the quick reply! I'll try and get the logs and example project to you asap!

[lyndsey-ferguson]

Yes, if you can provide a project, that would be fantastic. Then I can test against that as I make changes.

[lyndsey-ferguson]

Cleaning up abandoned issues. Closed as no example project was provided.