Fastlane-plugin-test_center: multi_scan does not work with :skip_build like scan does, and fails

[lyndsey-ferguson]

Thank you for the bug report. I removed some api_tokens and build_secrets from the description that you provided me.

[lyndsey-ferguson]

Can you remove the option to skip_build? I think that your use of clear_derived_data is wiping out the test runner that is needed to run the tests.

The xctestrun file is created when building the app with tests and is stored in clear_derived_data. I could optionally error out if the derived data directory is empty...

[Sandhu412-zz]

@lyndsey-ferguson -- I have skip_build as true because if not, it builds twice, once for "debug" and then once for the test build. Is that not true anymore? I run 'cleear_derived_data' for a clean build everytime. Also, the 'cleear_derived_data' runs before it evens starts building right? So why would it delete the xctestrun if it hasnt even built?

[lyndsey-ferguson]

With multi_scan, I am using the build for testing and test without building flags. So, if you remove the skip_build option, the action will create a build for testing, then run tests up to try_count times (without building again).

I am not very familiar with the skip_build option (which I will soon change). It could be "building" for test in a different way than I am used to.

If you can confirm that everything works with the removal of the skip_build parameter, I can look into how scan does what it does.

[Sandhu412-zz]

I tried both the suggestions: removing skip_build and removing clear_derived_data. I am still getting the same error:
bundler: failed to load command: fastlane (/usr/local/bin/fastlane)
FastlaneCore::Interface::FastlaneError: [!] Error: cannot find xctestrun file ''

But if I use scan before multi_scan it builds and runs the tests fine. Below is the updated lane I am using:

desc "Runs all the TK UI Tests"
lane :toolkit_ui_tests do
ENV['FASTLANE_XCODE_LIST_TIMEOUT'] = '120'
clear_derived_data
sh('mkdir -p ../reports/uitest-coverage/')
sh('mkdir -p ../reports/uitest-xcov-coverage/')
cocoapods(repo_update: false, silent: true) # CP repo update moved to a Jenkins task that runs once a day
scan(
scheme: "ToolkitUITests",
device: "iPhone 8",
build_for_testing: true,
fail_build: true,
code_coverage: true,
skip_build: true, # needed otherwise scan builds Toolkit twice
clean: true
)
multi_scan(
workspace: File.absolute_path('../Toolkit.xcworkspace'),
test_without_building: true,
device: "iPhone 8",
scheme: 'ToolkitUITests',
output_directory: "reports/ui-tests",
include_simulator_logs: false,
skip_build: true,
try_count: 3,
fail_build: true,
code_coverage: true,
)

[lyndsey-ferguson]

Are you telling multi_scan to test_without_building when it fails?

[Sandhu412-zz]

@lyndsey-ferguson -- sorry for the confusion. I think I have fixed the lane now following your suggestion. I didn't know you were setting the flags build for testing and test without building. Now below is the lane I am using and I am not using scan anymore and I removed the skip_build option as well. (I guess in the documentation you can remove 'skip_build`)

multi_scan(
  workspace: File.absolute_path('../Toolkit.xcworkspace'),
  device: "iPhone 8",
  scheme: 'ToolkitUITests',
  output_directory: "reports/ui-tests",
  include_simulator_logs: false,
  try_count: 3,
  batch_count: 3,
  fail_build: true,
  code_coverage: true,
)

Only question I have for you is: If some of the tests fail and when it tries to rerun the failed tests, will it build again or just rerun the failed tests without building (my preference is to just rerun without building)?

[lyndsey-ferguson]

Oh, it will just re-run the tests without building.

[lyndsey-ferguson]

I'm going to keep this issue, but make it revolve around the difference between how multi_scan treats skip_build vs scan.

[Sandhu412-zz]

Sure, that makes sense. scan builds twice if that flag is not set to true. Thanks so much for putting together this plugin, it saves a lot of false failures for people like me.

[lyndsey-ferguson]

You're very welcome!

[Sandhu412-zz]

@lyndsey-ferguson -- Had another question, when we batch the tests, it generates multiple reports, is it possible to aggregate all the reports into just 1 report?

[lyndsey-ferguson]

It should do that when it finishes _unless_ there are multiple test targets. Are there multiple test targets? If so, it doesn't make sense to collate them as the format doesn't merge properly.

[Sandhu412-zz]

my UI lane only has one test target "ToolkitUItests"

[lyndsey-ferguson]

What reports does it generate? When it completes, it should collate the reports into one report. You're talking about the junit reports? Or the HTML reports? If HTML, multi_scan does _not_ collate those. Nor does it collate plist reports in a results_bundle. Both are possible but not done.

[Sandhu412-zz]

HTML has 3 parts, report.html, report-2.html and report-3.html (for batch count of 3). Junit only has 1 report and as per your comment above, it seems like junit is consolidated?

[lyndsey-ferguson]

Right, only junits are consolidated. If you'd like HTML reports consolidated, can you create a new Issue?

[Sandhu412-zz]

Absolutely. I will create a new issue.

[LucianoTurrini]

Hello, excellent plugin, but I'm having the same problem, waiting for fix, thanks

[lyndsey-ferguson]

Some personal notes. From what I've discovered, the :skip_build option will simply _omit_ the build parameter to xcodebuild. This means that if the project's build products already exist, xcodebuild will not build the product but just proceed to testing. If the build products do not exist, then it will build and then test.

[lyndsey-ferguson]

Can I properly determine if the correct build products exist so that I can simply grab the tests for only_testing?

[lyndsey-ferguson]

Can I simply pass it down to scan so that we too omit the build command, and then all the tests run the first pass. I could imagine it would also call my refactored MultiScanManager::Runner's run_first_run_of_invocation_based_tests (renaming the method) and then let the normal events unfold? That seems reasonable.

[lyndsey-ferguson]

Slated to be fixed by issue-41 support skip_build

[lyndsey-ferguson]

Fixed in v3.8.0