Fastlane-plugin-test_center: multi_scan does not work with scan's skip_testing option

[aslattum]

Feature Request

Implement scan skip_testing option for multi_scan

Motivation Behind Feature

Implementing the skip_testing option that is a part of fastlanes scan action will allow developers to choose between using a whitelist or blacklist while testing their code. Right now, multi_scan only supports whitelist testing through the use of only_testing whereas it would be beneficial to provide support for blacklisting (i.e. skip_testing). The benefits of this are that developers can have large test sets within their Xcode scheme and rather than having to be explicit with which ones to tests, they can just list the ones they don't. This is especially helpful for developers who bundle their unit tests and ui tests into one scheme because there could be many unit test targets with one ui test target, but the dev just wants to run the unit tests and not the ui tests. Blacklisting here would be especially beneficial.

Feature Description

Add the skip_testing option from fastlanes scan action to multi_scan
Link to skip_testing option in scan: https://github.com/fastlane/fastlane/blob/master/scan/lib/scan/options.rb#L112
What it would look like in use:

multi_scan(
      workspace: "Adam.xcworkspace",
      scheme: "Adam",
      try_count: 3,
      code_coverage: true,
      skip_testing: ["AdamUITests"],
)

Alternatives or Workarounds

The workaround I'm using now is a whitelist for testing my scheme; however, this will not scale because the amount of test suites in the Xcode scheme is growing and will eventually require developers to add their test target to the whitelist upon each new pull request adding a suite of tests.

Thank you!

[lyndsey-ferguson]

@aslattum multi_scan also incorporates _all_ of the scan options, so you _should_ be able to use skip_testing. I know I have code that observes it, but it has been a while since I've used it.

Have you tried this option, or do you think I need to make it more obvious to people who come to use multi_scan that _all_ of the scan options are available?

I have created a scan_options method that brings in the options.

[aslattum]

@lyndsey-ferguson I have tried to use skip_testing just as it is used in scan and receive an error for it being unsupported. I've tried it both as a string (i.e. skip_testing: "AdamTests) and as an array of strings (i.e. skip_testing: ["AdamTests"]).

I was searching in that file you linked in the response and did not find any sign of skip_testing. Not sure if that's because you don't have to explicitly include it as an option or not.

I'll post the error message from fastlane here once I try it one more time!

[aslattum]

@lyndsey-ferguson so if I use the skip_testing parameter, it ends up running completely incorrect tests. Looks like it adds a bunch of individual tests to the only_testing variable that have nothing to do with what I actually would like to test

[lyndsey-ferguson]

Here is the definition of scan_options, which in turn uses the ScanAction.available_options to extend multi_scan's array of options.

Note: I do not use scans :output_types as I've changed it to also include json.

---

With regards to it not working, I apologize for that. As I wrote, I haven't used it for a while and I may have broken it (or simply not adequately tested it).

I won't be able to fix this issue at this moment.The time that I do have for this plugin, I use to focus on finishing Issue-10 parallel multi scan.

However, there may be a work-around for you _if_ you already have a built-for-testing product of your Adam.xcworkspace. You can use the tests_from_xctestrun action to gather the tests, and then remove the tests you want to skip and pass that to the only_testing parameter.

A rough, untested example:

  all_tests = tests_from_xctestrun(xctestrun: 'path/to/build/products/AdamSimulatorVersion.xctestrun')
  only_testing = all_tests.reject { |t| t.include?("AdamUITests") }
  multi_scan(
      workspace: "Adam.xcworkspace",
      scheme: "Adam",
      try_count: 3,
      code_coverage: true,
      only_testing: only_testing,
  )

[aslattum]

@lyndsey-ferguson not a problem just wanted to bring it your attention! I do in fact have a build_for_testing product for the workspace so I'm going to try using the above example 👍

[aslattum]

@lyndsey-ferguson just as a follow up, the only_testing works since right now I have two sets of test suites; however, in the future I imagine there being many (over 50) so I think it would be helpful to provide a blacklisting option through skip_testing explicitly!

[SameerYoussef]

Thank you @lyndsey-ferguson for this awesome plugin.

I assume this is the same issue afflicting our iOS project currently...

We depend on the pod 'GoogleMaps'

When I run multi_scan in parallel in verbose mode I see:

DEBUG: Found the following tests: GMSPlacesClient/testingPhotoFetchTimeout GMSPlacesClient/testingPhotosURLConfiguration GMSPlacesServerClient/testBlock GMSx_GTLBatchQuery/testBlock GMSx_GTLQuery/testBlock GMSx_GTLService/testBlock GMSx_GTMSessionFetcher/testBlock GMSx_GTMSessionFetcherService/testBlock
which are tests for Google Maps - my tests follow underneath those. Despite me having specified my scheme it still insists on running these unwanted tests from the scheme of that pod.

[lyndsey-ferguson]

Hi @technomark, I use a fairly crude method of getting the tests from your generated xctest file. I ask what methods are there, and those that start with test are considered part of the tests you want to run. Are you including the GoogleMaps as part of your testtarget? Is it necessary to link your tests against the GoogleMaps SDK?

Typically, your test runner should not be linking against 3rd party SDKs and should just be in charge of _running_ your app.

Failing that, you can add the Google tests to the skip testing list (you have to add the entire test identifier). For example, ['GMSPlacesClient/testingPhotoFetchTimeout', 'GMSPlacesClient/testingPhotosURLConfiguration', 'GMSPlacesServerClient/testBlock', 'GMSx_GTLBatchQuery/testBlock', 'GMSx_GTLQuery/testBlock', 'GMSx_GTLService/testBlock', 'GMSx_GTMSessionFetcher/testBlock' 'GMSx_GTMSessionFetcherService/testBlock']

[SameerYoussef]

Thanks - it appears GoogleMaps.bundle is copied into the test target output (not sure why) but in any case after having deleted it I still found the tests were being run. skip_testing the tests didn't help (it ran xcodebuild with both -only-testing:'GMSPlacesClient/testingPhotoFetchTimeout' -skip-testing:'GMSPlacesClient/testingPhotoFetchTimeout' and still took time from the simulator) so I'll just modify the execution to use only_testing then pass in the tests I want.

[lyndsey-ferguson]

@technomark ok, the fact that the GoogleMaps.bundle is copied into the test target "output" is a clue, but I do not think it is the answer.

I think that the GoogleMaps tests are being copied into your xctest binary file (where the test methods are). I would check to see in a link build phase of your test target if it is linking GoogleMaps. (see this image for what the build phase looks like).

OR, it is being linked in via the ld BUILD_SETTING (see image of where that is).

_remember, the tests are being found in the xctest binary itself, not outside of that binary file in other bundles_

[SameerYoussef]

Just to answer my own question - turns out the test target was inheriting all the cocoapod dependencies of the main app - which was unnecessary but also meant that the GoogleMaps bundle was being included - hence the extra tests. Thanks @lyndsey-ferguson for your assistance.

[lyndsey-ferguson]

Makes perfect sense!

[lyndsey-ferguson]

@technomark I've finally been able to look into this. Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-119-clean-up-skip-testing-option"

If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.

[lyndsey-ferguson]

@aslattum I'd also appreciate your feedback, (see my above comment).

[lyndsey-ferguson]

Fixed in v3.10.2