Fastlane-plugin-test_center: Multi scan runs all the disabled tests in the scheme

Created on 9 Jun 2019 · 25Comments · Source: lyndsey-ferguson/fastlane-plugin-test_center

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

I have a bunch of UI tests disabled in my test scheme. Scan runs only the enabled tests in the scheme. But, when I used multi_scan, it passed all the tests in the scheme including the disabled ones to the "test_only" parameter.

Would be great if mult_scan would only run the tests that are enabled in the scheme

NOTE: I assumed that multi_scan runs the first iteration of tests and reruns if tests fail. Or, should I first run my tests using scan and pass the results to multi_scan to rerun only the tests that failed in the testrun by scan ?

☑️ tests_from_xctestrun 🐞bug 🐣good first issue 🔨in-progress 🙏 help wanted

Source

sneha-je

Most helpful comment

Fixed in v3.8.11

lyndsey-ferguson on 23 Oct 2019

🎉2

All 25 comments

multi_scan is supposed to skip the tests that have been disabled in the Scheme. Can you provide a log so I can take a look at what is going on?

Remember to please remove any sensitive data before posting.

lyndsey-ferguson on 9 Jun 2019

I have the same problem here, all UI Tests from the project runs even if the scheme disabled some of them.

MarianoRappazzo on 12 Jun 2019

@MarianoRappazzo since @sneha-je has not posted a log, would you be able to?

Also would you be able to try the previous version of the plugin? You can do so by modifying your Pluginfile per my instructions below, run bundle install, and then run your fastlane again?

Pluginfile:

gem 'fastlane-plugin-test_center', '3.6.3'

lyndsey-ferguson on 13 Jun 2019

Also, can you re-run your fastlane with the --verbose option and include _that_ log

lyndsey-ferguson on 13 Jun 2019

Can you provide the logs that I requested? I'll go ahead and close this Issue on Friday (EST) if there are no logs to provide.

lyndsey-ferguson on 18 Jun 2019

Remember, I will close this Issue tomorrow unless you can provide me with logs. When providing me with logs, please use the --verbose flag.

BTW, I have confirmed that this works with the current feature that I am working on for parallel test runs. If you would like to use that beta, please see this pull request

lyndsey-ferguson on 20 Jun 2019

Sorry for the late reply. When I run the example tests in this repo by disabling one of the UI tests, the disabled one didn't run.

I'm using "workspace" in my fastfile scan instead of "project" as used in this project's fastfile. I initially thought that might be the issue but that isn't the case. I'm checking what is so different in my work project's setup that it's running all the disabled ones as well.

Here is the log I've removed all the compiling logs and test cases names
`bundle exec fastlane sweep
[✔] 🚀
+-----------------------------+---------+----------------------------------------------------------------------------------+
| Used plugins |
+-----------------------------+---------+----------------------------------------------------------------------------------+
| Plugin | Version | Action |
+-----------------------------+---------+----------------------------------------------------------------------------------+
| fastlane-plugin-versioning | 0.3.4 | get_version_number_from_git_branch get_version_number_from_plist |
| | | get_build_number_from_plist ci_build_number increment_version_number_in_plist |
| | | get_app_store_version_number get_info_plist_path increment_build_number_in_plist |
| fastlane-plugin-test_center | 3.7.0 | suppressed_tests suppress_tests suppress_tests_from_junit tests_from_xctestrun |
| | | collate_test_result_bundles tests_from_junit multi_scan collate_html_reports |
| | | collate_junit_reports collate_json_reports |
+-----------------------------+---------+----------------------------------------------------------------------------------+

+----------------------+----------------------+
| Summary for multi_scan (test_center v3.7.0) |
+----------------------+----------------------+
| try_count | 3 |
| fail_build | false |
+----------------------+----------------------+

+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| clean | true |
+----------------------+---------------------+

[

Compiling Logs

+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

sneha-je on 21 Jun 2019

I'm sorry, but this log doesn't provide information on what tests were run. I can understand if you do not want to provide highly sensitive information: each developer faces certain restrictions that cannot be explained.

Given that, can you please provide a sample Xcode project and Fastfile that calls multi_scan with that project so that I can debug this? As I wrote before, everything is currently working fine for me.

I can leave this Issue open until July 5th. I'll close the Issue on the 6th. You can always re-open the issue once you can provide the sample Xcode project and Fastfile.

lyndsey-ferguson on 23 Jun 2019

I'm closing the issue as I cannot reproduce it, and I don't have enough information to debug it.

lyndsey-ferguson on 6 Jul 2019

We have the same issue.
Lots of tests are disabled in a specific suite, but multi_scan still tries to run everything.

in 3.6.3 it has the same output as in the latest version

Update
Ok so i think i got what happens here.
In our project we have 5 different UITests Schemes.
When i put the 1st scheme in your fastlane plugin it also includes all the rest of the schemes.
If i put the last scheme (in displayed order) it runs the correct batch of tests. I will provide an example if i find out that that's case.

Also skip_testing is not working

kwstasna on 9 Sep 2019

@kwstasna yes, if you can provide an example project along with the Fastfile used to call multi_scan on it, I can re-open this Issue and begin investigating.

lyndsey-ferguson on 9 Sep 2019

@lyndsey-ferguson sorry but whatever i tested in a clean project (with pods etc) seems to work fine. Although in our own project it has the following reaction.

When i put some scheme that is not the last one on the list of schemes (lets say last_scheme-1), as xcode shows it in its ui.
It takes the rest of the schemes (for example the last scheme) actual tests that are in the file. And _not_ the tests that are checked in "Edit Scheme/Tests".

I don't know if this can help you a bit.

P.S. checked also with the latest fastlane version (2.130.0) and (2.120.0)

original scan from fastlane, runs the correct tests of the same scheme that we try with multi_scan

kwstasna on 12 Sep 2019

So after some debugging this is the "possible fix" that is actually solving the issue for our tests.

We have multiple Schemes under our AppNameUITests Target.
All Schemes have their own Selected/Unselected tests.

What your implementation does is to go to AppNameUITests and get All of the tests.
What i do is to open CertainSchemeUITests_iphonesimulator12.2-x86_64.xctestrun
and find the proper tests under the OnlyTestIdentifiers array in the plist.
This array contains ONLY the Selected tests in the Scheme that i setted in my FastFile!

Although your code XCTestList.tests(xctest_path) contains ALL the tests in the Target!

https://github.com/trading-point/fastlane-plugin-test_center/blob/investigateWeirdSchemeBug/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb#L21

What is your opinion about this?

kwstasna on 12 Sep 2019

Hello @kwstasna. It seems that you have performed a deep analysis of the issue; however, I apologize, but I do not understand the results/theory very well. Let me try to summarize what I _think_ you are writing:

You have multiple schemes that include your AppNameUITests target. Schemes include targets, not the other way around.
(this part is unclear to me) When you pass the first scheme to multi_scan (let's call it Scheme1), it not only runs the tests in Scheme1, but also the tests in Scheme2, Scheme3... and SchemeN.
If you pass the last scheme SchemeN to multi_scan it tests only the tests that are in SchemeN.
Your last finding has me confused and I cannot summarize any understanding. Let me elaborate on how multi_scan deals with this.

There are two ways to start.

multi_scan is told to build and then test the project.
multi_scan is told to test the xctestrun file (as you mentioned above).

Both the build and the test will ultimately use a xctestrun file. The only difference with the build is that it clears out xctestrun files, runs the build with the given Scheme that you provided, and then continues on to the test phase.

The test phase will read the xctestrun file that it found. In that file we have 1) the binary of the test app that has the tests and 2) the list of tests that should be skipped (from #1). Unless you provide only_testing to multi_scan it will get use all the tests from the binary _minus_ the skipped tests.

If you could provide a sample project and sample Fastfile, that would help me a lot.

lyndsey-ferguson on 12 Sep 2019

I have a repro for this issue, my scenario is i have 1 test scheme say 'scheme1' which was created in some previous version of xcode, and recently i created a new scheme 'scheme2' in xcode 11.0.

When i run scheme1 with multi_scan, it runs all the tests regardless whatever i have defined in scheme1.
When i run scheme2, it runs as expected.

So, I compared .xcsheme files of both schemes, and found that:

scheme1 has a xml tag <SelectedTests> and all enabled tests are listed there.
scheme2 has a xml tag <SkippedTests> and all disabled tests are listed there.

So, my guess is if "it will get use all the tests from the binary minus the skipped tests", then maybe for scheme1 as there are no SkippedTests, it is running all tests.

rahulsharma2905 on 4 Oct 2019

This is helpful. I've re-opened this Issue.

lyndsey-ferguson on 4 Oct 2019

I may not be able to address this right away, but if you want to fork this repo and try fixing it, you can look in the tests_from_xctestrun action to make sure that the test_identifiers are the only ones that are passed in (rather than using SkippedTests to filter out tests).

lyndsey-ferguson on 4 Oct 2019

tests_from_xctestrun uses the xctestrun file generated from a build to determine which tests to skip. I can only assume that that file for an older version of Xcode, an older version of xctestrun file has some sort of identifier that makes explicit which tests _are to run_. Can you provide a sample of the older generated xctestrun?

lyndsey-ferguson on 6 Oct 2019

i can't provide a sample xctestrun, but i did some changes in tests_from_xctestrun file and my observations are:

xctestrun of schemes generated by XCode 11 have key SkipTestIdentifiers, which you are currently using.
xctestrun of schemes generated by old XCode does not have SkipTestIdentifiers, but have another key OnlyTestIdentifiers.

My changes seems to be working for me for both new and old schemes.
Take a look at it:
https://github.com/rahulsharma2905/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb#L28

rahulsharma2905 on 7 Oct 2019

@rahulsharma2905 thank you for the confirmation! If you're open to creating a PR with tests, I'd welcome it. I suggest that you check for OnlyTestIdentifiers before even pulling out the tests from the XCTest call. Here: https://github.com/rahulsharma2905/fastlane-plugin-test_center/blob/master/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb#L19

lyndsey-ferguson on 9 Oct 2019

Hi guys. Me and my team are having the same issue (multi_scan runs the disabled tests with the newer versions of the test_center plugin).

I'm glad @rahulsharma2905 shared those insights, but I think you only have half the picture.

From what I tested, the difference between having the SelectedTests tag in the xcscheme (backed by OnlyTestIdentifiers in the xctestrun) or SkippedTests tag in the xcscheme (backed by SkipTestIdentifiers in the xctestrun) sits in the useTestSelectionWhitelist option from each scheme (visually identifiable as Scheme Tests tab, Options... Automatically include new tests). See screenshot

Screenshot 2019-10-22 at 13 57 47

Setting this on / off back and forth, I saw the xcscheme change between the 2 tags mentioned above and consistent with the xctestrun tags.

So in my opinion, this is not directly influenced by the Xcode version (but it may be indirectly if Xcode 11 has the option default changed).

PS: glad to see #155 looking so close to being merged. 👍

I have a repro for this issue, my scenario is i have 1 test scheme say 'scheme1' which was created in some previous version of xcode, and recently i created a new scheme 'scheme2' in xcode 11.0.

When i run scheme1 with multi_scan, it runs all the tests regardless whatever i have defined in scheme1.

When i run scheme2, it runs as expected.

So, I compared .xcsheme files of both schemes, and found that:

scheme1 has a xml tag and all enabled tests are listed there.

scheme2 has a xml tag and all disabled tests are listed there.

So, my guess is if "it will get use all the tests from the binary minus the skipped tests", then maybe for scheme1 as there are no SkippedTests, it is running all tests.

i can't provide a sample xctestrun, but i did some changes in tests_from_xctestrun file and my observations are:
xctestrun of schemes generated by XCode 11 have key SkipTestIdentifiers, which you are currently using.
xctestrun of schemes generated by old XCode does not have SkipTestIdentifiers, but have another key OnlyTestIdentifiers.