Fastlane-plugin-test_center: Failing tests aren't re-run due to incorrect test path

[dionc]

I have multiple test targets in my workspace:
Unit tests: MyTests
UI tests: MyUITests

When a UI test fails in MyUITests it is not re-run by multi_scan because the test path passed for only_testing is incorrect.

Failing test: MyUITests/PasswordTests/testShowHidePassword

multi_scan generates this path for only_testing:
only_testing | ["MyTests/PasswordTests/testShowHidePassword"]

Rescan command generated by multi_scan:

|                                                  Summary for scan 2.79.0                                                   |
+------------------------+---------------------------------------------------------------------------------------------------+
| devices                | ["iPhone 8"]                                                                                      |
| workspace              | /Users/xcodeserver/gitlab-runner/builds/abd90b58/0/90Seconds/ios/MyProject.xcworkspace        |
| scheme                 | MyScheme                                                                                     |
| clean                  | false                                                                                             |
| skip_build             | false                                                                                             |
| output_directory       | ./fastlane/test_output                                                                            |
| output_types           | html,junit                                                                                        |
| output_files           | report-2.html,report-2.junit                                                                      |
| buildlog_path          | ~/Library/Logs/scan                                                                               |
| include_simulator_logs | false                                                                                             |
| derived_data_path      | /Users/xcodeserver/Library/Developer/Xcode/DerivedData/MyProject-esewnzwinqdkrrgvswuoavkavmky |
| open_report            | false                                                                                             |
| only_testing           | ["MyTests/PasswordTests/testShowHidePassword"]                                 |
| skip_slack             | false                                                                                             |
| slack_only_on_failure  | false                                                                                             |
| use_clang_report_name  | false                                                                                             |
| fail_build             | true                                                                                              |
| test_without_building  | true                                                                                              |
| xcode_path             | /Applications/Xcode.app                                                                           |
+------------------------+---------------------------------------------------------------------------------------------------+
[13:59:00]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/xcodeserver/gitlab-runner/builds/abd90b58/0/90Seconds/ios/MyProject.xcworkspace -scheme MyScheme -destination 'platform=iOS Simulator,id=B12A8269-FCC6-4486-8DF2-2656B8AE6BD8' -derivedDataPath '/Users/xcodeserver/Library/Developer/Xcode/DerivedData/MyProject-esewnzwinqdkrrgvswuoavkavmky' -only-testing:MyTests/PasswordTests/testShowHidePassword test-without-building | tee '/Users/xcodeserver/Library/Logs/scan/MyProject-MyScheme.log' | xcpretty  --report html --output '/Users/xcodeserver/gitlab-runner/builds/abd90b58/0/90Seconds/ios/fastlane/test_output/report-2.html' --report junit --output '/Users/xcodeserver/gitlab-runner/builds/abd90b58/0/90Seconds/ios/fastlane/test_output/report-2.junit' --report junit --output '/var/folders/my/2vn8q9n935g9lykd7c3pktvc0000gp/T/junit_report20180201-14937-v8l26b'
[13:59:00]: ▸ Loading...
[13:59:01]: ▸ 2018-02-01 13:59:01.532 xcodebuild[21329:8365757]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
[13:59:01]: ▸ /var/folders/my/2vn8q9n935g9lykd7c3pktvc0000gp/T/com.apple.dt.XCTest/IDETestRunSession-8021AF66-BB64-4232-AAE3-AA58CB182E7B/MyTests-6324BEBE-9EF2-493C-A0CE-1B376FBA666F/Session-MyTests-2018-02-01_135901-UOm3bI.log
[13:59:01]: ▸ 2018-02-01 13:59:01.532 xcodebuild[21329:8365755] [MT] IDETestOperationsObserverDebug: (1A570C26-4FFF-4BFE-8E18-E51CA49B5E30) Beginning test session MyTests-1A570C26-4FFF-4BFE-8E18-E51CA49B5E30 at 2018-02-01 13:59:01.533 with Xcode 9C40b on target <DVTiPhoneSimulator: 0x7fc1435f4940> {
[13:59:01]: ▸ SimDevice: iPhone 8 (B12A8269-FCC6-4486-8DF2-2656B8AE6BD8, iOS 11.2, Booted)
[13:59:01]: ▸ } (11.2 (15C107))
[13:59:05]: ▸ Selected tests
[13:59:05]: ▸ Test Suite MyTests.xctest started
[13:59:05]: ▸    Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
[13:59:05]: ▸ 
[13:59:07]: ▸ Test execute Succeeded
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 0 |
| Number of failures | 0 |
+--------------------+---+

Using fastlane 2.79.0, test_center 2.4.2.

[lyndsey-ferguson]

Ug, I was hoping it would be a while before someone ran into this problem. Even if you're not curious, I'm going to describe the problem here:

1. These tools rely on piping the xcodebuild output to xcpretty.
2. xcpretty creates a xml file with the root element being the "testable" (testsuites), or in your case MyTests.xctest. It does this for the first testable target it sees in the output.
3. It puts every test suite that is run into that file underneath the "MyTests.xctest" testsuites element.
4. My code reviews the xml file for the results, sees that the testsuite PasswordTests had a failing testcase testShowHidePassword. It reconstructs the path by looking at the elements to re-construct the testidentifer: MyTests/PasswordTests/testShowHidePassword.

In order to get around this, a few options come to mind for multi_scan. Use the new :batch_count option implicitly to batch tests into the number of testable targets and collect all related tests for each target into its own batch: running each batch separately through scan|xcpretty. That would create a correct xml file.

Or, when a test fails, review the identifier against the list of tests that I retrieved from each target to see where it _really_ lives. Then fix the identifiers.

Or, fix xcpretty to handle this (or learn how to configure it so that it does). I do not see any open Issues regarding this problem, so perhaps I am not using it correctly.

Finally, use a different method of getting the reports.

If you have any other suggestions, I'm open to hearing them. In the meantime, maybe create two Schemes that has only one target in each and call multi_scan twice? Ugly, I know, but at least it will keep you going.

[fishercraigj]

I ran into this yesterday as well. I actually like things uber organized
so what I did was created new schemes and enabled specific tests for those
schemes. Then I have multiple lanes that use multi_scan appropriately.
IE: 1 Lane for BVT tests, 1 lane for automation tests, 1 lane for unit
tests.

On Thu, Feb 1, 2018 at 6:20 AM, Lyndsey Ferguson [e-mail]
wrote:

Ug, I was hoping it would be a while before someone ran into this problem.
Even if you're not curious, I'm going to describe the problem here:

1. These tools rely on piping the xcodebuild output to xcpretty.
2. xcpretty creates a xml file with the root element being the
   "testable" (testsuites), or in your case MyTests.xctest. It does this
   for the first testable target it sees in the output.
3. It puts every test suite that is run into that file underneath the
   "MyTests.xctest" testsuites element.
4. My code reviews the xml file for the results, sees that the
   testsuite PasswordTests had a failing testcase testShowHidePassword.
   It reconstructs the path by looking at the elements to re-construct the
   testidentifer: MyTests/PasswordTests/testShowHidePassword.

In order to get around this, a few options come to mind for multi_scan.
Use the new :batch_count option implicitly to batch tests into the number
of testable targets and collect all related tests for each target into its
own batch: running each batch separately through scan|xcpretty. That
would create a correct xml file.

Or, when a test fails, review the identifier against the list of tests
that I retrieved from each target to see where it really lives. Then
fix the identifiers.

Or, fix xcpretty to handle this (or learn how to configure it so that it
does). I do not see any open Issues regarding this problem, so perhaps I am
not using it correctly.

Finally, use a different method of getting the reports.

If you have any other suggestions, I'm open to hearing them. In the
meantime, maybe create two Schemes that has only one target in each and
call multi_scan twice? Ugly, I know, but at least it will keep you going.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/19#issuecomment-362279358,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABWvaUoby1kDisaomjnmUtzBjaem62Xnks5tQcgcgaJpZM4R09sg
.

--

Craig Fisher

Software Developer in Test
C: [phone]
doctorondemand.com https://www.doctorondemand.com/

[dionc]

In the meantime, maybe create two Schemes that has only one target in each and call multi_scan twice?

This workaround solved the issue for me. Thanks for your quick response!

[lyndsey-ferguson]

You're welcome. And yes, this issue should remain open as it is truly a defect.

[lyndsey-ferguson]

Fixed in v3.0.1

[dionc]

Tested, works now.