Fastlane-plugin-test_center: Path to test in results showing wrong folder path.

[fishercraigj]

ENV: Using mutli_scan 2.4.4

For some reason when I log out the result variable for failed test cases, it showing my the correct failed test cases but it's showing me the wrong path in the begging of the path.

# Disable unit tests
disabled_tests = suppress_tests(
  xcodeproj: ENV['SOME_PROJECT'],
  tests: ['UnitTests/GlobalStringHelperTests', 'UnitTests/TourVCTests', 'UnitTests/UILabelHelperTests'],
  scheme: ENV['SOME_SCHEME']
  )

    retryCount = 3
    devicesToTest = ["iPhone 7 (10.3.1)"]
    multi_scan(
         devices: devicesToTest,
         scheme: ENV['SOME_SCHEME'],
         workspace: File.absolute_path(ENV['SOME_XCWORKSPACE_ABS_PATH']),
         try_count: retryCount)

result = tests_from_junit(junit: ENV['SCAN_JUNIT_RESULTS'])
failedCount = result[:failed].count
passingCount = result[:passing].count
testCount = failedCount + passingCount
UI.message("<-------- Failed tests: #{result[:failed]}")

What I'm being show when I log the failed results is: ["UnitTests/AccountTests/testC217UpdatePaymentMethod"]

Expected:
["UITests/AccountTests/testC217UpdatePaymentMethod"]

I have a UnitTests target and a UITests target. Both are wired up to the same screen. At the beginning of the lane I disable the UnitTests class and proceed from there. Wondering if the result is grabbing something from the disabled test internal variables or something odd like that.

[lyndsey-ferguson]

I am confident that this is due to the same problem experienced in Issue #19. Can you upgrade to v3.0.2 and try again?

[fishercraigj]

Sure, I'll give it a try tomorrow if that's alright. From the time I submitted this ticket 5 minutes ago, about 3 fires just got dropped on me.

[lyndsey-ferguson]

Also note that if you are not passing fail_build: false failing tests will cause this action to throw when it is complete.

[fishercraigj]

lol, thanks for the reminder on that one! It's the only reason I didn't upgrade quite yet. :)

[fishercraigj]

Did something change in the disabled_test test variable? You can see above how I was using it (when I was on 2.4.4).

Now I'm getting this:

INFO [2018-02-16 10:21:18.04]: ▸ Loading...
INFO [2018-02-16 10:21:18.49]: ▸ User defaults from command line:
INFO [2018-02-16 10:21:18.49]: ▸ IDEDerivedDataPathOverride = /Users/craigfisher/Library/Developer/Xcode/DerivedData/doctorondemand-bgeepoznqbrqwofnsepqgqbbnhco
INFO [2018-02-16 10:21:18.49]: ▸ IDETestRunOnlyIdentifiers = (
INFO [2018-02-16 10:21:18.49]: ▸ "UnitTests/UILabelHelperTests/testSetLabelLineHeight",
INFO [2018-02-16 10:21:18.49]: ▸ "UnitTests/UILabelHelperTests/testLabel.getter"
INFO [2018-02-16 10:21:18.49]: ▸ )
INFO [2018-02-16 10:21:18.49]: ▸ xcodebuild: error: Unknown build action ':'.
INFO [2018-02-16 10:21:18.50]: ▸ sh: line 1: -only-testing:UnitTests/UILabelHelperTests/testLabel.materializeForSet: No such file or directory
INFO [2018-02-16 10:21:18.50]: ▸ sh: line 2: -only-testing:UnitTests/UILabelHelperTests/testLabel.setter: No such file or directory
INFO [2018-02-16 10:21:18.50]: ▸ sh: line 3: -only-testing:UnitTests/UILabelHelperTests/testLabel: No such file or directory
INFO [2018-02-16 10:21:18.50]: ▸ sh: line 4: -only-testing:UnitTests/GlobalStringHelperTests/testChoppingStringPrefixCount: No such file or directory

I'll play with the paths to the test files but what I used to pass used to work. Just curious. :)

PS - Test run just fine locally through Xcode

UPDATE: Well, it's nothing to do with the disabled tests. I comment that out and I still get the same error.

[lyndsey-ferguson]

I think that my xctest_list tool is failing to parse out invalid tests symbols. Can you rename testLabel to _testLabel or something that does _not_ start with test...? xctest_list is looking at the symbols in the xctest bundle and matching functions that start with that keyword.

[lyndsey-ferguson]

Also, I'm not sure what you mean by the disabled_test variable; is there code that you're referring to?

[lyndsey-ferguson]

Can you also provide me the log that you see when the scan action starts? Specifically, the commandline call to the xcodebuild tool.

[fishercraigj]

Yeah, sorry, it was my variable I was referring to in the above code.

+----------------------+----------------------+
| Summary for multi_scan (test_center v3.0.2) |
+----------------------+----------------------+
| fail_build | false |
| try_count | 3 |
+----------------------+----------------------+

INFO [2018-02-16 14:50:19.64]: $ xcodebuild -list -workspace /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace
INFO [2018-02-16 14:50:20.84]: $ xcodebuild -showBuildSettings -workspace /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace -scheme doctorondemand
DEBUG [2018-02-16 14:50:23.45]: Fetching available simulator devices
DEBUG [2018-02-16 14:50:23.66]: Detected derived data path '/Users/craigfisher/Library/Developer/Xcode/DerivedData/doctorondemand-bgeepoznqbrqwofnsepqgqbbnhco'
INFO [2018-02-16 14:50:23.70]: $ xcodebuild -list -workspace /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace
INFO [2018-02-16 14:50:24.81]: $ xcodebuild -showBuildSettings -workspace /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace -scheme doctorondemand
DEBUG [2018-02-16 14:50:26.86]: Fetching available simulator devices

+------------------------+----------------------------------------------------------------------------------------------------+
| Summary for scan 2.81.0 |
+------------------------+----------------------------------------------------------------------------------------------------+
| devices | ["iPhone 7 (10.3.1)"] |
| fail_build | false |
| clean | true |
| scheme | doctorondemand |
| workspace | /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace |
| build_for_testing | true |
| derived_data_path | /Users/craigfisher/Library/Developer/Xcode/DerivedData/doctorondemand-bgeepoznqbrqwofnsepqgqbbnhco |
| skip_build | false |
| output_directory | ./fastlane/test_output |
| output_types | html,junit |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| xcode_path | /Applications/Xcode.app |
+------------------------+----------------------------------------------------------------------------------------------------+

INFO [2018-02-16 14:50:27.08]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/craigfisher/Developer/Repositories/dod-ios/doctorondemand.xcworkspace -scheme doctorondemand -destination 'platform=iOS Simulator,id=16B5F0C7-581B-4759-BDF6-CE9B9B3B9118' -derivedDataPath '/Users/craigfisher/Library/Developer/Xcode/DerivedData/doctorondemand-bgeepoznqbrqwofnsepqgqbbnhco' clean build-for-testing | tee '/Users/craigfisher/Library/Logs/scan/dodv3-doctorondemand.log' | xcpretty --report html --output '/Users/craigfisher/Developer/Repositories/dod-ios/fastlane/test_output/report.html' --report junit --output '/Users/craigfisher/Developer/Repositories/dod-ios/fastlane/test_output/report.junit' --report junit --output '/var/folders/rm/4h98gmw56q5_qzmn7dk1tkvr0000gn/T/junit_report20180216-71629-1m8vehr'

[lyndsey-ferguson]

Sorry, this is the portion of the log where the app is built. There should be another one later on that has test_without_building before the | tee portion. Can you look for that and attach it to this Issue?

I want to see how the test-identifiers are formed when we are sending them to xcodebuild via -only-testing as I suspect that I am sending them in an invalid way.

Can you also look at renaming the testLabel symbol in the UILabelHelperTests class and re-runnnig multi_scan?

[lyndsey-ferguson]

I just saw some of your code:

# Disable unit tests
disabled_tests = suppress_tests(
  xcodeproj: ENV['SOME_PROJECT'],
  tests: ['UnitTests/GlobalStringHelperTests', 'UnitTests/TourVCTests', 'UnitTests/UILabelHelperTests'],
  scheme: ENV['SOME_SCHEME']
  )

If you want to get a list of tests that are suppressed, you should call suppressed_tests not suppress_tests.

suppress_tests may be returning an Xcodeproj::XCScheme instance.

[lyndsey-ferguson]

_If_ this issue is due to Issue #28, v3.0.3 may have fixed this. Can you update to that and try that?

[fishercraigj]

Yeah, seem to be good now!

[lyndsey-ferguson]

Does every work now and we can close this issue?