Fastlane-plugin-test_center: multi_scan fails when trying to collate HTML reports that contain < > as part of the message
New Issue Checklist
- [x] Updated
fastlane-plugin-test_centerto the latest version - [x] I read the README.md
- [x] I reviewed the example Fastfile
Issue Description
multi_scan gives an error, "malformed XML: missing tag start", when trying to collate reports with test failure text containing angle brackets. At least, that's my best guess for what's causing this. I'm guessing it's messing with XML tags in the junit file it's trying to create. For example, the text of this type of failure on our project seems to be setting off this error:
testCopyWords, Failed to scroll to visible (by AX action) Button, 0x604000387c50, traits: 8858370049, label: 'Lessons', error: Error -25204 performing AXAction 2003 on element <XCAccessibilityElement: 0x7f927d70e7f0> pid: 83000, elementOrHash.elementID: 140550659773824.326
If this error doesn't happen on a particular run (it's intermittent) or if I specifically run tests that this error never seems to happen on, then multi_scan finishes just fine.
Environment
✅ fastlane environment ✅
Stack
| Key | Value |
| --------------------------- | ------------------------------------------- |
| OS | 10.13.4 |
| Ruby | 2.5.1 |
| Bundler? | true |
| Git | git version 2.15.1 (Apple Git-101) |
| Installation Source | /usr/local/opt/gems/bin/fastlane |
| Host | Mac OS X 10.13.4 (17E202) |
| Ruby Lib Dir | /usr/local/Cellar/ruby/2.5.1/lib |
| OpenSSL Version | OpenSSL 1.0.2o 27 Mar 2018 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 9.3.1 |
System Locale
| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |
fastlane files:
./fastlane/Fastfile
# fastlane Fastfile - Contains lane definitions for batch fastlane actions.
# This is fastlane's main point of entry.
# Author: Jason Charrier
# Created: April 10, 2017
# Customise this file, documentation can be found here:
# https://github.com/fastlane/fastlane/tree/master/fastlane/docs
# All available actions: https://docs.fastlane.tools/actions
# can also be listed using the `fastlane actions` command
# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`
# If you want to automatically update fastlane if a new version is available:
# update_fastlane
# This is the minimum version number required.
# Update this, if you use features of a newer version
fastlane_version "2.40.0" # we use precheck which was introduced in this version
default_platform :ios
my_hostname = ""
gather_code_coverage = lambda do |testrun_info|
# now collect our code coverage data
try_attempt = testrun_info[:try_count]
slack(
message: "Gathering code coverage for attempt #{try_attempt}",
default_payloads: []
)
#xcov(
# scheme: "Innovative"
#)
slather(
scheme: "Innovative",
workspace: "InnovativeLanguage.com/Innovative.xcworkspace",
proj: "InnovativeLanguage.com/Innovative.xcodeproj",
cobertura_xml: "true",
jenkins: "true",
output_directory: "InnovativeLanguage.com/slather_output"
)
end
platform :ios do
# actions to perform before executing any lane
before_all do
ENV["SLACK_URL"] = "https://hooks.slack.com/services/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"
my_hostname = sh "hostname"
cocoapods
end # // before_all
desc "Runs ALL the iPhone tests"
lane :iphonetest do
# use the iPhone test channel instead
ENV["SLACK_URL"] = "https://hooks.slack.com/services/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"
slack(
message: "iPhone Tests started on host: #{my_hostname}",
default_payloads: []
)
match(type: "development", readonly: true)
multi_scan(
fail_build: false,
try_count: 3,
output_directory: "test_output",
testrun_completed_block: gather_code_coverage
)
slack(
message: "iPhone Tests finished on host: #{my_hostname}"
)
end # // iphonetests
desc "Runs ALL the iPad tests"
lane :ipadtest do
# use the iPad test channel instead
ENV["SLACK_URL"] = "https://hooks.slack.com/services/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"
slack(
message: "iPad Tests started on host: #{my_hostname}",
default_payloads: []
)
match(type: "development", readonly: true)
multi_scan(
fail_build: false,
devices: ["iPad Pro (12.9-inch) (2nd generation)"],
try_count: 3,
output_directory: "test_output",
testrun_completed_block: gather_code_coverage
)
slack(
message: "iPad Tests finished on host: #{my_hostname}"
)
end # // ipadtests
desc "Updates profiles and certificates"
lane :update_profiles do
match(type: "appstore")
match(type: "development")
match(type: "adhoc")
end # // update_profiles
desc "Submit a new Beta Build to Apple TestFlight"
desc "This will also make sure the profile is up to date."
lane :beta do
slack(
message: "Build started on host: #{my_hostname}",
default_payloads: []
)
match # more information: https://codesigning.guide
gym # Build your app - more options available
slack(
message: "Build succeeded. Uploading to TestFlight.",
default_payloads: []
)
# upload to TestFlight
pilot(skip_waiting_for_build_processing: true, itc_provider: "InnovativeLanguageLearningLLC")
slack(
message: "Upload to TestFlight succeeded. Checking for metadata issues.",
default_payloads: []
)
# check for metadata issues
precheck
slack(
message: "Successfully deployed new App Update."
)
end # // beta
desc "Deploy a new version to the App Store"
lane :release do |options|
# snapshot
build_number = options[:build_number]
if build_number && !build_number.empty?
slack(
message: "Submtting build number #{build_number} for review",
default_payloads: []
)
else
slack(
message: "Submtting latest build for review",
default_payloads: []
)
end
deliver(force: true, submit_for_review: true, build_number: build_number, skip_metadata: true)
# frameit
slack(
message: "Successfully submitted app update for review."
)
end # // release
# You can define as many lanes as you want
after_all do |lane|
# This block is called, only if the executed lane was successful
end
error do |lane, exception|
slack(
message: exception.message,
success: false
)
end
end # // platform ios
# More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Platforms.md
# All available actions: https://docs.fastlane.tools/actions
# fastlane reports which actions are used
# No personal data is recorded. Learn more at https://github.com/fastlane/enhancer
./fastlane/Appfile
# fastlane Appfile - Contains some basic app info for all fastlane actions
# Author: Jason Charrier
# Created: April 10, 2017
app_identifier "com.innovativelanguage.illmobile" # The bundle identifier of your app
apple_id "[email protected]" # Your Apple email address
team_id "EV8CF9UN88" # Developer Portal Team ID
# you can even provide different app identifiers, Apple IDs and team names per lane:
# More information: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Appfile.md
fastlane gems
| Gem | Version | Update-Status |
| -------- | ------- | ------------- |
| fastlane | 2.96.1 | ✅ Up-To-Date |
Loaded fastlane plugins:
| Plugin | Version | Update-Status |
| --------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.2.3 | ✅ Up-To-Date |
Loaded gems
| Gem | Version |
| --------------------------- | ------------ |
| bundler | 1.16.2 |
| CFPropertyList | 2.3.6 |
| concurrent-ruby | 1.0.5 |
| i18n | 0.9.5 |
| minitest | 5.11.3 |
| thread_safe | 0.3.6 |
| tzinfo | 1.2.5 |
| activesupport | 4.2.10 |
| public_suffix | 2.0.5 |
| addressable | 2.5.2 |
| atomos | 0.1.2 |
| babosa | 1.0.2 |
| claide | 1.0.2 |
| clamp | 0.6.5 |
| fuzzy_match | 2.0.4 |
| nap | 1.1.0 |
| cocoapods-core | 1.5.3 |
| cocoapods-deintegrate | 1.0.2 |
| cocoapods-downloader | 1.2.1 |
| cocoapods-plugins | 1.0.0 |
| cocoapods-search | 1.0.0 |
| cocoapods-stats | 1.0.0 |
| netrc | 0.11.0 |
| cocoapods-trunk | 1.3.0 |
| cocoapods-try | 1.1.0 |
| colored2 | 3.1.2 |
| escape | 0.0.4 |
| fourflusher | 2.0.1 |
| gh_inspector | 1.1.3 |
| molinillo | 0.6.5 |
| ruby-macho | 1.1.0 |
| nanaimo | 0.2.5 |
| xcodeproj | 1.5.9 |
| cocoapods | 1.5.3 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| declarative | 0.0.10 |
| declarative-option | 0.1.0 |
| unf_ext | 0.0.7.5 |
| unf | 0.1.4 |
| domain_name | 0.5.20180417 |
| dotenv | 2.4.0 |
| emoji_regex | 0.1.1 |
| excon | 0.62.0 |
| multipart-post | 2.0.0 |
| faraday | 0.15.2 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.6 |
| faraday_middleware | 0.12.2 |
| fastimage | 2.1.3 |
| jwt | 2.1.0 |
| little-plugger | 1.1.4 |
| multi_json | 1.13.1 |
| logging | 2.2.2 |
| memoist | 0.16.0 |
| os | 0.9.6 |
| signet | 0.8.1 |
| googleauth | 0.6.2 |
| httpclient | 2.8.3 |
| mime-types-data | 3.2016.0521 |
| mime-types | 3.1 |
| uber | 0.1.0 |
| representable | 3.0.4 |
| retriable | 3.1.1 |
| google-api-client | 0.21.2 |
| json | 2.1.0 |
| mini_magick | 4.5.1 |
| multi_xml | 0.6.0 |
| plist | 3.4.0 |
| rubyzip | 1.2.1 |
| security | 0.1.3 |
| naturally | 2.1.0 |
| simctl | 1.6.4 |
| slack-notifier | 2.3.2 |
| terminal-notifier | 1.8.0 |
| unicode-display_width | 1.3.2 |
| terminal-table | 1.8.0 |
| tty-screen | 0.6.4 |
| tty-cursor | 0.5.0 |
| tty-spinner | 0.8.0 |
| word_wrap | 1.0.0 |
| rouge | 2.0.7 |
| xcpretty | 0.2.8 |
| xcpretty-travis-formatter | 1.0.0 |
| xctest_list | 1.1.3 |
| fastlane-plugin-test_center | 3.2.3 |
| mini_portile2 | 2.3.0 |
| nokogiri | 1.8.2 |
| slather | 2.4.5 |
| xcov | 1.4.2 |
generated on: 2018-05-25
jcharr1
All 15 comments
Thanks for the bug report @jcharr1, can you attach a copy of the xml file that is causing the problem?
lyndsey-ferguson
on 26 May 2018
Here's the Terminal output:
[13:12:01]: malformed XML: missing tag start
Line: 219
Position: 10599
Last 80 unconsumed characters:
<XCAccessibilityElement: 0x7f9a5a8090e0> pid: 66370, elementOrHash.elementID: 140
And the junit report with the test failure:
<?xml version='1.0' encoding='UTF-8'?>
<testsuites name='InnovativeUITests.xctest' tests='5' failures='2'>
<testsuite name='WordBankLabelUITests' tests='5' failures='2'>
<testcase classname='WordBankLabelUITests' name='testCopyWords' time='67.157'/>
<testcase classname='WordBankLabelUITests' name='testDeleteWords'>
<failure message='(([cellsNames indexOfObject:firstWord] == NSNotFound) is true) failed'>WordBankLabelUITests.m:151</failure>
</testcase>
<testcase classname='WordBankLabelUITests' name='testMoveWords' time='62.656'/>
<testcase classname='WordBankLabelUITests' name='testRenameLabel' time='57.422'/>
<testcase classname='WordBankLabelUITests' name='testUnlabelWords'>
<failure message='Failed to scroll to visible (by AX action) Button, 0x60000019f140, traits: 8858370049, label: 'Lessons', error: Error -25204 performing AXAction 2003 on element <XCAccessibilityElement: 0x7f9a5a8090e0> pid: 66370, elementOrHash.elementID: 140225245626416.326'>WordBankUITests.m:465</failure>
</testcase>
</testsuite>
</testsuites>
I have two requests:
Can you provide me with the console log for when you are running the code? I want to see the stack trace for when
multi_scanfails. Right now, I am guessing that it is failing when it is trying to read the Junit XML files to collate them. However, when I try and collate the file that you have given me (with one without failing tests), it works fine.Can you try adding this snippet of code and running it after the failure? This works fine for me, but perhaps you have accidentally edited the Junit file and the error will occur for you?
lane :collate do
UI.important(
'example: ' \
'collate the xml reports to a temporary file \'result.xml\''
)
final_report = File.join(Dir.mktmpdir, 'result.xml')
report_files = Dir['./test_output/*.xml'].map { |relpath| File.expand_path(relpath) }
sorted_report_files = report_files.sort { |f1, f2| File.mtime(f1) <=> File.mtime(f2) }
collate_junit_reports(
reports: sorted_report_files,
collated_report: final_report
)
UI.message("Collated to file: #{final_report}")
end
Hi @lyndsey-ferguson. I tried #2 and it worked fine. On a hunch, I changed your code snippet to collate the html reports instead, and bam, there's the error message. So it's happening with collating the html reports and not the junit ones. We don't actually use the html reports so an effective workaround for me would just be to disable html reports for scan. But I'd be happy to help you figure out the problem as I'm sure others would probably run into it at some point. I've attached the html reports from the same example run I gave before with that test failure. See if trying to collate them gives you the same error.
jcharr1
on 26 May 2018
It seems that xcodebuild test piped to xcpretty to generate HTML files is not properly escaping < and >. <XCAccessibilityElement: 0x7f9a5a8090e0>. I will see if that happens locally and then file an Issue with xcpretty if it looks like it has nothing to do with something I've done.
lyndsey-ferguson
on 27 May 2018
@jcharr1 I have not been able to force this particular error. Is there anyway you could provide a sample project that makes this error occur?
lyndsey-ferguson
on 27 May 2018
I may be able to fake this out by outputting this message during a test and my theory is that xcpretty will fail to escape the < and > at which point I can log a bug against xcpretty with my sample. Hopefully someone will pick it up, otherwise I will have to fix it. 😉
lyndsey-ferguson
on 30 May 2018
I think I'm facing the same problem. My build fails with this exception:
REXML::ParseException: [31m[!] Missing end tag for 'unknown' (got "section")
Line: 262
Position: 12119
Last 80 unconsumed characters:
[0m
Line: 262
Position: 12119
Last 80 unconsumed characters:
When looking to the html reports, I think the part that generates the error is this one - which contains this wrong and unclosed html tag there
<section class="test-detail snippet"></section>
<section class="test-detail"><unknown>:0</section>
Which actually seems to be part of the test error message, that was something like
No matches found for Find: Elements matching predicate '"button_register" IN identifiers' from input <unknown>
lucasecf
on 1 Jun 2018
Thanks @lucasecf, I've used that and the error from @jcharr1 to create this xcpretty issue. If you want to +1 it, that may draw their attention. In the meantime, let's see if I can provide a temporary clean up section of code here...
lyndsey-ferguson
on 1 Jun 2018
for sure will drop the +1 there. Thanks!
lucasecf
on 1 Jun 2018
If I may ask, what's the workaround to 'disable' the html reports? I tried to removed them from multi_scan by passing output_types: "junit" (since default is html,junit). According to the generated xcodebuild command, now xcpretty is not trying to generate the html anymore:
set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace ./XYZ.xcworkspace -scheme Smoke\ Tests -destination 'platform=iOS Simulator,id=85133256-457F-4F65-BCE2-B720B287B717' -derivedDataPath '/Users/XYZ-ci/Library/Developer/Xcode/DerivedData/XYZ-evukfpidstickyeicweyeextqfyj' clean build-for-testing | tee '/Users/XYZ-ci/Library/Logs/scan/XYZ-Smoke Tests.log' | xcpretty --report junit --output '/Users/XYZ-ci/Jenkins/workspace/XYZ-iOS-PullRequests/fastlane/test_output/report.junit' --report junit --output '/var/folders/mc/j9jzjcw1653f2ls_jqmf0d2h0000gn/T/junit_report20180601-7961-1j86ri4
But now multi_scan is crashing for me. I'm not sure why, but it seems that it's trying to still get the html files from the filesystem, but it's obviously not there:
bundler: failed to load command: fastlane (/Users/XYZ-ci/.rbenv/versions/2.3.0/bin/fastlane)
TypeError: [31m[!] no implicit conversion from nil to integer[0m
/Users/XYZ-ci/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/fastlane-plugin-test_center-3.2.4/lib/fastlane/plugin/test_center/helper/reportname_helper.rb:81:in `[]'
/Users/XYZ-ci/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/fastlane-plugin-test_center-3.2.4/lib/fastlane/plugin/test_center/helper/reportname_helper.rb:81:in `html_last_reportname'
/Users/XYZ-ci/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/fastlane-plugin-test_center-3.2.4/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:173:in `testrun_info'
Checking the reported line, seems like html_index is nil, which crashes the next line. But why is this method being invoked?
That's how I'm invoking multi_scan:
multi_scan(scheme: "Smoke Tests",
devices: ["iPhone 8"],
clean: true,
output_types: "junit",
try_count: 3)
@lucasecf first to the issue you are seeing: I just accidentally introduced that defect with the latest update. I'll get on a fix right away. The idea with the workaround is for me to use the callback after each run to correct the file if it is malformed.
This commit will give you an idea of my plan. This is not functional yet, so don't use it. 😉
lyndsey-ferguson
on 1 Jun 2018
just wondering If I was doing something wrong 🤔 😅 . Thanks for the feedback! 👍
lucasecf
on 1 Jun 2018
@lucasecf, the crash that occurs when you _do not_ pass in 'html' as one of the report types is fixed in #54
lyndsey-ferguson
on 1 Jun 2018
Fixed in v3.2.6
lyndsey-ferguson
on 4 Jun 2018
Related issues
Ravi61
·
8Comments
CorbinMontague
·
7Comments
ssharma0312
·
4Comments
ishammoho
·
9Comments
ryanpato
·
4Comments