Fastlane-plugin-test_center: 2.3.1 - undefined method `text=' for nil:NilClass

[szymon-kazmierczak]

New Issue Checklist

• [x] Updated fastlane-plugin-test_center to the latest version
• [x] I read the README.md
• [x] I reviewed the example Fastfile

Issue Description

When running multi_scan, after a finished rerun I get an error: undefined method `text=' for nil:NilClass

See stacktrace below.

An example of how the test_center action is being called.

```default_platform(:ios)

PROJECT = "xxx.xcodeproj"
SCHEME = "xxxAppTests"
DEVICE = "iPhone SE (10.3.1)"
DERIVED_DATA_PATH = "derivedData"
RETRY_COUNT = 3

platform :ios do

#build the project and store the Derived Data
desc 'build-for-testing'
lane :build_for_testing do
    scan(
        project: PROJECT,
        scheme: SCHEME,
        device: DEVICE,
        build_for_testing: true,
        derived_data_path: DERIVED_DATA_PATH
        )
end

lane :ui_test_without_building do
    multi_scan(
        project: PROJECT,
        scheme: SCHEME,
        device: DEVICE,
        derived_data_path: DERIVED_DATA_PATH,
        try_count: RETRY_COUNT,
        test_without_building: true,
        custom_report_file_name: 'earlGrey_report.html,earlGrey_report.xml',
        only_testing: ["xxxAppTests"]
        )
end

desc 'Run UI Tests'
lane :ui_test do
    build_for_testing
    ui_test_without_building
end

end

##### Complete output when running fastlane, including the stack trace and command used

bundler: failed to load command: fastlane (/usr/local/bin/fastlane)
NoMethodError: [!] undefined method text=' for nil:NilClass /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/actions/collate_html_reports.rb:101:inupdate_test_counts'
/Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/actions/collate_html_reports.rb:19:in run' /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:123:incollate_reports'
/Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:83:in scan_testable' /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:39:inblock in scan'
/Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:38:in each' /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:38:inscan'
/Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.2.1/lib/fastlane/plugin/test_center/actions/multi_scan.rb:23:in run' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:257:inblock (2 levels) in execute_action'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/actions/actions_helper.rb:50:in execute_action' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:236:inblock in execute_action'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:231:in chdir' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:231:inexecute_action'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:157:in trigger_action_by_name' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/fast_file.rb:148:inmethod_missing'
Fastfile:36:in block (2 levels) in parsing_binding' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/lane.rb:33:incall'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:204:in try_switch_to_lane' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:146:intrigger_action_by_name'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/fast_file.rb:148:in method_missing' Fastfile:51:inblock (2 levels) in parsing_binding'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/lane.rb:33:in call' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:49:inblock in execute'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:45:in chdir' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/runner.rb:45:inexecute'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/lane_manager.rb:59:in cruise_lane' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/command_line_handler.rb:36:inhandle'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/commands_generator.rb:108:in block (2 levels) in run' /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:incall'
/Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in run' /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:inrun_active_command'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:75:in run!' /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:inrun!'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/commands_generator.rb:333:in run' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/commands_generator.rb:42:instart'
/Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/fastlane/lib/fastlane/cli_tools_distributor.rb:107:in take_off' /Library/Ruby/Gems/2.3.0/gems/fastlane-2.91.0/bin/fastlane:20:in /usr/local/bin/fastlane:22:in load' /usr/local/bin/fastlane:22:in

```

[lyndsey-ferguson]

@szymon-kazmierczak could you attach the HTML reports after removing any sensitive information?

[szymon-kazmierczak]

Sure, I'll attach them in a few minutes

[szymon-kazmierczak]

Here you go.
reports.zip

There are two reports. First one is a full test suite with 3 failing tests. The 2nd run reruns the failing 3 tests and they all pass.

[lyndsey-ferguson]

Ug, the collate_html_reports worked for me when I tested it using this lane:

lane :collate_html do
  html_reports = Dir.glob('./test_output/*.html')
  UI.message("Found these reports: #{html_reports}")
  collate_html_reports(
    reports: html_reports.map { |report| File.absolute_path(report) },
    collated_report: File.absolute_path('./test_output/report.html')
  )
end

Console

[✔] 🚀
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
|                                                                    Used plugins                                                                     |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
| Plugin                      | Version | Action                                                                                                      |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+
| fastlane-plugin-test_center | 3.2.1   | suppressed_tests suppress_tests suppress_tests_from_junit tests_from_xctestrun tests_from_junit multi_scan  |
|                             |         | collate_html_reports collate_junit_reports                                                                  |
+-----------------------------+---------+-------------------------------------------------------------------------------------------------------------+

[09:59:41]: You have require'd a gem, if this is a third party gem, please use `fastlane_require 'fastlane/actions/scan'` to ensure the gem is installed locally.
[09:59:41]: Driving the lane 'collate_html' 🚀
[09:59:41]: Found these reports: ["./test_output/earlGrey_report.html", "./test_output/earlGrey_report-2.html"]
[09:59:41]: ----------------------------------
[09:59:41]: --- Step: collate_html_reports ---
[09:59:41]: ----------------------------------

+------+----------------------+-------------+
|             fastlane summary              |
+------+----------------------+-------------+
| Step | Action               | Time (in s) |
+------+----------------------+-------------+
| 1    | collate_html_reports | 0           |
+------+----------------------+-------------+

[09:59:41]: fastlane.tools finished successfully 🎉

#######################################################################
# fastlane 2.91.0 is available. You are on 2.86.2.
# You should use the latest version.
# Please update using `bundle update fastlane`.
#######################################################################

2.91.0 Improvements and Fix For Crash
* Fix fastlane crash when running a lane (#12298) via Iulian Onofrei
* [pilot]remove emoji from changelog (#12191) via Koen Punt
* Fix incorrect action deprecation notes and details formatting in docs and terminal (#12287) via Iulian Onofrei
* [adb] Pick up ANDROID_SDK_ROOT environment variable (#12198) via Adam Cohen-Rose

2.90.0 Improvements
* [match][spaceship] Fixed matchfile overriding fastfile team_id (#12235) via Josh Holtz
* [action] Allow multiple paths with wildcards in git_add action. (#9875) via Thi
* [action] Adds an option android_gcm_sender_id to onesignal action as optional (#12245) via Bruno Arueira
* [action] Getting Derived Data path value from Xcode preferences (#12232) via Bruno Guidolim
* Add new parameter to allow custom Fastfile path for LaneManager (#12239) via Felix Krause
* Improve behavior of cfpropertylist extension (#11506) via Jimmy Dee
* [screengrab] Adding Falcon as a ScreenshotStrategy (#11535) via Carlos Sessa
* [action] Add support for Apple TV to set_changelog  (#12199) via David Cordero
* [action] Added option to set path to lint using swiftlint action (#12197) via Bruno Guidolim
* [gym] Fix Gym skip_archive option (#12220) via Renzo

2.89.0 Improvements
* [action] Make sure get_version_number targets respond to test_target_type? (#12212) via Josh Holtz
* [action] Make enable_automatic_code_signing for all configurations (#12187) via Nicolas Braun
* [produce]Use correct spelling of icloud service in product/service.rb (#12134) via Jakub Knejzlík
* [action] Add Support for Registering Mac Devices on Apple Developer Portal (#11978) via Baumchen
* [spaceship] Spaceship delete testflight groups (#12111) via Kenneth Wong
* [produce] Enable set 'NFC Tag Reading' at 'fastlane produce' cli. (#12179) (#12201) via Kazuhide Takahashi
* [frameit] frameit documentation describes all parameters from the Framefile (#11662) via funnel20
* [supply] Clarify difference between SUPPLY_JSON_KEY and SUPPLY_JSON_KEY_DATA via Aaron Brager
* [action] added custom api_url option to testfairy (#12153) via Josh Holtz

To see all new releases, open https://github.com/fastlane/fastlane/releases

Please update using `bundle update fastlane`

Attached is result html file.

report.html.zip

[szymon-kazmierczak]

hmm, I'm on fastlane-2.91.0. Maybe there were some changes between your version 2.86.2 and 2.91.0 ?

I was using

custom_report_file_name: 'earlGrey_report.html,earlGrey_report.xml',

and I wasn't specifying the output types at all. I updated it to match the suggested format

output_types: 'html,junit',
output_files: 'earlGrey_report.html, earlGrey_report.xml'

but its still failing, so that's definitely not the issue here.

I'll keep poking around!

[lyndsey-ferguson]

@szymon-kazmierczak if you use the same report files and the lane I attached, does it work for you?

[szymon-kazmierczak]

@lyndsey-ferguson yep it did

[lyndsey-ferguson]

So, it seems that the 2nd report does _not_ have a test-failures element. If, for some reason, the code is picking that up as the first modified, and using that as the basis for updating the reports, this thing will fail. Are you by chance modifying that file in a completion block after at least one test run?

[szymon-kazmierczak]

No changes are being made between test runs.

I Just realised that I haven't attached the final collated report to the zip file above. There should be 3 files - 2 test runs, earlGrey_report.html, earlGrey_report2.html and the third file, report.html, which is the collated report. It doesn't seem to have anything of value in it though.

report.html.zip

I have to note that I have not used the collated HTML reports feature before, until today I was using the plugin version without that feature (and the reports were being generated successfully). I was on version 3.1.3 to be exact.

[lyndsey-ferguson]

For now, you could avoid this issue by moving the HTML files out of your results folder using the completion block: sorry for the bug, I'll have it sorted out in the next few days (busy schedule).

[szymon-kazmierczak]

I'll skip the HTML reports for now, jUnit style is more than enough for the time being 👍Appreciate the prompt response and support! If there's anything I can do to help, let me know 👍

[lyndsey-ferguson]

@szymon-kazmierczak during a the few mins of downtime that I have, I started looking at replicating the issue you've encountered and noticed something: you are building for testing before hand. You don't have to do that. Just run multi_scan without the :test_without_building and it will perform the scan version of building for testing.

[lyndsey-ferguson]

@szymon-kazmierczak, I've made a branch that will print out debug messages; could you update your ./fastlane/Pluginfile to be:

gem 'fastlane-plugin-test_center', :git => '[email protected]:lyndsey-ferguson/fastlane-plugin-test_center.git', :branch => ''issue-45-multi_scan-fails-when-collating-htmls"

_instead of_

gem 'fastlane-plugin-test_center'

?

Then run bundle install and then re-run your lane and attach the output to this Issue?

Thanks!

[szymon-kazmierczak]

@lyndsey-ferguson interesting, didnt know that!

So just to be sure - if I run multi_scan and pass in derived_data_path, it will first build for testing and store it in that location, and then it will run the tests (without building) using that derived_data_path? The reason I'm asking is because I have other tasks that reuse that derived data path :)

I rerun the tests using your branch and got the following error

[!] Valid HTML file, PATH_TO_PROJECT/fastlane/test_output/report.html, is not an Xcode test report

[lyndsey-ferguson]

@szymon-kazmierczak yes, multi_scan will use scan to build for testing, and then run scan with test without building up to the :try_count number of times. That derived data path will have the same thing that you would have with your setup.

I guessed the problem this morning while brushing my teeth this morning. I _think_ that the first report, report.html is an invalid test HTML file (missing test-suites) and it is left over from some prior activity and thus is the oldest and is chosen as the _main_ report file which the others should be merged into.

The code is looking for *.html not reportname*.html, I think if I specify the report name better, it will fix the bug.

[szymon-kazmierczak]

ah. I cleaned up my test_output directory, and rerun the tests and it seems to have worked (I'm still on your dev branch).

The reports generated:

reports: ["/PATH_TO_PROJECT/fastlane/test_output/earlGreyReport.html", "/PATH_TO_PROJECT/fastlane/test_output/earlGreyReport-2.html"]

Thing is, the process collated the 2 reports that were generated (earlGreyReport.html and earlGreyReport-2.html) into earlGreyReport.html, and the earlGreyReport-2.html is gone. Is that intended effect? I thought the collated report would be a separate file and the original reports would be left behind.

[lyndsey-ferguson]

It is intended that you get one final report of the whole multi_scan to give you an overall view of what is really a problem (or not).

[lyndsey-ferguson]

Why do you want the intermediate report files?

[lyndsey-ferguson]

Fixed in v3.2.2

[szymon-kazmierczak]

Thats fair! My use case is to store all the test results (grouped together per test run) with their passes, failures and exceptions in datadog 👍

[lyndsey-ferguson]

Sounds good, then the testrun completion callback is perfect. Thanks for filing the bug report and working with me to figure this out!

Don’t forget to update your Pluginfile to use just the gem (not repo/branch).