Fastlane-plugin-test_center: multi_scan's use of tests_from_xctestrun action not able to find xctest binary

Created on 6 Aug 2018 · 58Comments · Source: lyndsey-ferguson/fastlane-plugin-test_center

I executed the following lane. Just when it is about to start the test, I got the following error and saying /Library/Ruby/Gems/2.3.0/gems/xctest_list-1.1.3/lib/xctest_list.rb:27:in `binary_path': [!] Missing xctest binary:' Please help me out in resolving this issue. Thank you in advance!

fastlane buildBVT

+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

+------+------------------+-------------+
| fastlane summary |
+------+------------------+-------------+
| Step | Action | Time (in s) |
+------+------------------+-------------+
| 1 | opt_out_usage | 0 |
| 2 | default_platform | 0 |
| 💥 | multi_scan | 37 |
+------+------------------+-------------+

Looking for related GitHub issues on fastlane/fastlane...

Found no similar issues. To create a new issue, please visit:
https://github.com/fastlane/fastlane/issues/new
Run fastlane env to append the fastlane environment to your issue
/Library/Ruby/Gems/2.3.0/gems/xctest_list-1.1.3/lib/xctest_list.rb:27:in binary_path': [!] Missing xctest binary: '../DerivedData/Build/Products/Release/...DesktopUITests-Runner.app/Contents/PlugIns/...DesktopUITests.xctest/...DesktopUITests' (RuntimeError) from /Library/Ruby/Gems/2.3.0/gems/xctest_list-1.1.3/lib/xctest_list.rb:36:inobjc_tests'
from /Library/Ruby/Gems/2.3.0/gems/xctest_list-1.1.3/lib/xctest_list.rb:68:in tests' from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb:15:inblock in xctestrun_tests'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb:14:in each' from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb:14:inxctestrun_tests'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/actions/tests_from_xctestrun.rb:7:in run' from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/helper/test_collector.rb:58:intestables_tests'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:54:in scan_testable' from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:41:inblock in scan'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:40:in each' from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/helper/correcting_scan_helper.rb:40:inscan'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-plugin-test_center-3.5.3/lib/fastlane/plugin/test_center/actions/multi_scan.rb:23:in run' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:253:inblock (2 levels) in execute_action'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/actions/actions_helper.rb:50:in execute_action' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:232:inblock in execute_action'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:227:in chdir' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:227:inexecute_action'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:157:in trigger_action_by_name' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/fast_file.rb:149:inmethod_missing'
from Fastfile:327:in block (2 levels) in parsing_binding' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/lane.rb:33:incall'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:49:in block in execute' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:45:inchdir'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/runner.rb:45:in execute' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/lane_manager.rb:56:incruise_lane'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/command_line_handler.rb:36:in handle' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/commands_generator.rb:108:inblock (2 levels) in run'
from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call' from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:inrun'
from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:74:inrun!'
from /Library/Ruby/Gems/2.3.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/commands_generator.rb:333:inrun'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/commands_generator.rb:41:in start' from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/fastlane/lib/fastlane/cli_tools_distributor.rb:108:intake_off'
from /Library/Ruby/Gems/2.3.0/gems/fastlane-2.101.1/bin/fastlane:20:in fastlane:22:inload'
from /usr/local/bin/fastlane:22:in `

✅ fastlane environment ✅

Stack

| Key | Value |
| --------------------------- | -------------------------------------------------------------- |
| OS | 10.13.5 |
| Ruby | 2.3.3 |
| Bundler? | false |
| Git | git version 2.15.2 (Apple Git-101.1) |
| Installation Source | /usr/local/bin/fastlane |
| Host | Mac OS X 10.13.5 (17F70a) |
| Ruby Lib Dir | /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/lib |
| OpenSSL Version | LibreSSL 2.2.7 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode.app/Contents/Developer/ |
| Xcode Version | 9.4.1 |

System Locale

| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | | |
| LANGUAGE | | |

fastlane files:

./Fastfile

# Customise this file, documentation can be found here:
# https://github.com/fastlane/fastlane/tree/master/fastlane/docs
# All available actions: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Actions.md
# can also be listed using the `fastlane actions` command

# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`

# If you want to automatically update fastlane if a new version is available:
# update_fastlane
#update_fastlane
# This is the minimum version number required.
# Update this, if you use features of a newer version
#fastlane_version "1.105.1"
opt_out_usage

default_platform :mac

platform :mac do
before_all do
# ENV["SLACK_URL"] = "https://hooks.slack.com/services/..."
end

desc "Execute BVT Tests"
lane :bvt do
scan(
workspace:"../CitrixFiles.xcworkspace",
scheme:"SFMDriveMapperBVT",
configuration:"Release",
code_coverage: true,
derived_data_path: "../DerivedData",
output_types: "html,junit,json-compilation-database",
output_directory: "../build/reports/UI_test",
buildlog_path: "../build/logs/xcodebuild"
)
end


desc "Build ShareFile Drive Mapper Project and run BVT test"
lane :buildBVT do |options|
    scheme_to_test = "SFMDriveMapperBVT"
    multi_scan(
        workspace:"../CitrixFiles.xcworkspace",
        scheme: scheme_to_test,
        configuration:"Release",
        code_coverage: true,
        derived_data_path: "../DerivedData",
        output_types: "html,junit,json-compilation-database",
        output_directory: "../build/reports/UI_test",
        try_count: 6,
        buildlog_path: "../build/logs/xcodebuild"
    )
end

# Rename JUnit report file to match standard (required for SonarQube analysis)
  # sh "cd ../build/reports/UI_test && mv cobertura.xml results/SFDesktopTestCoverage.xml "
end


# You can define as many lanes as you want

after_all do |lane|
# This block is called, only if the executed lane was successful

# slack(
#   message: "Successfully deployed new App Update."
# )
end

error do |lane, exception|
# slack(
#   message: exception.message,
#   success: false
# )
end


# More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Platforms.md
# All available actions: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Actions.md

# fastlane reports which actions are used
# No personal data is recorded. Learn more at https://github.com/fastlane/enhancer

./Appfile

app_identifier "com.sharefile.desktop.widget" # The bundle identifier of your app
apple_id "[email protected]" # Your Apple email address

team_id "RM4N8HY7K5"  # Developer Portal Team ID

# you can even provide different app identifiers, Apple IDs and team names per lane:
# More information: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Appfile.md

fastlane gems

| Gem | Version | Update-Status |
| -------- | ------- | ------------- |
| fastlane | 2.101.1 | ✅ Up-To-Date |

Loaded fastlane plugins:

| Plugin | Version | Update-Status |
| --------------------------- | ------- | ------------- |
| fastlane-plugin-test_center | 3.5.3 | ✅ Up-To-Date |

Loaded gems

| Gem | Version |
| --------------------------- | ------------ |
| did_you_mean | 1.0.0 |
| slack-notifier | 2.3.2 |
| atomos | 0.1.2 |
| claide | 1.0.2 |
| colored2 | 3.1.2 |
| nanaimo | 0.2.6 |
| xcodeproj | 1.5.9 |
| rouge | 2.0.7 |
| xcpretty | 0.2.8 |
| addressable | 2.5.2 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| public_suffix | 2.0.5 |
| tty-screen | 0.6.5 |
| tty-cursor | 0.6.0 |
| tty-spinner | 0.8.0 |
| babosa | 1.0.2 |
| colored | 1.2 |
| highline | 1.7.10 |
| commander-fastlane | 4.4.6 |
| excon | 0.62.0 |
| faraday | 0.15.2 |
| unf_ext | 0.0.7.5 |
| unf | 0.1.4 |
| domain_name | 0.5.20180417 |
| http-cookie | 1.0.3 |
| faraday-cookie_jar | 0.0.6 |
| fastimage | 2.1.3 |
| gh_inspector | 1.1.3 |
| mini_magick | 4.5.1 |
| multi_json | 1.13.1 |
| multi_xml | 0.6.0 |
| rubyzip | 1.2.1 |
| security | 0.1.3 |
| xcpretty-travis-formatter | 1.0.0 |
| dotenv | 2.5.0 |
| bundler | 1.16.3 |
| faraday_middleware | 0.12.2 |
| naturally | 2.2.0 |
| simctl | 1.6.5 |
| declarative | 0.0.10 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| mime-types-data | 3.2016.0521 |
| mime-types | 3.1 |
| jwt | 2.1.0 |
| memoist | 0.16.0 |
| os | 1.0.0 |
| signet | 0.8.1 |
| googleauth | 0.6.3 |
| httpclient | 2.8.3 |
| google-api-client | 0.23.4 |
| emoji_regex | 0.1.1 |
| json | 2.1.0 |
| io-console | 0.4.5 |
| CFPropertyList | 3.0.0 |
| libxml-ruby | 2.9.0 |
| plist | 3.4.0 |
| unicode-display_width | 1.4.0 |
| terminal-table | 1.8.0 |
| xctest_list | 1.1.3 |
| fastlane-plugin-test_center | 3.5.3 |

generated on: 2018-08-06

☑️ tests_from_xctestrun ♻️ multi_scan 🐞bug

Source

homebizjunk

Most helpful comment

Oh, could you include this step (selecting All versions):

gem uninstall xctest_list

And then run the above steps

lyndsey-ferguson on 14 Aug 2018

🎉1 😄1 👍1

All 58 comments

There is a problem parsing the xctestrun file. Would you be willing to list out the directory contents of ../DerivedData/Build/Products/Release/...DesktopUITests-Runner.app and attach the xctestrun file found in the ../DerivedData/Build/Products directory?

lyndsey-ferguson on 6 Aug 2018

Hi ! Thank you for looking into my issue.

The code that I put into Fastfile are as follows:
lane :buildBVT do |options|
test_run_block = lambda do |testrun_info|
failed_test_count = testrun_info[:failed].size

    if failed_test_count > 0
        UI.important('The run of tests would finish with failures due to fragile tests here.')

        try_attempt = testrun_info[:try_count]
        if try_attempt < 2
            UI.header('Since we are using :multi_scan, we can re-run just those failing tests!')
        end
    end
end

result = multi_scan(
    workspace:"../CitrixFiles.xcworkspace",
    scheme: "SFMDriveMapperBVT",
    code_coverage: true,
    derived_data_path: "../DerivedData",
    output_types: "html,junit,json-compilation-database",
    output_directory: "../build/reports/UI_test",
    try_count: 5,
    buildlog_path: "../build/logs/xcodebuild",
    testrun_completed_block: test_run_block
)
unless result[:failed_testcount].zero?
    UI.info("There are #{result[:failed_testcount]} legitimate failing tests")
end

end

Please find attached the structure for the DesktopUITests-Runner.app
screen shot 2018-08-06 at 2 54 50 pm

Please find attached the structure for the xctestrun file
screen shot 2018-08-06 at 2 55 33 pm

homebizjunk on 6 Aug 2018

The xctestrun file is too large to attached here.

homebizjunk on 6 Aug 2018

Thanks @homebizjunk, in order to investigate further, I need the file that has the extension xctestrun. I'm including an example image and a sample xctestrun file (zipped) to help clarify what I am looking for.

screen shot 2018-08-06 at 3 06 25 pm

AtomicBoy_iphonesimulator11.1-x86_64.xctestrun.zip

lyndsey-ferguson on 6 Aug 2018

The xctestrun is just a plist file, so I don't imagine it could be too large. I suppose it could be possible...how big is your xctestrun file?

lyndsey-ferguson on 6 Aug 2018

Hi lyndsey-ferguson! Please find the file attached.
SFMDriveMapperBVT_macosx10.13-x86_64.xctestrun.zip

homebizjunk on 6 Aug 2018

Screenshot with the error message screen shot 2018-08-06 at 4 02 22 pm

homebizjunk on 6 Aug 2018

Thank you, could you also provide the structure of the _parent_ directory for ../DerivedData/Build/Products/Release/...DesktopUITests-Runner.app?

lyndsey-ferguson on 6 Aug 2018

screen shot 2018-08-06 at 5 08 23 pm

homebizjunk on 6 Aug 2018

Thank you for all your help!

homebizjunk on 6 Aug 2018

Sorry for being so confusing (I wasn't sure of your directory structure/naming). What about your DerivedData/Build/Products/Release directory?

lyndsey-ferguson on 6 Aug 2018

screen shot 2018-08-06 at 6 08 38 pm

homebizjunk on 7 Aug 2018

Ok, thank you. I'll be able to investigate this tomorrow morning

lyndsey-ferguson on 7 Aug 2018

I think that the problem is that you are using a relative path for the :derived_data_path. Can you update your code to pass in derived_data_path: File.absolute_path("../DerivedData") instead of derived_data_path: "../DerivedData"?

If you can verify, I can update the plugin code to fix this.

lyndsey-ferguson on 7 Aug 2018

Hi @lyndsey-ferguson, I've made code change based on your suggestion and ran it again. I'm still getting the same error.

homebizjunk on 7 Aug 2018

Okay. Before I start adding logging to the plugin and its actions, can you do one more thing?

Change the code in the lane that calls multi_scan to be:

derived_data_path = File.absolute_path("../DerivedData")
puts "derived_data_path: #{derived_data_path}"
next unless Dir.exist?(derived_data_path)

result = multi_scan(
    workspace:"../CitrixFiles.xcworkspace",
    scheme: "SFMDriveMapperBVT",
    code_coverage: true,
    derived_data_path: derived_data_path,
    output_types: "html,junit,json-compilation-database",
    output_directory: "../build/reports/UI_test",
    try_count: 5,
    buildlog_path: "../build/logs/xcodebuild",
    testrun_completed_block: test_run_block
)

If this is not the correct path, fastlane will stop early.

lyndsey-ferguson on 7 Aug 2018

@lyndsey-ferguson, I'm still getting the same error after I added the three lines of code as indicated in the screenshot.

screen shot 2018-08-07 at 12 27 39 pm

homebizjunk on 7 Aug 2018

And what is the derived data path that is printed to the console?

lyndsey-ferguson on 7 Aug 2018

The full path was printed starting from the /Users/........./ShareFileDesktopUITests.xctest/ShareFileDesktopUITests

homebizjunk on 7 Aug 2018

The whole path is as follows: /Users/calvinchang/Documents/LatestCFMacGA/sfdesktop/BuildSystem/DerivedData/Build/Products/Debug/ShareFileDesktopUITests-Runner.app/Contents/PlugIns/ShareFileDesktopUITests.xctest/ShareFileDesktopUITests

homebizjunk on 7 Aug 2018

And that is the output from puts "derived_data_path: #{derived_data_path}"? it looks like it should be /Users/calvinchang/Documents/LatestCFMacGA/sfdesktop/BuildSystem/DerivedData instead.

lyndsey-ferguson on 7 Aug 2018

Yes, it is. "sfdesktop" is the project name in Jenkins.

homebizjunk on 7 Aug 2018

I have added logging to the tool in the branch issue-63-cant-find-xctest-binary. Can you modify your Pluginfile from:

gem 'fastlane-plugin-test_center'

to:

gem 'fastlane-plugin-test_center', :git => "git://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-63-cant-find-xctest-binary"

Then run your :buildBVT lane again. When it fails, please attach the issue_63_log.txt file (in your :output_directory) to this Issue.

Thanks!

lyndsey-ferguson on 8 Aug 2018

@lyndsey-ferguson I replaced the line as you instructed and ran the :buildBVT lane. I don't see the issue_63_log.txt file got generated where the output_directory is located. screen shot 2018-08-08 at 11 52 40 pm

homebizjunk on 9 Aug 2018

Whoops, I forgot to tell you to run bundle install to get the latest.

lyndsey-ferguson on 9 Aug 2018

Hi @lyndsey-ferguson, Is there something else missing for not getting the log file generated then?

homebizjunk on 9 Aug 2018

Nope, you just need to bundle install then run the :buildBVT lane again.

lyndsey-ferguson on 9 Aug 2018

I ran "bundle install" already last night before I ran fastlane. That's why I attached the screenshot to show that I ran the "bundle install". It didn't generate the log file.

homebizjunk on 9 Aug 2018

I added a log statement as soon as :multi_scan starts to print out the log file location.

lyndsey-ferguson on 9 Aug 2018

you may need to remove the fastlane-plugin-test_center entries from the Gemfile.lock for the updated version to take.

lyndsey-ferguson on 9 Aug 2018

There are two entries for fastlane-plugin-test_center in the Gemfile.lock file. Which one should I be removing? There is one entry under GIT, the other entry is under Dependencies.
screen shot 2018-08-09 at 1 42 29 pm
screen shot 2018-08-09 at 1 43 10 pm

homebizjunk on 9 Aug 2018

First, remove the entire GIT entry. Then, in dependencies, remove that one too.

OR, can you remove the Gemfile.lock file for now?

lyndsey-ferguson on 9 Aug 2018

@homebizjunk would you be able to remove the Gemfile.lock file for testing this? That would be the easiest and most reliable step.

lyndsey-ferguson on 10 Aug 2018

Sorry, I've been tied up with other more urgent work. I'm trying it now.

homebizjunk on 10 Aug 2018

👍1

I removed the Gemfile.lock file and ran "bundle install" and then ran "fastlane buildBVT". For some reason, I'm still unable to generate the log file that you mentioned above.
Looks like it is fetching the test_center plugin from the right branch
screen shot 2018-08-10 at 12 11 04 pm

homebizjunk on 10 Aug 2018

would you mind debugging this? If you can follow, these steps:

gem 'pry-byebug'

to your Gemfile.

run bundle install
Add:

require 'pry-byebug'

To your Fastfile

byebug

Just before the call to :multi_scan

Run your fastlane command:

bundle exec fastlane buildBVT

When the debugger appears in your Terminal, create a breakpoint by typing:

b Fastlane::Actions::MultiScanAction.run

Allow the debugger to continue by typing c for continue.
It should break here:

[5, 14] in /Users/lyndsey.ferguson/repo/fastlane-plugin-test_center/lib/fastlane/plugin/test_center/actions/multi_scan.rb
    5:     require 'shellwords'
    6:     require 'xctest_list'
    7:     require 'plist'
    8:
    9:     class MultiScanAction < Action
=> 10:       def self.run(params)
    10:           FileUtils.mkdir_p(params[:output_directory])
    11:           $issue_63_logger = File.open(File.join(params[:output_directory], 'issue_63_log.txt'), 'w')
    12:           UI.message("#{File.basename($issue_63_logger)} will be found at: '#{File.dirname($issue_63_logger)}'")

Does it?

If it does, press n to 4 times to step to the next line. After it passes UI.message(...) you should see something printed to the Console.

lyndsey-ferguson on 10 Aug 2018

screen shot 2018-08-10 at 4 22 42 pm

homebizjunk on 10 Aug 2018

This is what is printed on the console.
screen shot 2018-08-10 at 4 27 09 pm

homebizjunk on 10 Aug 2018

Does the file end up being put in ../build/reports/UI_test? OR do you see an extra build directory up one level from where you are expecting your results?

lyndsey-ferguson on 10 Aug 2018

I've updated the code to output the absolute path. Now that we know that the log file is being created, you can remove the byebug command and let it run as normal.

lyndsey-ferguson on 13 Aug 2018

Hi @lyndsey-ferguson! Please find the issue_63 log file attached. issue_63_log.txt