Fastlane-plugin-test_center: multi_scan doesn't find test cases in Xcode 10-generated .xcrun files

Created on 27 Sep 2018 · 80Comments · Source: lyndsey-ferguson/fastlane-plugin-test_center

New Issue Checklist

[x] Updated fastlane-plugin-test_center to the latest version
[x] I read the README.md
[x] I reviewed the example(s) for the action(s) I am using
[x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see

Issue Description

We're running multi_scan as part of our CI pipeline. It's invoked like this:

    output_directory = options[:output_directory]
    multi_scan(
      scheme: 'PipelineUITests',
      devices: options[:devices].split(':'),
      output_directory: output_directory,
      derived_data_path: options[:derived_data_path],
      fail_build: false,
      output_files: 'report.junit',
      output_types: 'junit',
      try_count: 3,
      slack_message: options[:slack_message],
      slack_channel: '...',
      xcargs: 'IDEBuildOperationMaxNumberOfConcurrentCompileTasks=`sysctl -n hw.ncpu`',
      include_simulator_logs: true,
      test_without_building: true
    )

As you can see, we pass some options to it in Ruby. We're confident that these parameters are not a problem because scan works normally (with the same parameters, minus try_count of course!).

Expected result

UI tests run

Actual result

We see the message There are no tests to run in testable 'PipelineUITests'. Skipping.

Environment

🚫 fastlane environment 🚫


Stack

| Key                         | Value                                       |

| --------------------------- | ------------------------------------------- |

| OS                          | 10.13.6                                     |

| Ruby                        | 2.2.4                                       |

| Bundler?                    | false                                       |

| Git                         | git version 2.16.1                          |

| Installation Source         | ~/.fastlane/bin/bundle/bin/fastlane         |

| Host                        | Mac OS X 10.13.6 (17G65)                    |

| Ruby Lib Dir                | ~/.fastlane/bin/bundle/lib                  |

| OpenSSL Version             | OpenSSL 1.0.2g  1 Mar 2016                  |

| Is contained                | true                                        |

| Is homebrew                 | false                                       |

| Is installed via Fabric.app | false                                       |

| Xcode Path                  | /Applications/Xcode.app/Contents/Developer/ |

| Xcode Version               | 10.0                                        |

System Locale

| Variable | Value       |   |

| -------- | ----------- | - |

| LANG     | en_US.UTF-8 | ✅ |

| LC_ALL   | en_US.UTF-8 | ✅ |

| LANGUAGE | en_US.UTF-8 | ✅ |

fastlane gems

| Gem      | Version | Update-Status      |

| -------- | ------- | ------------------ |

| fastlane | 2.87.0  | 🚫 Update available |

Loaded fastlane plugins:

| Plugin                                         | Version | Update-Status |

| ---------------------------------------------- | ------- | ------------- |

| fastlane-plugin-setup_fragile_tests_for_rescan | 2.1.1   | ✅ Up-To-Date  |

| fastlane-plugin-test_center                    | 3.5.7   | ✅ Up-To-Date  |

Loaded gems


| Gem                                            | Version      |

| ---------------------------------------------- | ------------ |

| CFPropertyList                                 | 2.3.5        |

| addressable                                    | 2.5.1        |

| atomos                                         | 0.1.2        |

| babosa                                         | 1.0.2        |

| bigdecimal                                     | 1.2.6        |

| bundler                                        | 1.14.6       |

| claide                                         | 1.0.2        |

| colored                                        | 1.2          |

| colored2                                       | 3.1.2        |

| commander-fastlane                             | 4.4.6        |

| declarative                                    | 0.0.10       |

| declarative-option                             | 0.1.0        |

| domain_name                                    | 0.5.20170404 |

| dotenv                                         | 2.2.0        |

| excon                                          | 0.55.0       |

| faraday                                        | 0.12.1       |

| faraday-cookie_jar                             | 0.0.6        |

| faraday_middleware                             | 0.11.0.1     |

| fastimage                                      | 2.1.0        |

| fastlane-plugin-setup_fragile_tests_for_rescan | 2.1.1        |

| fastlane-plugin-test_center                    | 3.5.7        |

| fastlane-plugin-xbluepill                      | 0.1.2        |

| gh_inspector                                   | 1.1.3        |

| google-api-client                              | 0.13.6       |

| googleauth                                     | 0.5.1        |

| highline                                       | 1.7.8        |

| http-cookie                                    | 1.0.3        |

| httpclient                                     | 2.8.3        |

| hurley                                         | 0.2          |

| io-console                                     | 0.4.3        |

| json                                           | 1.8.1        |

| jwt                                            | 1.5.6        |

| libxml-ruby                                    | 3.0.0        |

| little-plugger                                 | 1.1.4        |

| logging                                        | 2.2.2        |

| memoist                                        | 0.15.0       |

| mime-types                                     | 3.1          |

| mime-types-data                                | 3.2016.0521  |

| mini_magick                                    | 4.5.1        |

| minitest                                       | 5.4.3        |

| multi_json                                     | 1.12.1       |

| multi_xml                                      | 0.6.0        |

| multipart-post                                 | 2.0.0        |

| nanaimo                                        | 0.2.4        |

| naturally                                      | 2.1.0        |

| nokogiri                                       | 1.5.6        |

| os                                             | 0.9.6        |

| plist                                          | 3.2.0        |

| power_assert                                   | 0.2.2        |

| psych                                          | 2.0.8        |

| public_suffix                                  | 2.0.5        |

| rake                                           | 10.4.2       |

| rdoc                                           | 4.2.0        |

| representable                                  | 3.0.4        |

| retriable                                      | 2.1.0        |

| rouge                                          | 1.11.1       |

| rubygems-update                                | 2.5.2        |

| rubyzip                                        | 1.2.1        |

| security                                       | 0.1.3        |

| signet                                         | 0.7.3        |

| simctl                                         | 1.6.2        |

| slack-notifier                                 | 2.3.2        |

| sqlite3                                        | 1.3.13       |

| terminal-notifier                              | 1.7.1        |

| terminal-table                                 | 1.7.3        |

| test-unit                                      | 3.0.8        |

| tty-cursor                                     | 0.5.0        |

| tty-screen                                     | 0.6.4        |

| tty-spinner                                    | 0.8.0        |

| uber                                           | 0.0.15       |

| unf                                            | 0.1.4        |

| unf_ext                                        | 0.0.7.4      |

| unicode-display_width                          | 1.1.3        |

| word_wrap                                      | 1.0.0        |

| xcodeproj                                      | 1.5.7        |

| xcpretty                                       | 0.2.6        |

| xcpretty-travis-formatter                      | 0.0.4        |

| xctest_list                                    | 1.1.4        |



generated on: 2018-09-27

Spotted: changes in build output

I've noticed that the generated .xctestrun file contains some changes when we build-without-testing in Xcode 10:

EnvironmentVariables no longer contains DYLD_FRAMEWORK_PATH, DYLD_LIBRARY_PATH, __XCODE_BUILT_PRODUCTS_DIR_PATHS, __XPC_DYLD_FRAMEWORK_PATH or __XPC_DYLD_LIBRARY_PATH entries
Under TestingEnvironmentVariables, entries DYLD_FRAMEWORK_PATH and DYLD_LIBRARY_PATH have changed from a very long string:

__TESTROOT__/Debug-iphonesimulator:__PLATFORMS__/iPhoneSimulator.platform/Developer/Library/Frameworks:__PLATFORMS__/iPhoneOS.platform/Developer/Library/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/Developer/Library/PrivateFrameworks

to
__TESTROOT__/Debug-iphonesimulator

There is a new section in the file:

        <key>__XCODE_BUILT_PRODUCTS_DIR_PATHS</key>
        <string>__TESTROOT__/Debug-iphonesimulator</string>
        <key>__XPC_DYLD_FRAMEWORK_PATH</key>
        <string>__TESTROOT__/Debug-iphonesimulator</string>
        <key>__XPC_DYLD_LIBRARY_PATH</key>
        <string>__TESTROOT__/Debug-iphonesimulator</string>

♻️ multi_scan 🐞bug

Source

craigomac

👍4

Most helpful comment

Fixed in v3.5.8

lyndsey-ferguson on 15 Oct 2018

👍2 🎉1 😄1

All 80 comments

@craigomac can you provide a sample project that demonstrates the issue? When I run multi_scan with my test project, it finds both the UI and standard unit tests. I tried upgrading the project file format to Xcode 10 and multi_scan still has no problem.

The code to find where the .xctest is fairly simple.

Two questions:

Is the .xctestrun file full of SkipTestIdentifiers?
Could you run Xcode's build-without-testing to create a new xctestrun file and the test artifacts. Then add a new fastlane:

lane :issue_74 do
   puts tests_from_xctestrun(xctestrun: 'path/to/the/file.xctestrun)
end

Run it and post the output?

lyndsey-ferguson on 28 Sep 2018

@lyndsey-ferguson thanks for fast response! I have the same problem as @craigomac and I can provide my answer:

Yes, I have 12 skipping test cases in .xctestrun
Here is my output:
`[17:16:28]: ----------------------------------------
+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | tests_from_xctestrun | 0 |
+------+----------------------------+-------------+

LippyFlo on 28 Sep 2018

@LippyFlo or @craigomac can either of you provide a sample .xctestrun file and the accompanying folders? Looking at @craigomac's text, the problem is probably __PLATFORMS__ and I'm not seeing that in my test project.

Unless...are your Xcode projects building iOS and Mac targets and maybe that is why it is showing up?

lyndsey-ferguson on 28 Sep 2018

Oh, and @LippyFlo, you have 12 skipping test cases in the xctestrun file, but that is not all the tests, right?

lyndsey-ferguson on 28 Sep 2018

@craigomac sorry for late answer. Of course I have another tests! I tested scan with generated .xctestrun and it works good. But multi_scan still cannot find any tests.

LippyFlo on 1 Oct 2018

I don't think I can provide the build folder, but here's the .xctestrun:
PipelineUITests_iphonesimulator12.0-x86_64.xctestrun.zip

We don't have iOS and MacOS targets but we do have two separate targets. The target we want to test is identified correctly in the UITargetAppPath key.

The minimum steps to reproduce involve the following Fastlane lane definition:

  def multitest(scheme_to_test, options)
    output_directory = options[:output_directory]
    multi_scan(
      scheme: scheme_to_test,
      devices: options[:devices].split(':'),
      output_directory: output_directory,
      derived_data_path: options[:derived_data_path],
      test_without_building: true
    )
  end

and running these two commands from Terminal:

xcodebuild build-for-testing -workspace 'Marktplaats.xcworkspace' -scheme 'PipelineUITests' -destination 'platform=iOS Simulator,name=iPhone 8,OS=12.0' -derivedDataPath 'build/ui_tests_12.0'

fastlane uitest devices:'iPhone 8' output_directory:'build/ui_tests_12.0/results' derived_data_path:'build/ui_tests_12.0'

Here's my complete output from fastlane issue_74

+------------------------+---------+------------------------+
|                       Used plugins                        |
+------------------------+---------+------------------------+
| Plugin                 | Version | Action                 |
+------------------------+---------+------------------------+
| fastlane-plugin-setup  | 2.1.1   | setup_fragile_tests_f  |
| _fragile_tests_for_re  |         | or_rescan              |
| scan                   |         |                        |
| fastlane-plugin-test_  | 3.5.7   | suppressed_tests       |
| center                 |         | suppress_tests         |
|                        |         | suppress_tests_from_j  |
|                        |         | unit                   |
|                        |         | tests_from_xctestrun   |
|                        |         | collate_test_result_b  |
|                        |         | undles                 |
|                        |         | tests_from_junit       |
|                        |         | multi_scan             |
|                        |         | collate_html_reports   |
|                        |         | collate_junit_reports  |
|                        |         | collate_json_reports   |
+------------------------+---------+------------------------+

[12:59:21]: ----------------------------------------
[12:59:21]: --- Step: Verifying fastlane version ---
[12:59:21]: ----------------------------------------
[12:59:21]: Your fastlane version 2.105.2 matches the minimum requirement of 2.60.1  ✅
[12:59:21]: ------------------------------
[12:59:21]: --- Step: default_platform ---
[12:59:21]: ------------------------------
[12:59:21]: Driving the lane 'ios issue_74' 🚀
[12:59:21]: -----------------------------
[12:59:21]: --- Step: create_keychain ---
[12:59:21]: -----------------------------
[12:59:21]: $ security list-keychains -d user
[12:59:21]: ▸ "/Users/crmcmahon/Library/Keychains/login.keychain-db"
[12:59:21]: ----------------------------------
[12:59:21]: --- Step: tests_from_xctestrun ---
[12:59:21]: ----------------------------------
[12:59:21]: {"UITests"=>[]}
[12:59:21]: -----------------------------
[12:59:21]: --- Step: delete_keychain ---
[12:59:21]: -----------------------------

+------+----------------------+-------------+
|             fastlane summary              |
+------+----------------------+-------------+
| Step | Action               | Time (in s) |
+------+----------------------+-------------+
| 1    | Verifying fastlane   | 0           |
|      | version              |             |
| 2    | default_platform     | 0           |
| 3    | create_keychain      | 0           |
| 4    | tests_from_xctestrun | 0           |
| 5    | delete_keychain      | 0           |
+------+----------------------+-------------+

[12:59:21]: fastlane.tools finished successfully 🎉

craigomac on 1 Oct 2018

@craigomac same thing! We have multiple schemes for test, maybe here is something wrong?

LippyFlo on 1 Oct 2018

Thanks, I'll take a look at this soon. It could be the case that there are multiple schemes, but I'm not sure. I think (I have to look at the code) I look at the testrun file that has the scheme name as the prefix.

lyndsey-ferguson on 1 Oct 2018

From what I understand @craigomac, the tests should be coming from Debug-iphonesimulator/UITests-Runner.app/PlugIns/UITests.xctest in your Products folder. Can you change your lane to also include:

at the top of the file:

require 'xctest_list'
# xctest_list is brought in by `fastlane-plugin-test_center`

and in your lane:

  puts XCTestList.tests(File.join('path/to/derived_data/Build/Products', 'Debug-iphonesimulator/UITests-Runner.app/PlugIns/UITests.xctest'))

And send me the output?

Also, are there any other xctest bundles in PlugIns, or the Build/Products/ directories or subdirectories?

lyndsey-ferguson on 2 Oct 2018

@lyndsey-ferguson -- I am seeing this exact same issue as well. Since I upgraded my build server to Xcode 10, the multi_scan UI test with batches always says 0 tests found. Attached is the .xctestrun file.

Also, with Xcode 10 now we can run on multiple simulators and the test suite is divided among the number of simulators launched. Is this feature supported within multi_scan or is it yet to be made available? Please clarify.

Sandhu412-zz on 8 Oct 2018

ToolkitUITests_iphonesimulator12.0-x86_64.xctestrun.zip

Sandhu412-zz on 8 Oct 2018

@lyndsey-ferguson -- attached is the .xctestrun file for the output as per your suggestion:ToolkitUITests_iphonesimulator12.0-x86_64.xctestrun.zip

Below is the output from "fastlane env"

🚫 fastlane environment 🚫

Stack

| Key | Value |
| --------------------------- | ---------------------------------------------- |
| OS | 10.13.6 |
| Ruby | 2.3.3 |
| Bundler? | false |
| Git | git version 2.17.1 (Apple Git-112) |
| Installation Source | /usr/local/bin/fastlane |
| Host | Mac OS X 10.13.6 (17G65) |
| Ruby Lib Dir | /usr/local/Cellar/ruby/2.3.3/lib |
| OpenSSL Version | OpenSSL 1.0.2j 26 Sep 2016 |
| Is contained | false |
| Is homebrew | false |
| Is installed via Fabric.app | false |
| Xcode Path | /Applications/Xcode_10.app/Contents/Developer/ |
| Xcode Version | 10.0 |

System Locale

| Variable | Value | |
| -------- | ----------- | - |
| LANG | en_US.UTF-8 | ✅ |
| LC_ALL | | |
| LANGUAGE | | |

fastlane files:

./fastlane/Fastfile

# Customise this file, documentation can be found here:
# https://github.com/fastlane/fastlane/tree/master/fastlane/docs
# All available actions: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Actions.md
# can also be listed using the `fastlane actions` command

# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`

# If you want to automatically update fastlane if a new version is available:
# update_fastlane

# This is the minimum version number required.
# Update this, if you use features of a newer version
fastlane_version "2.30.0"

default_platform :ios

require 'xctest_list'
# xctest_list is brought in by `fastlane-plugin-test_center`

platform :ios do
  before_all do
    # ENV["SLACK_URL"] = "https://hooks.slack.com/services/..."
    # this folder contains the Unit Test Reports from each build.
    system "rm -rf ../test_reports/"
  end

  after_all do
  end

  # version_string returns the verion string for the build.
  # Ex:v1.0.1(15) v1.0.1 is the version number and
  # (15) is the build number
  def version_string(version_number, build_number)
    "#{version_number} (#{build_number})"
  end

  # change_log_since_last_tag retuns the list of changes commited to git since
  # the last tag
  def change_log_since_last_tag
    # http://git-scm.com/docs/pretty-formats
    # <short hash> <commit title>
    return changelog_from_git_commits(
        tag_match_pattern: 'gm/*',
        pretty: '* (%h) [%aN] %s', # added formatting for release notes from: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Jenkins.md
      #  merge_commit_filtering: 'exclude_merges'.
      )[0..500]
  end

  lane :test do
    change_log_since_last_tag

#    version_number = get_version_number
#    build_number = get_build_number
#    ipa_filename = "ConstantContact-#{version_number}-#{build_number}"
#    upload_symbols_to_crashlytics(
#      api_token: "27f32e9229b459170cac7b7ae321d5bb29762f1b",
#      dsym_path: "archives/#{ipa_filename}.app.dSYM.zip",
#      binary_path: "./Pods/Fabric/upload-symbols"
#    )

  end

  desc "Runs all the TK UI Tests"
  lane :toolkit_ui_tests do
    ENV['FASTLANE_XCODE_LIST_TIMEOUT'] = '120'
    clear_derived_data
    sh('mkdir -p ../reports/uitest-coverage/')
    sh('mkdir -p ../reports/uitest-xcov-coverage/')
    cocoapods(repo_update: false, silent: true) # CP repo update moved to a Jenkins task that runs once a day
    multi_scan(
      workspace: File.absolute_path('../Toolkit.xcworkspace'),
      device: "iPhone 8",
      scheme: 'ToolkitUITests',
      xcargs: "-UseNewBuildSystem=NO",
      output_directory: "reports/ui-tests",
      include_simulator_logs: false,
      try_count: 2, # runs only the failed tests again without building to the set count
      batch_count: 3, # batches the total number of tests into set count to reduce the load on simulator
      fail_build: true,
      code_coverage: true
    )
    slather(
      scheme: "ToolkitUITests",
      proj: "Toolkit.xcodeproj",
      workspace: "Toolkit.xcworkspace",
      output_directory: "reports/uitest-coverage",
      binary_basename: "Toolkit.Dev",
      source_files: "EMM/ToolkitSwift/**/*.swift",
      cobertura_xml: true,
      jenkins: true
    )
    xcov(
      workspace: "Toolkit.xcworkspace",
      scheme: "ToolkitUITests",
      output_directory: "reports/uitest-xcov-coverage/",
      html_report: true,
      configuration: 'debug'
    )
    puts XCTestList.tests(File.join('path/to/derived_data/Build/Products', 'Debug-iphonesimulator/UITests-Runner.app/PlugIns/UITests.xctest'))
  end



# More information about multiple platforms in fastlane: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Platforms.md
# All available actions: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Actions.md

# fastlane reports which actions are used
# No personal data is recorded. Learn more at https://github.com/fastlane/enhancer

./fastlane/Appfile

# you can even provide different app identifiers, Apple IDs and team names per lane:
# More information: https://github.com/fastlane/fastlane/blob/master/fastlane/docs/Appfile.md

fastlane gems

| Gem | Version | Update-Status |
| -------- | ------- | ------------------ |
| fastlane | 2.97.0 | 🚫 Update available |

Loaded fastlane plugins:

| Plugin | Version | Update-Status |
| --------------------------- | ------- | ------------------ |
| fastlane-plugin-test_center | 3.2.2 | 🚫 Update available |

Loaded gems

| Gem | Version |
| --------------------------- | ------------ |
| did_you_mean | 1.0.0 |
| slack-notifier | 2.3.2 |
| multipart-post | 2.0.0 |
| word_wrap | 1.0.0 |
| tty-cursor | 0.5.0 |
| tty-spinner | 0.8.0 |
| babosa | 1.0.2 |
| colored | 1.2 |
| commander-fastlane | 4.4.6 |
| faraday-cookie_jar | 0.0.6 |
| mini_magick | 4.5.1 |
| rubyzip | 1.2.1 |
| security | 0.1.3 |
| declarative-option | 0.1.0 |
| representable | 3.0.4 |
| mime-types-data | 3.2016.0521 |
| mime-types | 3.1 |
| google-api-client | 0.21.2 |
| emoji_regex | 0.1.1 |
| tty-screen | 0.6.4 |
| highline | 1.7.10 |
| io-console | 0.4.5 |
| excon | 0.62.0 |
| CFPropertyList | 3.0.0 |
| mini_portile2 | 2.3.0 |
| nokogiri | 1.8.2 |
| plist | 3.4.0 |
| atomos | 0.1.2 |
| claide | 1.0.2 |
| colored2 | 3.1.2 |
| nanaimo | 0.2.5 |
| xcodeproj | 1.5.9 |
| faraday | 0.15.2 |
| faraday_middleware | 0.12.2 |
| http-cookie | 1.0.3 |
| unf | 0.1.4 |
| domain_name | 0.5.20180417 |
| os | 0.9.6 |
| googleauth | 0.6.2 |
| signet | 0.8.1 |
| fastimage | 2.1.3 |
| json | 2.1.0 |
| bundler | 1.16.1 |
| xctest_list | 1.1.3 |
| fastlane-plugin-test_center | 3.2.2 |

generated on: 2018-10-11

Below is the output from running the lane: "fastlane toolkit_ui_tests" As you can see from the output, for each batch it says "number of tests" as 0.

```INFO [2018-10-11 14:00:42.82]: Running Tests: ▸ Touching ToolkitUITests.xctest
INFO [2018-10-11 14:00:42.84]: ▸ Signing /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Build/Products/Debug-iphonesimulator/ToolkitUITests-Runner.app/PlugIns/ToolkitUITests.xctest
INFO [2018-10-11 14:00:44.11]: ▸ Signing /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Build/Products/Debug-iphonesimulator/ToolkitUITests-Runner.app
INFO [2018-10-11 14:00:45.72]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-11 14:00:46.73]: ------------------------------------------------------
INFO [2018-10-11 14:00:46.73]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-11 14:00:46.73]: ------------------------------------------------------
INFO [2018-10-11 14:00:47.88]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:00:50.89]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:00:55.20]: Fetching available simulator devices
INFO [2018-10-11 14:00:55.99]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:00:59.00]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:01:02.94]: Fetching available simulator devices

INFO [2018-10-11 14:01:03.65]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -UseNewBuildSystem=NO -only-testing:ToolkitUITests/CCTClearcutLogEvent/testCodes -only-testing:ToolkitUITests/CCTClearcutLogEvent/testID test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181011-43967-wxcrjw'
INFO [2018-10-11 14:01:03.65]: ▸ Loading...
INFO [2018-10-11 14:01:06.27]: ▸ 2018-10-11 14:01:06.271 xcodebuild[50954:5783978] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:01:06.41]: ▸ No devices are booted.
INFO [2018-10-11 14:01:06.48]: ▸ 2018-10-11 14:01:06.480 xcodebuild[50946:5783757] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-11 14:01:06.48]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.11_14-01-05--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-B674FD20-FB77-459D-AA07-2AE87443B2D8/ToolkitUITests-EC747793-9454-44ED-915B-43A2072649BA/Session-ToolkitUITests-2018-10-11_140106-wNALJM.log
INFO [2018-10-11 14:01:06.48]: ▸ 2018-10-11 14:01:06.480 xcodebuild[50946:5783721] [MT] IDETestOperationsObserverDebug: (0AFF23A8-4C84-46B2-8FA4-90327F6ACC9B) Beginning test session ToolkitUITests-0AFF23A8-4C84-46B2-8FA4-90327F6ACC9B at 2018-10-11 14:01:06.480 with Xcode 10A255 on target {
INFO [2018-10-11 14:01:06.48]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-11 14:01:06.48]: ▸ } (12.0 (16A366))
INFO [2018-10-11 14:01:45.15]: ▸ Selected tests
INFO [2018-10-11 14:01:45.15]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-11 14:01:45.16]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
INFO [2018-10-11 14:01:45.16]: ▸
INFO [2018-10-11 14:01:45.47]: ▸ 2018-10-11 14:01:45.471 xcodebuild[50946:5783721] [MT] IDETestOperationsObserverDebug: 39.048 elapsed -- Testing started completed.
INFO [2018-10-11 14:01:45.47]: ▸ 2018-10-11 14:01:45.471 xcodebuild[50946:5783721] [MT] IDETestOperationsObserverDebug: 0.001 sec, +0.001 sec -- start
INFO [2018-10-11 14:01:45.47]: ▸ 2018-10-11 14:01:45.471 xcodebuild[50946:5783721] [MT] IDETestOperationsObserverDebug: 39.048 sec, +39.047 sec -- end
INFO [2018-10-11 14:01:45.96]: ▸ 2018-10-11 14:01:45.968 xcodebuild[51093:5788239] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:01:49.67]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-11 14:01:49.93]: ------------------------------------------------------
INFO [2018-10-11 14:01:49.93]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-11 14:01:49.93]: ------------------------------------------------------
INFO [2018-10-11 14:01:54.21]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:01:57.23]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:02:01.04]: Fetching available simulator devices
INFO [2018-10-11 14:02:01.76]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:02:04.77]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:02:08.60]: Fetching available simulator devices

INFO [2018-10-11 14:02:09.23]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -UseNewBuildSystem=NO -only-testing:ToolkitUITests/FIRInstanceIDTokenOperation/testBlock -only-testing:ToolkitUITests/PHTPhenotypeDataFlag/testOverride test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-2.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-2.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181011-43967-oz5cj4'
INFO [2018-10-11 14:02:09.23]: ▸ Loading...
INFO [2018-10-11 14:02:11.48]: ▸ 2018-10-11 14:02:11.482 xcodebuild[51195:5790308] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:02:11.62]: ▸ No devices are booted.
INFO [2018-10-11 14:02:11.64]: ▸ 2018-10-11 14:02:11.646 xcodebuild[51186:5790265] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-11 14:02:11.64]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.11_14-02-11--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-93B33209-2B69-4D34-B60B-581C8D92281D/ToolkitUITests-9AF7E99D-45E2-4A98-8474-1B891CEC1C30/Session-ToolkitUITests-2018-10-11_140211-nbds1y.log
INFO [2018-10-11 14:02:11.64]: ▸ 2018-10-11 14:02:11.646 xcodebuild[51186:5790254] [MT] IDETestOperationsObserverDebug: (A50C373C-5ACB-4AEA-AFF6-C818CDDE2CDC) Beginning test session ToolkitUITests-A50C373C-5ACB-4AEA-AFF6-C818CDDE2CDC at 2018-10-11 14:02:11.646 with Xcode 10A255 on target {
INFO [2018-10-11 14:02:11.64]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-11 14:02:11.64]: ▸ } (12.0 (16A366))
INFO [2018-10-11 14:02:26.36]: ▸ Selected tests
INFO [2018-10-11 14:02:26.36]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-11 14:02:26.37]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
INFO [2018-10-11 14:02:26.37]: ▸
INFO [2018-10-11 14:02:26.66]: ▸ 2018-10-11 14:02:26.663 xcodebuild[51186:5790254] [MT] IDETestOperationsObserverDebug: 15.031 elapsed -- Testing started completed.
INFO [2018-10-11 14:02:26.66]: ▸ 2018-10-11 14:02:26.663 xcodebuild[51186:5790254] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
INFO [2018-10-11 14:02:26.66]: ▸ 2018-10-11 14:02:26.663 xcodebuild[51186:5790254] [MT] IDETestOperationsObserverDebug: 15.031 sec, +15.031 sec -- end
INFO [2018-10-11 14:02:27.04]: ▸ 2018-10-11 14:02:27.049 xcodebuild[51306:5791993] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:02:29.67]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-11 14:02:29.90]: ------------------------------------------------------
INFO [2018-10-11 14:02:29.90]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-11 14:02:29.90]: ------------------------------------------------------
INFO [2018-10-11 14:02:35.21]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:02:38.22]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:02:41.92]: Fetching available simulator devices
INFO [2018-10-11 14:02:42.63]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-11 14:02:45.64]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-11 14:02:49.28]: Fetching available simulator devices

INFO [2018-10-11 14:02:49.99]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -UseNewBuildSystem=NO -only-testing:ToolkitUITests/PHTPhenotypeNumberFlag/testOverride -only-testing:ToolkitUITests/PHTPhenotypeStringFlag/testOverride test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-3.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-3.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181011-43967-1aqt3xn'
INFO [2018-10-11 14:02:49.99]: ▸ Loading...
INFO [2018-10-11 14:02:52.25]: ▸ 2018-10-11 14:02:52.252 xcodebuild[51422:5793886] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:02:52.41]: ▸ No devices are booted.
INFO [2018-10-11 14:02:52.42]: ▸ 2018-10-11 14:02:52.427 xcodebuild[51411:5793869] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-11 14:02:52.42]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.11_14-02-51--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-919CB73A-EBF8-464E-A018-18E5C0E2FDC8/ToolkitUITests-0E16FB7C-1D9E-4494-B8BB-166D26DD0155/Session-ToolkitUITests-2018-10-11_140252-u64B88.log
INFO [2018-10-11 14:02:52.42]: ▸ 2018-10-11 14:02:52.427 xcodebuild[51411:5793767] [MT] IDETestOperationsObserverDebug: (D3A6D0BF-A555-4BED-9122-2E11E6F1ECFF) Beginning test session ToolkitUITests-D3A6D0BF-A555-4BED-9122-2E11E6F1ECFF at 2018-10-11 14:02:52.428 with Xcode 10A255 on target {
INFO [2018-10-11 14:02:52.42]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-11 14:02:52.42]: ▸ } (12.0 (16A366))
INFO [2018-10-11 14:03:07.37]: ▸ Selected tests
INFO [2018-10-11 14:03:07.37]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-11 14:03:07.37]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
INFO [2018-10-11 14:03:07.37]: ▸
INFO [2018-10-11 14:03:07.66]: ▸ 2018-10-11 14:03:07.662 xcodebuild[51411:5793767] [MT] IDETestOperationsObserverDebug: 15.246 elapsed -- Testing started completed.
INFO [2018-10-11 14:03:07.66]: ▸ 2018-10-11 14:03:07.662 xcodebuild[51411:5793767] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
INFO [2018-10-11 14:03:07.66]: ▸ 2018-10-11 14:03:07.662 xcodebuild[51411:5793767] [MT] IDETestOperationsObserverDebug: 15.246 sec, +15.246 sec -- end
INFO [2018-10-11 14:03:08.07]: ▸ 2018-10-11 14:03:08.075 xcodebuild[51534:5795667] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-11 14:03:10.72]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

+-------------------+---------------------------------------------------------------------------+
| multi_scan results |
+-------------------+---------------------------------------------------------------------------+
| result | true |
| total_tests | 0 |
| passing_testcount | 0 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 0 |
| report_files | ["/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.junit", |
| | "/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.html"] |
+-------------------+---------------------------------------------------------------------------+

INFO [2018-10-11 14:03:11.00]: ---------------------
INFO [2018-10-11 14:03:11.00]: --- Step: slather ---
INFO [2018-10-11 14:03:11.00]: ---------------------
INFO [2018-10-11 14:03:11.00]: gem 'slather' is already installed
INFO [2018-10-11 14:03:11.00]: $ slather coverage --jenkins --cobertura-xml --output-directory reports/uitest-coverage --scheme ToolkitUITests --workspace Toolkit.xcworkspace --binary-basename Toolkit.Dev --source-files EMM/ToolkitSwift/**/*.swift Toolkit.xcodeproj
INFO [2018-10-11 14:03:12.95]: ▸ Slathering...
INFO [2018-10-11 14:03:18.93]: ▸ Slathered
INFO [2018-10-11 14:03:18.95]: ------------------
INFO [2018-10-11 14:03:18.95]: --- Step: xcov ---
INFO [2018-10-11 14:03:18.95]: ------------------
INFO [2018-10-11 14:03:19.06]: gem 'xcov' is already installed
INFO [2018-10-11 14:03:19.14]: Skipping file blacklisting as no ignore file was found at path ./fastlane/.xcovignore

INFO [2018-10-11 14:03:19.15]: $ xcodebuild -showBuildSettings -workspace Toolkit.xcworkspace -scheme ToolkitUITests -configuration debug
WARN [2018-10-11 14:03:22.16]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
INFO [2018-10-11 14:03:25.82]: Unable to find any .xccoverage file.
INFO [2018-10-11 14:03:25.82]: Make sure you have enabled 'Gather code coverage' setting on your scheme settings.
INFO [2018-10-11 14:03:25.82]: Alternatively you can provide the full path to your .xccoverage file.
WARN [2018-10-11 14:03:25.82]: Lane Context:
INFO [2018-10-11 14:03:25.82]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios toolkit_ui_tests", :SCAN_DERIVED_DATA_PATH=>"/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk", :SCAN_GENERATED_PLIST_FILES=>[], :SCAN_GENERATED_PLIST_FILE=>nil}
ERROR [2018-10-11 14:03:25.82]: Error creating your coverage report - see the log above
INFO [2018-10-11 14:03:25.83]: Successfully generated documentation at path '/Users/sanjaneya/Documents/toolkit-ios/fastlane/README.md'

Sandhu412-zz on 11 Oct 2018

@Sandhu412 the step to generate the coverage report is causing an error, and I think that is why I do not see the list of tests from the XCTestList.tests call being printed.

Could you comment out the call, and add this above the call to the XCTestList.tests call?

puts '-' * 80

lyndsey-ferguson on 12 Oct 2018

@lyndsey-ferguson -- I commented out the code coverage and the lane goes through fine. Below is the output from fastlane: But below is what is printed from the puts XCTestList.tests command:

CCTClearcutLogEvent/testCodes CCTClearcutLogEvent/testID FIRInstanceIDTokenOperation/testBlock PHTPhenotypeDataFlag/testOverride PHTPhenotypeNumberFlag/testOverride PHTPhenotypeStringFlag/testOverride

```INFO [2018-10-12 06:24:59.95]: ▸ Linking ToolkitUITests
INFO [2018-10-12 06:25:00.51]: ▸ Running script '[CP] Embed Pods Frameworks'
INFO [2018-10-12 06:25:31.31]: ▸ Running script '[CP] Copy Pods Resources'
INFO [2018-10-12 06:25:33.88]: Running Tests: ▸ Touching ToolkitUITests.xctest
INFO [2018-10-12 06:25:33.90]: ▸ Signing /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Build/Products/Debug-iphonesimulator/ToolkitUITests-Runner.app/PlugIns/ToolkitUITests.xctest
INFO [2018-10-12 06:25:35.29]: ▸ Signing /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Build/Products/Debug-iphonesimulator/ToolkitUITests-Runner.app
INFO [2018-10-12 06:25:36.69]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-12 06:25:37.14]: ------------------------------------------------------
INFO [2018-10-12 06:25:37.15]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-12 06:25:37.15]: ------------------------------------------------------
INFO [2018-10-12 06:25:38.29]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:25:41.31]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:25:45.05]: Fetching available simulator devices
INFO [2018-10-12 06:25:45.80]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:25:48.81]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:25:52.47]: Fetching available simulator devices

INFO [2018-10-12 06:25:53.07]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -only-testing:ToolkitUITests/CCTClearcutLogEvent/testCodes -only-testing:ToolkitUITests/CCTClearcutLogEvent/testID test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181012-2635-1g9r21i'
INFO [2018-10-12 06:25:53.07]: ▸ Loading...
INFO [2018-10-12 06:25:55.10]: ▸ 2018-10-12 06:25:55.099 xcodebuild[8681:6444036] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:25:55.23]: ▸ No devices are booted.
INFO [2018-10-12 06:25:55.25]: ▸ 2018-10-12 06:25:55.257 xcodebuild[8672:6443797] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-12 06:25:55.25]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.12_06-25-54--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-1662712F-0973-47D4-86FA-16686F4D7F81/ToolkitUITests-F4FA1357-7D3B-4E5C-B4F4-C0BBEA3D0DC0/Session-ToolkitUITests-2018-10-12_062555-6rwiLa.log
INFO [2018-10-12 06:25:55.25]: ▸ 2018-10-12 06:25:55.257 xcodebuild[8672:6443790] [MT] IDETestOperationsObserverDebug: (41311F28-B852-467C-AC3C-2413AA3388A3) Beginning test session ToolkitUITests-41311F28-B852-467C-AC3C-2413AA3388A3 at 2018-10-12 06:25:55.257 with Xcode 10A255 on target {
INFO [2018-10-12 06:25:55.25]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-12 06:25:55.25]: ▸ } (12.0 (16A366))
INFO [2018-10-12 06:26:26.05]: ▸ Selected tests
INFO [2018-10-12 06:26:26.05]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-12 06:26:26.05]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
INFO [2018-10-12 06:26:26.05]: ▸
INFO [2018-10-12 06:26:26.35]: ▸ 2018-10-12 06:26:26.355 xcodebuild[8672:6443790] [MT] IDETestOperationsObserverDebug: 31.118 elapsed -- Testing started completed.
INFO [2018-10-12 06:26:26.35]: ▸ 2018-10-12 06:26:26.356 xcodebuild[8672:6443790] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
INFO [2018-10-12 06:26:26.35]: ▸ 2018-10-12 06:26:26.356 xcodebuild[8672:6443790] [MT] IDETestOperationsObserverDebug: 31.118 sec, +31.118 sec -- end
INFO [2018-10-12 06:26:26.72]: ▸ 2018-10-12 06:26:26.725 xcodebuild[8815:6445939] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:26:29.73]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-12 06:26:29.88]: ------------------------------------------------------
INFO [2018-10-12 06:26:29.88]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-12 06:26:29.88]: ------------------------------------------------------
INFO [2018-10-12 06:26:35.12]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:26:38.13]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:26:41.25]: Fetching available simulator devices
INFO [2018-10-12 06:26:41.84]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:26:44.85]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:26:48.11]: Fetching available simulator devices

INFO [2018-10-12 06:26:48.69]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -only-testing:ToolkitUITests/FIRInstanceIDTokenOperation/testBlock -only-testing:ToolkitUITests/PHTPhenotypeDataFlag/testOverride test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-2.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-2.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181012-2635-19jbr65'
INFO [2018-10-12 06:26:48.69]: ▸ Loading...
INFO [2018-10-12 06:26:50.88]: ▸ 2018-10-12 06:26:50.882 xcodebuild[8924:6447433] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:26:51.00]: ▸ No devices are booted.
INFO [2018-10-12 06:26:51.02]: ▸ 2018-10-12 06:26:51.020 xcodebuild[8916:6447341] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-12 06:26:51.02]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.12_06-26-50--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-5081C4FF-9C6D-4D1A-90D6-D744E6DE301E/ToolkitUITests-383C9F73-02F6-4F50-9CB5-189D67279540/Session-ToolkitUITests-2018-10-12_062651-fd6f1P.log
INFO [2018-10-12 06:26:51.02]: ▸ 2018-10-12 06:26:51.020 xcodebuild[8916:6447337] [MT] IDETestOperationsObserverDebug: (3EC670EB-6A25-411E-98CF-BA6F6C74D393) Beginning test session ToolkitUITests-3EC670EB-6A25-411E-98CF-BA6F6C74D393 at 2018-10-12 06:26:51.020 with Xcode 10A255 on target {
INFO [2018-10-12 06:26:51.02]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-12 06:26:51.02]: ▸ } (12.0 (16A366))
INFO [2018-10-12 06:27:06.71]: ▸ Selected tests
INFO [2018-10-12 06:27:06.71]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-12 06:27:06.71]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
INFO [2018-10-12 06:27:06.71]: ▸
INFO [2018-10-12 06:27:07.00]: ▸ 2018-10-12 06:27:07.006 xcodebuild[8916:6447337] [MT] IDETestOperationsObserverDebug: 15.997 elapsed -- Testing started completed.
INFO [2018-10-12 06:27:07.00]: ▸ 2018-10-12 06:27:07.006 xcodebuild[8916:6447337] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
INFO [2018-10-12 06:27:07.00]: ▸ 2018-10-12 06:27:07.006 xcodebuild[8916:6447337] [MT] IDETestOperationsObserverDebug: 15.997 sec, +15.997 sec -- end
INFO [2018-10-12 06:27:07.35]: ▸ 2018-10-12 06:27:07.351 xcodebuild[9038:6448854] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:27:09.98]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-12 06:27:10.15]: ------------------------------------------------------
INFO [2018-10-12 06:27:10.15]: --- Starting test run on testable 'ToolkitUITests' ---
INFO [2018-10-12 06:27:10.15]: ------------------------------------------------------
INFO [2018-10-12 06:27:15.44]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:27:18.45]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:27:21.46]: Fetching available simulator devices
INFO [2018-10-12 06:27:22.03]: $ xcodebuild -showBuildSettings -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests
WARN [2018-10-12 06:27:25.04]: Command timed out after 3 seconds on try 1 of 4, trying again with a 6 second timeout...
DEBUG [2018-10-12 06:27:28.48]: Fetching available simulator devices

INFO [2018-10-12 06:27:29.03]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/sanjaneya/Documents/toolkit-ios/Toolkit.xcworkspace -scheme ToolkitUITests -destination 'platform=iOS Simulator,id=65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA' -derivedDataPath '/Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk' -enableCodeCoverage YES -only-testing:ToolkitUITests/PHTPhenotypeNumberFlag/testOverride -only-testing:ToolkitUITests/PHTPhenotypeStringFlag/testOverride test-without-building | tee '/Users/sanjaneya/Library/Logs/scan/Toolkit.Dev-ToolkitUITests.log' | xcpretty --report html --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-3.html' --report junit --output '/Users/sanjaneya/Documents/toolkit-ios/reports/ui-tests/report-3.junit' --report junit --output '/var/folders/nc/x2sh_3t17692n9jf3pybxt2n8c1sfm/T/junit_report20181012-2635-378fih'
INFO [2018-10-12 06:27:29.03]: ▸ Loading...
INFO [2018-10-12 06:27:30.96]: ▸ 2018-10-12 06:27:30.962 xcodebuild[9149:6450386] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:27:31.15]: ▸ No devices are booted.
INFO [2018-10-12 06:27:31.16]: ▸ 2018-10-12 06:27:31.168 xcodebuild[9141:6450322] IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
INFO [2018-10-12 06:27:31.16]: ▸ /Users/sanjaneya/Library/Developer/Xcode/DerivedData/Toolkit-cbjnzqipycngqehjhegjjguoghlk/Logs/Test/Test-ToolkitUITests-2018.10.12_06-27-30--0400.xcresult/1_Test/Diagnostics/ToolkitUITests-FE053E5A-C9B7-4454-8E17-276D3D9E444A/ToolkitUITests-85972CE0-CBC0-4AF7-99DE-8077DFCDF6C1/Session-ToolkitUITests-2018-10-12_062731-2ZuqAI.log
INFO [2018-10-12 06:27:31.16]: ▸ 2018-10-12 06:27:31.169 xcodebuild[9141:6450311] [MT] IDETestOperationsObserverDebug: (0390BFB8-85BD-49CD-B582-BF1C61F5EEFC) Beginning test session ToolkitUITests-0390BFB8-85BD-49CD-B582-BF1C61F5EEFC at 2018-10-12 06:27:31.169 with Xcode 10A255 on target {
INFO [2018-10-12 06:27:31.17]: ▸ SimDevice: iPhone 8 (65B5C280-4F76-43A8-BE58-D9CC4EE7AAAA, iOS 12.0, Shutdown)
INFO [2018-10-12 06:27:31.17]: ▸ } (12.0 (16A366))
INFO [2018-10-12 06:27:46.75]: ▸ Selected tests
INFO [2018-10-12 06:27:46.75]: ▸ Test Suite ToolkitUITests.xctest started
INFO [2018-10-12 06:27:46.75]: ▸ Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
INFO [2018-10-12 06:27:46.75]: ▸
INFO [2018-10-12 06:27:47.05]: ▸ 2018-10-12 06:27:47.056 xcodebuild[9141:6450311] [MT] IDETestOperationsObserverDebug: 15.902 elapsed -- Testing started completed.
INFO [2018-10-12 06:27:47.05]: ▸ 2018-10-12 06:27:47.056 xcodebuild[9141:6450311] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
INFO [2018-10-12 06:27:47.05]: ▸ 2018-10-12 06:27:47.056 xcodebuild[9141:6450311] [MT] IDETestOperationsObserverDebug: 15.902 sec, +15.902 sec -- end
INFO [2018-10-12 06:27:47.46]: ▸ 2018-10-12 06:27:47.465 xcodebuild[9264:6451931] [MT] DVTDeveloperPaths: WARNING: Environment variable XCODE_DEVELOPER_DIR_PATH is set, but is not an absolute path. The variable will be ignored. ()
INFO [2018-10-12 06:27:50.02]: ▸ Test execute Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+

INFO [2018-10-12 06:27:50.27]: --------------------------------------------------------------------------------
INFO [2018-10-12 06:27:50.56]: CCTClearcutLogEvent/testCodes CCTClearcutLogEvent/testID FIRInstanceIDTokenOperation/testBlock PHTPhenotypeDataFlag/testOverride PHTPhenotypeNumberFlag/testOverride PHTPhenotypeStringFlag/testOverride
INFO [2018-10-12 06:27:50.56]: Successfully generated documentation at path '/Users/sanjaneya/Documents/toolkit-ios/fastlane/README.md'

+------+-------------------------------------------+-------------+
| fastlane summary |
+------+-------------------------------------------+-------------+
| Step | Action | Time (in s) |
+------+-------------------------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | mkdir -p ../reports/uitest-coverage/ | 0 |
| 4 | mkdir -p ../reports/uitest-xcov-coverage/ | 0 |
| 5 | cocoapods | 19 |
| 6 | multi_scan | 453 |
+------+-------------------------------------------+-------------+

+-------------+--------------+----------------+
| Plugin updates available |
+-------------+--------------+----------------+
| Plugin | Your Version | Latest Version |
+-------------+--------------+----------------+
| test_center | 3.2.2 | 3.5.7 |
+-------------+--------------+----------------+
INFO [2018-10-12 06:27:50.60]: To update all plugins, just run
INFO [2018-10-12 06:27:50.60]: $ bundle exec fastlane update_plugins

INFO [2018-10-12 06:27:50.60]: fastlane.tools just saved you 8 minutes! 🎉

Sandhu412-zz on 12 Oct 2018

Ok, so the tests are found, but when asked, xcodebuild does not test them. Are these Swift tests? Is it possible you could build a sample project the exhibits the same behavior and attach it to this issue?

lyndsey-ferguson on 12 Oct 2018

yes, there are swift tests. And how do you know the tests are found? The output does not list any tests? Regarding the sample project, I dont know how to do that? Could you guide me, also the repo is of my company so I am not sure if I can share it.

Sandhu412-zz on 12 Oct 2018

What I see is that these tests are found:

CCTClearcutLogEvent/testCodes
CCTClearcutLogEvent/testID
FIRInstanceIDTokenOperation/testBlock
PHTPhenotypeDataFlag/testOverride
PHTPhenotypeNumberFlag/testOverride
PHTPhenotypeStringFlag/testOverride

I'm guess that something is wrong with the way that they are passed to xcodebuild. You can see that they _will_ be passed here:

| only_testing           | ["ToolkitUITests/PHTPhenotypeNumberFlag/testOverride",                                    |
|                        | "ToolkitUITests/PHTPhenotypeStringFlag/testOverride"]

And that they are being sent to xcodebuild here:

set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace ... -only-testing:ToolkitUITests/PHTPhenotypeNumberFlag/testOverride -only-testing:ToolkitUITests/PHTPhenotypeStringFlag/testOverride

lyndsey-ferguson on 12 Oct 2018

Can you make a sample project using just the names of your tests as empty tests in the UI test target? If you run multi_scan on that project, does it have the same problem? If so, then you can attach that sample project to this issue?

lyndsey-ferguson on 12 Oct 2018

Those are not the UI tests in my project. I dont even know what they mean. That is why, I was confused when you said that tests are found. Did you expect to see the actual test names in that list?

I will try to get the sample project.

Sandhu412-zz on 12 Oct 2018

These:

CCTClearcutLogEvent/testCodes
CCTClearcutLogEvent/testID
FIRInstanceIDTokenOperation/testBlock
PHTPhenotypeDataFlag/testOverride
PHTPhenotypeNumberFlag/testOverride
PHTPhenotypeStringFlag/testOverride

are not tests in your target? They are being found in your xctest bundle. These are probably methods that are included into the binary in the xctest bundle. Are you including extra frameworks or libraries in your test target in Xcode?

Would you be able to show me what libraries and frameworks you are linking in? Perhaps those have those methods that match the test pattern that XCTestList is examining.

lyndsey-ferguson on 12 Oct 2018

NO, they are not my tests. Not sure why they are showing up. It was not an issue with Xcode 9 using multi_scan for a long time, so something specifically with Xcode 10 is the reason then. I dont have any other libraries/frameworks in test target.

Below are all the tests I have in my target.

screen shot 2018-10-12 at 11 51 58 am

Sandhu412-zz on 12 Oct 2018

Your test target does not link in Firebase? I definitely see that as a symbol that is in your test target's binary. I also see CCTClearcutLogEvent which could be from the Clearcut log uploader codebase.

Can you show me a screen shot of what frameworks are being included? Both in the Frameworks build phase and in the Frameworks command line?
screen shot 2018-10-12 at 12 00 06 pm

lyndsey-ferguson on 12 Oct 2018

@lyndsey-ferguson -- Below are the two screenshots from my project that you are referring to. The main target (Toolkit) uses many pods and frameworks as shown above. But I dont use anything specific for UI tests. Also, these are the same set of frameworks that I have used with Xcode9.X/multi_scan combination.

screen shot 2018-10-12 at 12 15 01 pm

Sandhu412-zz on 12 Oct 2018

Do you need to link in the Pods framework for unit tests? I suppose if you're not mocking out calls to APIs found in the Pods frameworks you would (although mocks and dependency injection would make your tests run faster...).

lyndsey-ferguson on 12 Oct 2018

So @Sandhu412, it seems that the problem that you are seeing is that the UI Swift tests are not being found. Is this a new version of Swift that you are using? It could be that the naming convention has changed.

lyndsey-ferguson on 12 Oct 2018

@lyndsey-ferguson I experienced this after moving to Xcode 10 AND upgrading to Swift 4.2.

lquigley on 12 Oct 2018

I think that the cause may be the new version of Swift. I am getting more reports that point to that as a cause.

lyndsey-ferguson on 12 Oct 2018

With Xcode 9.X we were using Swift 4 and the only change is we upgraded to Xcode 10.

Sandhu412-zz on 12 Oct 2018

@Sandhu412 you did not change from Swift 4.0 to 4.2?

lyndsey-ferguson on 12 Oct 2018

They did change the mangling rules

lyndsey-ferguson on 12 Oct 2018

No, because there a ton of deprecated errors that we need to fix in the project and as of now, we are able to compile and build with 4.0. We are planning to set aside some time to fix the errors with swift 4.2 but as of now, its still swift 4.0.

Sandhu412-zz on 12 Oct 2018

The project I see the problem in is using Swift 4. You did not see this as a problem with Xcode 9 and Swift 4? Was that even a thing? I don't use Swift and am mostly writing DevOps systems/code for Mobile.

lyndsey-ferguson on 12 Oct 2018

The project I see the problem in is using Swift 4. You did not see this as a problem with Xcode 9 and Swift 4?

No, I did not. Multi_scan was fine running on swift 4 until we upgraded to Xcode 10.

Sandhu412-zz on 12 Oct 2018

Perhaps the xcrun swift-demangle tool changed.

lyndsey-ferguson on 12 Oct 2018

Tne XCTestList.swift_tests method is now returning early on next unless /.*__\w+test\w*/ =~ line. Removing the __ will get to the xcrun swift-demangle line but it has changed and it fails this if if /.*-\[.*\]/ !~ line && /\w+\.(?<testclass>[^\.]+)\.(?<testmethod>test[^\(]+)\(/

thehouck on 12 Oct 2018

Ok, thanks. I think that the pattern has changed and it is no longer picking it up. I have an alternative.

lyndsey-ferguson on 15 Oct 2018

Fixed in v3.5.8

lyndsey-ferguson on 15 Oct 2018

👍2 🎉1 😄1

Thanks so much @lyndsey-ferguson !!! :+1:

Sandhu412-zz on 15 Oct 2018

Thank you for your help, it was essential!

lyndsey-ferguson on 15 Oct 2018

I'm still seeing this issue occur with multi-scan and Xcode 10.1. I have verified I'm using the newest version of fastlane and test_center. Scan will find and run the tests just fine but multi_scan always finds 0 tests. What information can I provide to see if this is still an issue?

hilljon86 on 6 Nov 2018

@hilljon86 -- I am still running Xcode 10 and it works fine for me with the test center version 3.58 which has the fix for this issue. There is nothing else to do other than using 3.58 plugin version. So based on your notes, I am not sure if it breaks again with Xcode 10.1?

Sandhu412-zz on 6 Nov 2018

@Sandhu412 I revered to Xcode 10 and I'm still having the same issue using the v3.5.9 plugin.

hilljon86 on 6 Nov 2018

Can you re-run your fastlane command with --capture_output and include that output (after removing sensitive data)?

lyndsey-ferguson on 6 Nov 2018

To summarize:

@hilljon86 is seeing an issue where multi_scan is calling scan with an array of tests in the only_testing parameter. Please confirm.
@Sandhu412 does not see this issue with v3.5.8. v3.5.9 should also work as the only change is how it discards metadata from the xctestrun file.

I need to see what version of xctest_list is being used. That is where the fix was ultimately made. Please include the log output from when you run bundle install in the project directory. It will tells us which version of that Gem is being used (it should be v1.1.6).

Also, could you attach a copy of your Gemfile.lock file?

lyndsey-ferguson on 8 Nov 2018

@lyndsey-ferguson sorry for the delay. Here is the output of bundle install and the Gemfile.lock. I tried running with the fast lane command with --capture_out but I am not sure where to find that output.

bundleInstallOutput.txt
gemfilelock.txt

hilljon86 on 9 Nov 2018

Okay, so you are using the xctest_list version that has the fix that I made for it to properly parse the new format for Swift tests.

lyndsey-ferguson on 9 Nov 2018

@hilljon86 I need the output to see what you're seeing and understand. The output is what is printed to the terminal window when you run fastlane to run the tests. Please attach that (as a text file please). Make sure you remove any sensitive data such as passwords, private keys, etc.

lyndsey-ferguson on 9 Nov 2018

@lyndsey-ferguson thank you for the help. Here is that output you requested. Please let me know if there is anything else I can provide.

multi_scan_output.txt

hilljon86 on 9 Nov 2018

I have added some debugging and one fix (the metadata in the xctestrun file was appearing as a "testable").

Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again?

Pluginfile:

gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-74-resurrected-no-tests-found"

Please attach the console output as a text file to this issue (makes it easier for me to review).

lyndsey-ferguson on 11 Nov 2018

@lyndsey-ferguson here is the output you requested after pointing to your test_center branch:
multi_scan_output_test_center_branch_issue-74-resurrected-no-tests-found.txt

hilljon86 on 12 Nov 2018

Hello @hilljon86, I think that it is not using the code at the branch. I see this:

There are no tests to run in testable '__xctestrun_metadata__'. Skipping

In the log, which I believe I fixed in this commit for that branch. I also see:

[07:11:06]: fastlane detected a Gemfile in the current directory
[07:11:06]: however it seems like you don't use `bundle exec`
[07:11:06]: to launch fastlane faster, please use
[07:11:06]: 
[07:11:06]: $ bundle exec fastlane acceptance_tests

Which means that you are probably executing fastlane directly like so: fastlane acceptance_tests.

Could you:

remove your Gemfile.lock file.
run bundle install
run bundle exec fastlane acceptance_tests --verbose --capture_output
and then attach the console log to this Issue?

Thanks for helping me resolve this issue!

lyndsey-ferguson on 12 Nov 2018

@lyndsey-ferguson ok, here is the latest output. It did remove the metadata as testable but still no tests found. Thanks for your help!
multi_scan_output_test_center_branch_issue-74-resurrected-no-tests-found_2nd_Attempt.txt

hilljon86 on 12 Nov 2018

Did you use the --verbose --capture_output flags? I do not see the output that _must_ be there to get even as far as it did.

lyndsey-ferguson on 12 Nov 2018

If so, my brain will need a moment to process how it could have gotten to where it did without outputting.

lyndsey-ferguson on 12 Nov 2018

Sorry, I did not... I am re-running now.

hilljon86 on 12 Nov 2018

👍1

I'm sorry, I feel like I'm missing something simple but when I run with those flags I get no output...

` ~/workspace/XXXXXXX master bundle exec fastlane acceptance_tests --verbose --capture_output
[✔] 🚀
~/workspace/XXXXXXX master bundle exec fastlane acceptance_tests --verbose --capture_output
[✔] 🚀

~/workspace/XXXXXXX master `

hilljon86 on 12 Nov 2018

That is odd. Just use --verbose. That will work.

lyndsey-ferguson on 12 Nov 2018

Ok, here is the newest output. It correctly identifies the tests to skip but does not find any tests to run.
multi_scan_output_test_center_branch_issue-74-resurrected-no-tests-found_3rd_Attempt.txt

hilljon86 on 12 Nov 2018

Ok, I've updated the code to print out more debug info. Can you remove your Gemfile.lock file, run bundle install, then run bundle exec fastlane acceptance_tests --verbose, and then provide the console output again?

lyndsey-ferguson on 12 Nov 2018

Ok, here is the latest update:

multi_scan_output_test_center_branch_issue-74-resurrected-no-tests-found_4th_Attempt.txt

hilljon86 on 12 Nov 2018

Can you run the following commands and provide me the output?

Objective C symbols for the test binary

nm -U '/Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app/PlugIns/XXXXXXXXUITests.xctest/XXXXXXXXUITests'

Swift symbols for the test binary

nm -gU /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app/PlugIns/XXXXXXXXUITests.xctest/XXXXXXXXUITests'

Oh, and which version of Swift are you using?

lyndsey-ferguson on 12 Nov 2018

Here is the output. The Swift symbols command returned almost nothing for some reason. And we are using Swift 4.

Objective C symbols for the test binary.txt
Swift symbols for the test binary.txt

hilljon86 on 12 Nov 2018

Ok, so the Swift tests are not in the binary, that is a problem. That explains why the tests are not found. From what I understand, you have a lot of Objective-C tests as well. I'll run the output through the system to see why they are not being listed.

lyndsey-ferguson on 12 Nov 2018

Our project is mixed swift/objc but all of our tests in this scheme are UITests that are written in swift.

hilljon86 on 12 Nov 2018

Ok. That's interesting.

lyndsey-ferguson on 12 Nov 2018

The ObjectiveC tests file that you provided, looks to me by eyeballing it, that it contains the tests.

S21XXXXXXXXUITests017XXXXXXXXCreateProfileB0C010testCreateD20PageViewableElementsyyFSbyKXKfu_ This seems to be a testsuite named XXXXXXXXCreateProfile, with the test testCreateD20PageViewableElements. Maybe....

lyndsey-ferguson on 12 Nov 2018

Just to confirm, the test-runner, "AcceptanceTesting-iphonesimulator" should be running the UI tests in Swift.

lyndsey-ferguson on 12 Nov 2018

We do have a test class named "XXXXCreateProfileUITests.swift" with a test named "testCreateProfilePageViewableElements". Can you elaborate on your last comment about the test-runner? I'm not exactly sure what you are asking or how I could verify that for you. Thank you for your patience.

hilljon86 on 12 Nov 2018

There is something odd going on. The output for:

nm -U '/Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app/PlugIns/XXXXXXXXUITests.xctest/XXXXXXXXUITests'

Should list the Objective-C symbols. Instead, you are seeing the names from your Swift test files. Are you using some sort of wrapper or conversion tool that "converts" the tests from Swift to something else?

lyndsey-ferguson on 12 Nov 2018

Disregard my comment about the test runner, I did not have a well-formed question and I'm not sure what my point was. 😊

lyndsey-ferguson on 12 Nov 2018

No, it should be a pretty basic set-up. It is an old project but what is interesting is that if I use scan instead of multi-scan, it finds the tests without issue. Would it be helpful to compare the output of scan to that of multi_scan?

hilljon86 on 12 Nov 2018

I’m happy to hear your clarification about the test runner comment, I was feeling pretty dumb. Thanks again for your help.

hilljon86 on 12 Nov 2018

Sure, send me the output from scan. Scan will work because it tells xcodebuild to handle everything. multi_scan needs to parse the tests in order to determine things.

At the very least, I will see the format of the tests and how they are being run. It could help me if you attach a junit file for a test run in scan.

lyndsey-ferguson on 12 Nov 2018

I'm going to create a new Issue for this.

lyndsey-ferguson on 12 Nov 2018

@hilljon86 please use #86 for this issue as it is a different problem than this original Issue.

lyndsey-ferguson on 12 Nov 2018

Hi, I recieve ```
+------------------+------------------------+

Natasha-Cole-iOS-Developer on 10 Oct 2019

Hm. Can you create a new Issue (that looks like a bug) and make sure you include the complete console log of running your lane with the --verbose flag?

lyndsey-ferguson on 10 Oct 2019

bundler: failed to load command: fastlane (/Users/distiller/project/vendor/bundle/ruby/2.6.0/bin/fastlane)
FastlaneCore::Interface::FastlaneError: [!] Error: cannot find xctestrun file ''
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane_core/lib/fastlane_core/ui/interface.rb:141:in `user_error!'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane_core/lib/fastlane_core/ui/ui.rb:17:in `method_missing'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/test_collector.rb:16:in `initialize'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:43:in `new'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:43:in `setup_testcollector'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:24:in `initialize'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/actions/multi_scan.rb:33:in `new'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/actions/multi_scan.rb:33:in `run'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:261:in `block (2 levels) in execute_action'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/actions/actions_helper.rb:50:in `execute_action'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:253:in `block in execute_action'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:227:in `chdir'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:227:in `execute_action'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:157:in `trigger_action_by_name'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/fast_file.rb:159:in `method_missing'
  Fastfile:56:in `block (2 levels) in parsing_binding'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/lane.rb:33:in `call'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:49:in `block in execute'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:45:in `chdir'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/runner.rb:45:in `execute'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/lane_manager.rb:47:in `cruise_lane'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/command_line_handler.rb:36:in `handle'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/commands_generator.rb:108:in `block (2 levels) in run'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in `call'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:in `run'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in `run_active_command'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:in `run!'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in `run!'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/commands_generator.rb:352:in `run'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/commands_generator.rb:41:in `start'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/fastlane/lib/fastlane/cli_tools_distributor.rb:119:in `take_off'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/gems/fastlane-2.148.1/bin/fastlane:23:in `<top (required)>'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/bin/fastlane:23:in `load'
  /Users/distiller/project/vendor/bundle/ruby/2.6.0/bin/fastlane:23:in `<top (required)>'

It ran fine for local but failed for circleci environment.

I set custom path for derived_data_path -> build folder which contains Build/Products/*.xctestrun file but it doesn't seem like be able to locate that file?

intoxicated on 28 May 2020

local

DEBUG [2020-05-27 16:56:54.14]: After building, found xctestrun files ["./build/Build/Products/app_iphonesimulator13.2-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-05-27 16:56:54.14]: Removing report files generated by the build
DEBUG [2020-05-27 16:56:54.14]:   ./tmp/output/scan/report.html
DEBUG [2020-05-27 16:56:54.14]:   ./tmp/output/scan/report.junit

DEBUG [2020-05-27 23:25:30.15]: After building, found xctestrun files [] (choosing 1st)
DEBUG [2020-05-27 23:25:30.15]: Removing report files generated by the build
DEBUG [2020-05-27 23:25:30.15]:   /tmp/output/scan/report.html
DEBUG [2020-05-27 23:25:30.15]:   /tmp/output/scan/report.junit

intoxicated on 28 May 2020

@intoxicated can you create a new issue for this? Please make sure you follow the instructions when creating the issue to help me debug the issue faster.

lyndsey-ferguson on 28 May 2020

Was this page helpful?

0 / 5 - 0 ratings