Fastlane-plugin-test_center: tests_from_xctestrun is not able to get tests from xctestrun's binary
@hilljon86 is experiencing this problem:
The action tests_from_xctestrun is unable to retrieve tests from the xctest bundle. It seems that when it tries to pull out the Objective-C tests, it gets the names of the Swift tests. It does not see any tests in the Swift symbols.
New Issue Checklist
- [x] Updated
fastlane-plugin-test_centerto the latest version - [x] I read the README.md
- [x] I reviewed the example(s) for the action(s) I am using
- [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
Issue Description
tests_from_xctestrun
Complete output when running fastlane, including the stack trace and command used
Objective.C.symbols.for.the.test.binary.txt
Swift.symbols.for.the.test.binary.txt
bundle exec fastlane acceptance_tests --verbose
[✔] 🚀
DEBUG [2018-11-12 10:21:13.07]: Checking if there are any plugins that should be loaded...
DEBUG [2018-11-12 10:21:13.07]: Loading 'fastlane-plugin-test_center' plugin
INFO [2018-11-12 10:21:13.07]: gem 'fastlane-plugin-test_center' is already installed
DEBUG [2018-11-12 10:21:13.09]: Loading 'fastlane-plugin-badge' plugin
INFO [2018-11-12 10:21:13.09]: gem 'fastlane-plugin-badge' is already installed
+------------------------+---------+------------------------+
| Used plugins |
+------------------------+---------+------------------------+
| Plugin | Version | Action |
+------------------------+---------+------------------------+
| fastlane-plugin-test_ | 3.5.9 | suppressed_tests |
| center | | suppress_tests |
| | | suppress_tests_from_j |
| | | unit |
| | | tests_from_xctestrun |
| | | collate_test_result_b |
| | | undles |
| | | tests_from_junit |
| | | multi_scan |
| | | collate_html_reports |
| | | collate_junit_reports |
| | | collate_json_reports |
| fastlane-plugin-badge | 1.1.0 | add_badge |
+------------------------+---------+------------------------+
Successfully loaded Appfile at path '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/Appfile'
- apple_id: 'XXXXXXXX'
INFO [2018-11-12 10:21:13.24]: ---------------------------
INFO [2018-11-12 10:21:13.24]: --- Step: opt_out_usage ---
INFO [2018-11-12 10:21:13.25]: ---------------------------
INFO [2018-11-12 10:21:13.25]: Disabled upload of used actions
INFO [2018-11-12 10:21:13.25]: ----------------------------------------
INFO [2018-11-12 10:21:13.25]: --- Step: Verifying fastlane version ---
INFO [2018-11-12 10:21:13.25]: ----------------------------------------
INFO [2018-11-12 10:21:13.25]: Your fastlane version 2.108.0 matches the minimum requirement of 2.108.0 ✅
INFO [2018-11-12 10:21:13.26]: ------------------------------
INFO [2018-11-12 10:21:13.26]: --- Step: default_platform ---
INFO [2018-11-12 10:21:13.26]: ------------------------------
INFO [2018-11-12 10:21:13.26]: gem 'nokogiri' is already installed
INFO [2018-11-12 10:21:13.26]: Driving the lane 'ios acceptance_tests' 🚀
INFO [2018-11-12 10:21:13.26]: --------------------------------
INFO [2018-11-12 10:21:13.26]: --- Step: clear_derived_data ---
INFO [2018-11-12 10:21:13.26]: --------------------------------
INFO [2018-11-12 10:21:13.34]: Derived Data path located at: /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData
INFO [2018-11-12 10:21:15.59]: Successfully cleared Derived Data ♻️
INFO [2018-11-12 10:21:15.59]: -----------------
INFO [2018-11-12 10:21:15.59]: --- Step: cd ---
INFO [2018-11-12 10:21:15.59]: -----------------
INFO [2018-11-12 10:21:15.59]: $ cd
INFO [2018-11-12 10:21:15.61]: -------------------------------
INFO [2018-11-12 10:21:15.61]: --- Step: update_info_plist ---
INFO [2018-11-12 10:21:15.61]: -------------------------------
INFO [2018-11-12 10:21:15.64]: Setting plist value for key: User to: XXXXXXXX
INFO [2018-11-12 10:21:15.64]: Updated XXXXXXXX.xcodeproj/../XXXXXXXXUITests/TestConfiguration.plist 💾.
INFO [2018-11-12 10:21:15.64]: running tests*
INFO [2018-11-12 10:21:15.64]: ------------------------
INFO [2018-11-12 10:21:15.64]: --- Step: multi_scan ---
INFO [2018-11-12 10:21:15.64]: ------------------------
+----------------------+----------------------+
| Summary for multi_scan (test_center v3.5.9) |
+----------------------+----------------------+
| try_count | 3 |
+----------------------+----------------------+
INFO [2018-11-12 10:21:15.66]: Successfully loaded '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| scheme | XXXXXXXX |
| clean | true |
| device | iPad Air 2 |
+----------------------+---------------------+
INFO [2018-11-12 10:21:15.73]: $ xcodebuild -showBuildSettings -workspace /Users/XXXXXXXX/workspace/XXXXXXXX/XXXXXXXX.xcworkspace -scheme AcceptanceTests
DEBUG [2018-11-12 10:21:18.33]: Fetching available simulator devices
DEBUG [2018-11-12 10:21:18.73]: Detected derived data path '/Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs'
INFO [2018-11-12 10:21:18.73]: Successfully loaded '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/Scanfile' 📄
+----------------------+---------------------+
| Detected Values from './fastlane/Scanfile' |
+----------------------+---------------------+
| scheme | XXXXXXXX |
| clean | true |
| device | iPad Air 2 |
+----------------------+---------------------+
INFO [2018-11-12 10:21:18.74]: $ xcodebuild -showBuildSettings -workspace /Users/XXXXXXXX/workspace/XXXXXXXX/XXXXXXXX.xcworkspace -scheme AcceptanceTests
DEBUG [2018-11-12 10:21:20.86]: Fetching available simulator devices
+----------------------------+-----------------------------------------------------------------------------------------------+
| Summary for scan 2.108.0 |
+----------------------------+-----------------------------------------------------------------------------------------------+
| workspace | /Users/XXXXXXXX/workspace/XXXXXXXX/XXXXXXXX.xcworkspace |
| scheme | AcceptanceTests |
| build_for_testing | true |
| derived_data_path | /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs |
| device | iPad Air 2 |
| skip_detect_devices | false |
| clean | true |
| skip_build | false |
| output_directory | ./fastlane/test_output |
| output_types | html,junit |
| buildlog_path | ~/Library/Logs/scan |
| include_simulator_logs | false |
| disable_concurrent_testing | false |
| should_zip_build_products | false |
| result_bundle | false |
| open_report | false |
| skip_slack | false |
| slack_only_on_failure | false |
| use_clang_report_name | false |
| reinstall_app | false |
| fail_build | true |
| xcode_path | /Applications/Xcode.app |
+----------------------------+-----------------------------------------------------------------------------------------------+
INFO [2018-11-12 10:21:21.26]: $ set -o pipefail && env NSUnbufferedIO=YES xcodebuild -workspace /Users/XXXXXXXX/workspace/XXXXXXXX/XXXXXXXX.xcworkspace -scheme AcceptanceTests -destination 'platform=iOS Simulator,id=DC284ED0-542C-4A85-8ECF-3C11F74C648A' -derivedDataPath '/Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs' clean build-for-testing | tee '/Users/XXXXXXXX/Library/Logs/scan/XXXXXXXX-AcceptanceTests.log' | xcpretty --report html --output '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/test_output/report.html' --report junit --output '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/test_output/report.junit' --report junit --output '/var/folders/r4/j37739zn5x5br0245zr5qkk90k4cww/T/junit_report20181112-36821-12p9n6i'
INFO [2018-11-12 10:23:55.61]: ▸ Linking XXXXXXXXUITests
INFO [2018-11-12 10:23:55.70]: ▸ Generating 'XXXXXXXXUITests.xctest.dSYM'
INFO [2018-11-12 10:23:55.81]: ▸ Copying TestConfiguration.plist
INFO [2018-11-12 10:23:56.49]: Running Tests: ▸ Touching XXXXXXXXUITests.xctest
INFO [2018-11-12 10:23:56.50]: ▸ Signing /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app/PlugIns/XXXXXXXXUITests.xctest
INFO [2018-11-12 10:23:56.68]: ▸ Signing /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app
INFO [2018-11-12 10:23:57.03]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+
DEBUG [2018-11-12 10:23:57.27]: Getting tests from xctestrun file at '/Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTests_iphonesimulator12.1-x86_64.xctestrun'
DEBUG [2018-11-12 10:23:57.36]: xctest binary path is: /Users/XXXXXXXX/Library/Developer/Xcode/DerivedData/XXXXXXXX-eyjyiyqbfcqkrpaplsfgbsfxupvs/Build/Products/AcceptanceTesting-iphonesimulator/XXXXXXXXUITests-Runner.app/PlugIns/XXXXXXXXUITests.xctest/XXXXXXXXUITests
DEBUG [2018-11-12 10:23:57.36]: Found the following tests:
DEBUG [2018-11-12 10:23:57.36]: Removing skipped tests: XXXXXXXXXTests
XXXXXXXXX
XXXXXXXXX
XXXXXXXXX
WARN [2018-11-12 10:23:57.36]: There are no tests to run in testable 'XXXXXXXXUITests'. Skipping
+-------------------+--------------------------------------------------------------------------------+
| multi_scan results |
+-------------------+--------------------------------------------------------------------------------+
| result | true |
| total_tests | 0 |
| passing_testcount | 0 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 0 |
| report_files | ["/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/test_output/report.junit", |
| | "/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/test_output/report.html"] |
+-------------------+--------------------------------------------------------------------------------+
INFO [2018-11-12 10:23:57.37]: ------------------------
INFO [2018-11-12 10:23:57.37]: --- Step: rm -rf tmp ---
INFO [2018-11-12 10:23:57.37]: ------------------------
INFO [2018-11-12 10:23:57.37]: $ rm -rf tmp
INFO [2018-11-12 10:23:57.38]: Successfully generated documentation at path '/Users/XXXXXXXX/workspace/XXXXXXXX/fastlane/README.md'
+------+----------------------------+-------------+
| fastlane summary |
+------+----------------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------------+-------------+
| 1 | opt_out_usage | 0 |
| 2 | Verifying fastlane version | 0 |
| 3 | default_platform | 0 |
| 4 | clear_derived_data | 2 |
| 5 | cd | 0 |
| 6 | update_info_plist | 0 |
| 7 | multi_scan | 161 |
| 8 | rm -rf tmp | 0 |
+------+----------------------------+-------------+
+-------------+--------------+----------------+
| Plugin updates available |
+-------------+--------------+----------------+
| Plugin | Your Version | Latest Version |
+-------------+--------------+----------------+
| test_center | 3.5.9 | 3.5.10 |
+-------------+--------------+----------------+
INFO [2018-11-12 10:23:57.39]: To update all plugins, just run
INFO [2018-11-12 10:23:57.39]: $ bundle exec fastlane update_plugins
INFO [2018-11-12 10:23:57.39]: fastlane.tools finished successfully 🎉
lyndsey-ferguson
All 39 comments
@hilljon86 have you made any progress on getting a sample of the scan output? Also, a copy of the final junit files could help me.
lyndsey-ferguson
on 14 Nov 2018
I've been having unrelated computer problems. I hope to get something to you today or tomorrow. Thanks for your help.
hilljon86
on 15 Nov 2018
Thanks for your plugin.
I have a small problem, maybe it has a relationship to this bug.
I used to run tests like this
scan(scheme: options[:scheme],
devices: options[:devices],
configuration: "Test",
clean: true,
output_types: "junit",
thread_sanitizer: true)
Now I switched to multi_scan
multi_scan(
try_count: 3,
fail_build: false,
scheme: options[:scheme],
devices: options[:devices],
configuration: "Test",
clean: true,
thread_sanitizer: true,
output_types: "junit",
result_bundle: true,
testrun_completed_block: test_run_block
)
However, in this configuration, I can not run tests
12:36:24]: There are no tests to run in testable '__xctestrun_metadata__'. Skipping
[12:36:24]: There are no tests to run in testable 'alphadirectTests'. Skipping
[12:36:24]: There are no tests to run in testable 'alphadirectUITests'. Skipping
However, if I remove configuration: "Test", then everything is fine
The project has the following settings for the configuration:
Now I can use Debug but I would like Test.
PetecOvod
on 21 Nov 2018
@hilljon86 have you had a chance to putting together a sample project? No rush for the rest of this week as I'll be taking time off, but I wanted to remind you.
@PetecOvod I would have to see your project to know more; I will still hazard a guess. Perhaps your Scheme's Test section's "Build Configuration" is not set correctly? To truly answer the question, I would need a project that exhibits the same problem. If my guess is not the answer, please attach a sample project.
lyndsey-ferguson
on 21 Nov 2018
Yes, in the settings of the scheme I have configuration: "Debug", but if I switch there to configuration: "Test", then the result will be the same - tests will not be found. I am attaching a sample project. In it lane run_UItest passes, but lane run_UItest2 does not find tests.
SampleTest.zip
PetecOvod
on 22 Nov 2018
Initial findings
nm -gU path/to/testbinarydoes not provide any symbols.otool -tV path/to/testbinarydoes.- Initial Googling of "nm -gU no symbols" does not pull up useful results.
I don't want to rework the xctest_list gem to use otool as I don't understand why there are no symbols. My feeling is that there is some relation to how this is being built: it shows up in a "Release" folder in the "Products" folder. There probably is some setting that _hides_ the symbols from the nm tool. Either I need to add intelligence to xctest_list or tests_from_xctestrun, or I need to detect whether this is targetting a "Release" build (if that turns out to be the problem) and abort with an error asking for a "Debug" build.
I will continue investigating the next time I get back to multi_scan.
lyndsey-ferguson
on 26 Nov 2018
Ok, by _not_ using the 'Test' configuration, nm -gU path/to/testbinary provides symbols.
lyndsey-ferguson
on 30 Nov 2018
That's funny, this project is not using the 'Test' configuration in the Build Settings.
lyndsey-ferguson
on 30 Nov 2018
In sample project 'Test' configuration is 'Release'. Lane run_UItest2 use 'Release' configuration and not find test
PetecOvod
on 30 Nov 2018
Ok, that makes sense. In general, Release configs are pre-set to remove symbols. That's probably what is happening. I was looking at my test lane that copied the code from the Issue and pointed to the sample project.
lyndsey-ferguson
on 30 Nov 2018
Although, your sample Project overrides the defaults and says not to optimize, nor to strip debug symbols.
lyndsey-ferguson
on 30 Nov 2018
It does say DEBUG=1 for Debug config, and nothing for Release.
lyndsey-ferguson
on 30 Nov 2018
No wait, it leaves full optimization on for Release
lyndsey-ferguson
on 30 Nov 2018
@lyndsey-ferguson sorry for the delay. The recent activity on this issue inspired me to check our build settings and I noticed that "Enable Testability" was set to "NO" on our AcceptanceTesting build configuration... no idea why it was set this way. Once I changed it to "YES" then multi_scan was able to run the tests. Thanks for the help and I'm sorry it ended up being something so simple on our end.
hilljon86
on 30 Nov 2018
😮
lyndsey-ferguson
on 30 Nov 2018
That is something that multi_scan probably can enforce
lyndsey-ferguson
on 30 Nov 2018
I'll check @PetecOvod's sample project
lyndsey-ferguson
on 30 Nov 2018
Same setting
lyndsey-ferguson
on 30 Nov 2018
Yes Enable Testability enabled on Test and Release in Sample project
PetecOvod
on 30 Nov 2018
Right, turning on "Enable Testability" fixes the issue.
lyndsey-ferguson
on 30 Nov 2018
Yes, I wanted to run tests as close as possible to production, as I used to do using lane run_UItest in Sample project. It turns out that it can not get tests with these settings.
PetecOvod
on 30 Nov 2018
Right, but you don't need it turned off for the Test Target.
lyndsey-ferguson
on 30 Nov 2018
@hilljon86 @PetecOvod could you test your projects on the PR that I have ready to merge (with ENABLE_TESTABILITY=NO). I want to make sure you get the warning messages.
I don't want to force that setting when using scan to build for testing, as it will enable it for the App as well as the test targets. So, instead, I warn the user.
To test this, can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-86-no-tests-from-xctestrun"
Please confirm that the warning message appears and makes sense.
lyndsey-ferguson
on 30 Nov 2018
@lyndsey-ferguson looks like that works! I received this message in the output Is the Build Setting,ENABLE_TESTABILITYenabled for the test target XXXXXXUITests?
hilljon86
on 30 Nov 2018
@lyndsey-ferguson thanks again for all your help! I'm glad that something good came out of my simple mistake!
hilljon86
on 30 Nov 2018
Two things happened. I learned something, and I got to add code to teach others the same thing. Thanks for the lesson!
lyndsey-ferguson
on 30 Nov 2018
https://www.rubydoc.info/gems/xctest_list/1.1.6/XCTestList
I've run nm -gU '#{binary_path(xctest_bundle_path)}' | cut -d' ' -f3 | xargs -s 131072 xcrun swift-demangle | cut -d' ' -f3 | grep -e '[\\.|_]'test on the same binary on macOS 10.11 (xcrun version 31) and 10.12 (xcrun version 36) and the 10.11 results don't have () at the end, so the swift_tests result will be empty because the regex match fails for all of them
Cyberbeni
on 18 Jan 2019
This is the regex that you're speaking of, right? Just to confirm, those are Swift tests, yes?
Are the test methods in the output (without ())?
Could you create a new issue and include a sample binary?
lyndsey-ferguson
on 18 Jan 2019
I deleted the last ( from that regex and it gets past getting the test cases but fastlane exits with build error (code 7). Didn't have time to debug what caused it, so I just reverted to xcodebuild and xcpretty for now.
Cyberbeni
on 18 Jan 2019
I have made some changes to the xctest_list gem to account for Xcode 8. Can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-86-no-swift-tests-with-xcode8"
Please attach the console output as a text file to this issue (makes it easier for me to review).
lyndsey-ferguson
on 4 Feb 2019
I get a nice error. Also attached fastlane env output and terminal log with xcodebuld test.
Cyberbeni
on 6 Feb 2019
Is the error that you are referencing?
INFO [2019-02-06 16:14:00.03]: ▸ Run `bundle install` to install missing gems.
Could not find CFPropertyList-3.0.0 in any of the sources
Run `bundle install` to install missing gems.
ERROR [2019-02-06 16:14:00.58]: Exit status: 7
oh yeah, that's the error I'm getting on my 10.12 tests too (where I'm using a modified version of test center that also catches build errors and uses trainer to get the junit report)
Cyberbeni
on 6 Feb 2019
Ah, had to add the homebrew installed ruby to my path... Now I get the no tests found error.
fastlane_output.txt
Cyberbeni
on 6 Feb 2019
Had to add the xctest_list source to my own gemfile and it works now
Cyberbeni
on 6 Feb 2019
so, it works?
lyndsey-ferguson
on 6 Feb 2019
Yes and it also fixed my other issue about no retries when xcpretty fails on 10.12 with xcode 9.2. (Although using trainer instead of xcpretty output is still useful for iOS testing, IIRC we needed it for physical devices)
On 2019. Feb 6., at 17:20, Lyndsey Ferguson notifications@github.com wrote:
so, it works?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.
Cyberbeni
on 6 Feb 2019
Nice, thank you for your help. I'll release the fix soon.
lyndsey-ferguson
on 6 Feb 2019
Fixed in v3.6.2
lyndsey-ferguson
on 7 Feb 2019
Related issues
BohdanOrlov
·
9Comments
ishammoho
·
9Comments
lyndsey-ferguson
·
6Comments
ericwengelking
·
10Comments
dionc
·
6Comments