Fastlane-plugin-test_center: It seems test plans still doesn't work
New Issue Checklist
- [x] Updated
fastlane-plugin-test_centerto the latest version - [x] I read the README.md
- [x] I reviewed the example(s) for the action(s) I am using
- [x] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
Issue Description
Hi!
I've tried to use test plans for my project with multiscan and it seems doesn't work.
Every time I getxcodebuild: error: The flag -testPlan <name> cannot be used since the scheme does not use test plans. from my CI (Bamboo). But test plans works locally. It also works with native fastlane's scan.
I saw the issue https://github.com/lyndsey-ferguson/fastlane-plugin-test_center/issues/212, but after update to 3.10.2 version test plans still don't works.
Could you help me with this problem? Thank you in advance!
Complete output when running fastlane, including the stack trace and command used
["/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane_core/lib/fastlane_core/ui/interface.rb:163:in
build_failure!'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane_core/lib/fastlane_core/ui/ui.rb:17:inmethod_missing'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/scan/lib/scan/error_handler.rb:46:in handle_build_error'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/scan/lib/scan/runner.rb:78:inblock in test_app'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane_core/lib/fastlane_core/command_executor.rb:89:in execute'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/scan/lib/scan/runner.rb:69:intest_app'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/scan/lib/scan/runner.rb:22:in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb:68:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/retrying_scan.rb:54:in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/test_batch_worker.rb:18:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/parallel_test_batch_worker.rb:49:in block in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/parallel_test_batch_worker.rb:45:infork'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/parallel_test_batch_worker.rb:45:in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:176:inblock in run_test_batches'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:173:in each'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:173:ineach_with_index'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:173:in run_test_batches'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/helper/multi_scan_manager/runner.rb:70:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-plugin-test_center-3.10.2/lib/fastlane/plugin/test_center/actions/multi_scan.rb:34:in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:261:inblock (2 levels) in execute_action'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/actions/actions_helper.rb:50:in execute_action'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:253:inblock in execute_action'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:227:in chdir'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:227:inexecute_action'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:157:in trigger_action_by_name'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/fast_file.rb:159:inmethod_missing'",
"scripts/shared_lanes.rb:16:in block in parsing_binding'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/lane.rb:33:incall'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:49:in block in execute'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:45:inchdir'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/runner.rb:45:in execute'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/lane_manager.rb:56:incruise_lane'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/command_line_handler.rb:36:in handle'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/commands_generator.rb:108:inblock (2 levels) in run'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:178:in call'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/command.rb:153:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/runner.rb:476:in run_active_command'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:76:inrun!'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/commander-fastlane-4.4.6/lib/commander/delegates.rb:15:in run!'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/commands_generator.rb:349:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/commands_generator.rb:41:in start'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/fastlane/lib/fastlane/cli_tools_distributor.rb:119:intake_off'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/fastlane-2.146.1/bin/fastlane:23:in <top (required)>'",
"/Users/bamboo/.rbenv/versions/2.6.3/bin/fastlane:23:inload'",
"/Users/bamboo/.rbenv/versions/2.6.3/bin/fastlane:23:in <top (required)>'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:inload'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in kernel_load'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in exec'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:inrun'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:indispatch'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in dispatch'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:instart'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in start'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:inblock in with_friendly_errors'",
"/Users/bamboo/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:inload'",
"/Users/bamboo/.rbenv/versions/2.6.3/bin/bundle:23:in
multiscan_parametres:
multi_scan(
workspace: PROJECT::Path::WORKSPACE_FILE_NAME,
scheme: 'myScheme',
testplan: 'testPlanSmoke',
device: PROJECT::Environment::SIMULATOR,
parallel_testrun_count: parallel_simulators_count,
try_count: PROJECT::Constants::TEST_RETRY_COUNT,
batch_count: 1,
clean: true,
code_coverage: true,
collate_reports: false,
result_bundle: true,
output_types: 'junit',
output_directory: PROJECT::Path::BUILD_FOLDER,
derived_data_path: PROJECT::Path::DERIVED_DATA_FOLDER,
buildlog_path: "#{PROJECT::Path::BUILD_FOLDER}/Log"
)
scan_parametres:
scan(
workspace: PROJECT::Path::WORKSPACE_FILE_NAME,
scheme: 'myScheme',
device: PROJECT::Environment::SIMULATOR,
clean: true,
code_coverage: true,
result_bundle: true,
output_directory: PROJECT::Path::BUILD_FOLDER,
derived_data_path: PROJECT::Path::DERIVED_DATA_FOLDER,
buildlog_path: "#{PROJECT::Path::BUILD_FOLDER}/Log",
testplan: "testPlanSmoke"
)
vdavydovHH
All 40 comments
Hello @vdavydovHH, thank you for letting me know about this issue. I am not able to spend a lot of time right now, so what ever you can do to help me with this would help get this through faster.
while it seems that what you have should work, as you've found it uses Test Plans, but can you please confirm that the
mySchemescheme is actually using test plans?If the answer to the above is yes, can you include a sample project/fastfile that exhibits this problem? That will allow me to get directly to the problem.
lyndsey-ferguson
on 6 May 2020
- Yes.
- I attach mini project where problem is reproduced. It should helps you to understand where is the issue.
MultiScanTestPlans.zip
Some useful commands to interact with project:
// Ruby Gems
bundle install
// Cocoapods
bundle exec pod install
// Test with multi_scan
bundle exec fastlane test_multi_scan
// Test with scan
bundle exec fastlane test_scan
vdavydovHH
on 7 May 2020
Thank you! Hopefully I can make progress on fixing this over the weekend. 🤞
lyndsey-ferguson
on 7 May 2020
@vdavydovHH can you modify your Pluginfile per my instructions below, run bundle install, and then run your fastlane again (with the --verbose flag)?
Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-221-testplan-crash-when-building"
If there are still problems, please let me know and attach the console output as a text file to this issue (makes it easier for me to review). If it works, please let me know.
lyndsey-ferguson
on 9 May 2020
@lyndsey-ferguson It doesn't help :(
Attach .txt file as you requested
logs.txt
vdavydovHH
on 12 May 2020
Are there any more logs? This is showing a crash, but I don't know what is the problem. Please include the surrounding terminal output so I can have more context.
🙏
lyndsey-ferguson
on 12 May 2020
Ok!
It is a full log from terminal
terminalOutput.txt
If you need anything else - tell me
I'll try to get i :)
vdavydovHH
on 12 May 2020
Ok. I could swear that I actually had it running, but now it is failing in the same way for me as it is for you. It is stating that the Scheme is not set up for test plans. I will have to look at this later, maybe this weekend.
lyndsey-ferguson
on 14 May 2020
Ok, I had a few mins, it seems that scan is sending the -scheme flag, and multi_scan is not....
lyndsey-ferguson
on 14 May 2020
Ok, the problem is that multi_scan is supplying the xctestrun file, which normally _should_ mean that we don't need the Scheme. AND, I believe my prior change was to actually use the smarts of the system to get the tests from the xctestrun file, not the test plan. So, maybe I do not need to pass down the -testPlan option to the testing either.
I have to confirm that the xctestrun file has the tests we want.
lyndsey-ferguson
on 14 May 2020
Here's the location where the scheme is _Not_ added to the xcodebuild command: https://github.com/fastlane/fastlane/blob/67267841a84f6781c6b1663bc2492b0cc0bcefef/scan/lib/scan/test_command_generator.rb#L34
lyndsey-ferguson
on 14 May 2020
Hm. I suppose you can change the test plan to be something other than the default for the scheme, right?
lyndsey-ferguson
on 14 May 2020
Schemes can have multiple test plans. The default test plan is selected if another test plan is not specified. So I can add another testplan and it won't be default.
On my main project I have two test plans - default and not. I tried to select not default test plan but result is the same - xcodebuild: error: The flag -testPlan <name> cannot be used since the scheme does not use test plans.
If i have not specified test plan, default will be selected and then all will be allright - multiscan won't crash.
I guess i should said about this moment early but i forgot, sorry :)
vdavydovHH
on 14 May 2020
Okay, thanks for the information. My thought is that _if_ multi_scan has to build, like in your case, and testplan is specified, to grab the tests from that plan and update the only_testing option and then not pass on 'testplan' to scan when running the tests.
lyndsey-ferguson
on 14 May 2020
Okay!
I will wait news from you after the weekend:)
vdavydovHH
on 14 May 2020
@vdavydovHH if you would like to test the work-in-progress that seems functional, you can update your Pluginfile:
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-221-testplan-crash-when-building"
I have a couple of things left to do.
lyndsey-ferguson
on 19 May 2020
Hi!
I tried to test your changes
Now i get next message: Error: Test Plan does not exist at path './myTestPlan.xctestplan'
Is it normal for this moment?
I am absolutely sure that the test plan exists at the path
vdavydovHH
on 21 May 2020
Hi @vdavydovHH, you've found a bug, thanks for testing!
lyndsey-ferguson
on 21 May 2020
It works fine for me with the sample. Can you share your actual Fastfile and testplan? Or, are you specifying any path with ./?
lyndsey-ferguson
on 21 May 2020
@vdavydovHH I've pushed up some updates with more verbose logging. Can you re-run this, but include the --verbose flag, and then attach the logs?
lyndsey-ferguson
on 23 May 2020
gem 'fastlane-plugin-test_center', :git => "https://github.com/lyndsey-ferguson/fastlane-plugin-test_center.git", :branch => "issue-221-testplan-crash-when-building"
You _may_ need to remove the Gemfile.lock file to get the actual latest commit (I've squashed my commits)
lyndsey-ferguson
on 23 May 2020
Hi @lyndsey-ferguson
Thanks for your attention!
I cannot reproduce this problem at the sample app as you
It happens on my main project
Now I get next error: Error setting value './../TestPlans/myTestPlan.xctestplan' for option 'testplan'
Today I will try modify sample app to you will be able reproduce this issue
Sorry for the delay, I've been very busy in the last few days :(
vdavydovHH
on 25 May 2020
Ok
I replaced .xcodeproj file into the folder with other project files as on my main project
Also I corrected Podfile to pass build phase
Now I get the same issue as on my main project
PS I changed iOS version. Now it is 13.4.1. If you need another version you may change it in Fastfile
vdavydovHH
on 25 May 2020
Hi @lyndsey-ferguson
Do you have any ideas about the message Error setting value './../TestPlans/myTestPlan.xctestplan' for option 'testplan' ?
vdavydovHH
on 27 May 2020
Hello @vdavydovHH, no not yet. I'll need to run your sample to get a clue: I hope that I'll be able to do that this weekend.
lyndsey-ferguson
on 27 May 2020
Okay
It is what I get with enabled --verbose flag:
INFO [2020-05-28 10:39:13.69]: ▸ Test build Succeeded
+--------------------+---+
| Test Results |
+--------------------+---+
| Number of tests | 0 |
| Number of failures | 0 |
+--------------------+---+
DEBUG [2020-05-28 10:39:13.71]: After building, found xctestrun files ["DerivedData/Build/Products/MultiScanTestPlans_MultiScanTestPlans_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-05-28 10:39:13.71]: Removing report files generated by the build
DEBUG [2020-05-28 10:39:13.71]: Build/report.junit
DEBUG [2020-05-28 10:39:13.72]: Looking in Scheme '/Users/v.davydov/Downloads/MultiScanTestPlans/MultiScanTestPlans/MultiScanTestPlans.xcodeproj/xcshareddata/xcschemes/MultiScanTestPlans.xcscheme' for any testplans
DEBUG [2020-05-28 10:39:13.72]: found testplan './../TestPlans/MultiScanTestPlans.xctestplan'
ERROR [2020-05-28 10:39:13.72]: Error setting value './../TestPlans/MultiScanTestPlans.xctestplan' for option 'testplan'
WARN [2020-05-28 10:39:13.72]: Lane Context:
INFO [2020-05-28 10:39:13.72]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>nil, :LANE_NAME=>"test_multi_scan"}
ERROR [2020-05-28 10:39:13.72]: Error: Test Plan does not exist at path './../TestPlans/MultiScanTestPlans.xctestplan'
@vdavydovHH can you try it again? Remove the Gemfile.lock file to ensure you get the freshest commit.
lyndsey-ferguson
on 29 May 2020
I don't have this problem at last commit
But there is another issue
I have two test plans
And now default test plan is always selected regardless of which test plan I specified
vdavydovHH
on 30 May 2020
I created a second test plan in the sample you gave me, updated the Fastfile to select that one, and when running multi_scan, it selected my second one. The second one is not the default.
I've updated the logging for this branch, can you remove your Gemfile.lock, run bundle install and retry your fastlane?
lyndsey-ferguson
on 31 May 2020
Okay
Forget my last comment about default and non-default test plans, it is a bullshit I guess
I have new idea about how it happens but not why it happens
I've created two test plans
When I select 'TestPlan1' and run lane with --verbose flag I get next message:
First message
```
DEBUG [2020-06-01 13:41:21.89]: After building, found xctestrun files ["DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun", "DerivedData/Build/Products/MultiScanTestPlans_TestPlan2_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-06-01 13:41:21.90]: Removing report files generated by the build
DEBUG [2020-06-01 13:41:21.90]: Build/report.junit
DEBUG [2020-06-01 13:41:21.90]: Looking in Scheme '/Users/v.davydov/Downloads/MultiScanTestPlans/MultiScanTestPlans/MultiScanTestPlans.xcodeproj/xcshareddata/xcschemes/MultiScanTestPlans.xcscheme' for any testplans
DEBUG [2020-06-01 13:41:21.91]: found testplan '/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan1.xctestplan'
DEBUG [2020-06-01 13:41:21.91]: found testplan '/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan2.xctestplan'
DEBUG [2020-06-01 13:41:21.91]: TestCollector found testplans: ["/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan1.xctestplan", "/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan2.xctestplan"]
DEBUG [2020-06-01 13:41:21.91]: using :testplan option, TestPlan1, using found one: /Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan1.xctestplan
DEBUG [2020-06-01 13:41:21.91]: Examining testplan JSON: {"configurations"=>[{"id"=>"41D42490-8358-4E73-A213-13ED9ADD36CF", "name"=>"Configuration 1", "options"=>{}}], "defaultOptions"=>{"codeCoverage"=>false, "targetForVariableExpansion"=>{"containerPath"=>"container:MultiScanTestPlans.xcodeproj", "identifier"=>"C07E667824643A99002813F0", "name"=>"MultiScanTestPlans"}}, "testTargets"=>[{"target"=>{"containerPath"=>"container:MultiScanTestPlans.xcodeproj", "identifier"=>"C07E668E24643A9B002813F0", "name"=>"MultiScanTestPlansTests"}}, {"target"=>{"containerPath"=>"container:MultiScanTestPlans.xcodeproj", "identifier"=>"C07E669924643A9B002813F0", "name"=>"MultiScanTestPlansUITests"}}], "version"=>1}
DEBUG [2020-06-01 13:41:21.91]: No selected tests, using testable 'MultiScanTestPlansTests'
DEBUG [2020-06-01 13:41:21.91]: No selected tests, using testable 'MultiScanTestPlansUITests'
DEBUG [2020-06-01 13:41:21.91]: Getting tests from xctestrun file at 'DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun'
DEBUG [2020-06-01 13:41:21.99]: Found the following tests: MultiScanTestPlansTests/testExample
MultiScanTestPlansTests/testPerformanceExample
DEBUG [2020-06-01 13:41:22.06]: Found the following tests: MultiScanTestPlansUITests/testExample
MultiScanTestPlansUITests/testLaunchPerformance
DEBUG [2020-06-01 13:41:22.06]: Getting tests from xctestrun file at 'DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun'
DEBUG [2020-06-01 13:41:22.12]: Found the following tests: MultiScanTestPlansTests/testExample
MultiScanTestPlansTests/testPerformanceExample
DEBUG [2020-06-01 13:41:22.19]: Found the following tests: MultiScanTestPlansUITests/testExample
MultiScanTestPlansUITests/testLaunchPerformance
DEBUG [2020-06-01 13:41:22.19]: Removing pre-existing xcresult bundles:
DEBUG [2020-06-01 13:41:22.19]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansTests-batch-1/report.xcresult
DEBUG [2020-06-01 13:41:22.19]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansUITests-batch-4/report.xcresult
DEBUG [2020-06-01 13:41:22.19]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansUITests-batch-3/report.xcresult
DEBUG [2020-06-01 13:41:22.19]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansTests-batch-2/report.xcresult
DEBUG [2020-06-01 13:41:22.23]: Fetching available simulator devices
DEBUG [2020-06-01 13:41:22.47]: Fetching available simulator devices
DEBUG [2020-06-01 13:41:22.70]: Cloning simulator
DEBUG [2020-06-01 13:41:22.94]: Cloning simulator
DEBUG [2020-06-01 13:41:23.18]: Cloning simulator
INFO [2020-06-01 13:41:25.63]: Starting test run 1
INFO [2020-06-01 13:41:25.63]: Starting test run 2
INFO [2020-06-01 13:41:25.63]: Starting test run 3
</details>
When I select 'TestPlan2' and run lane with --verbose flag I get next message:
<details>
<summary>Second message</summary>
DEBUG [2020-06-01 13:50:48.61]: After building, found xctestrun files ["DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun", "DerivedData/Build/Products/MultiScanTestPlans_TestPlan2_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-06-01 13:50:48.61]: Removing report files generated by the build
DEBUG [2020-06-01 13:50:48.61]: Build/report.junit
DEBUG [2020-06-01 13:50:48.62]: Looking in Scheme '/Users/v.davydov/Downloads/MultiScanTestPlans/MultiScanTestPlans/MultiScanTestPlans.xcodeproj/xcshareddata/xcschemes/MultiScanTestPlans.xcscheme' for any testplans
DEBUG [2020-06-01 13:50:48.62]: found testplan '/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan1.xctestplan'
DEBUG [2020-06-01 13:50:48.62]: found testplan '/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan2.xctestplan'
DEBUG [2020-06-01 13:50:48.62]: TestCollector found testplans: ["/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan1.xctestplan", "/Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan2.xctestplan"]
DEBUG [2020-06-01 13:50:48.62]: using :testplan option, TestPlan2, using found one: /Users/v.davydov/Downloads/MultiScanTestPlans/TestPlans/TestPlan2.xctestplan
DEBUG [2020-06-01 13:50:48.62]: Examining testplan JSON: {"configurations"=>[{"id"=>"DA3CFDD2-E5CF-4ED6-ABB6-690471F74958", "name"=>"Configuration 1", "options"=>{}}], "defaultOptions"=>{}, "testTargets"=>[{"target"=>{"containerPath"=>"container:MultiScanTestPlans.xcodeproj", "identifier"=>"3E3568BE247BC409004D5D86", "name"=>"MultiScanTestPlansUITests"}}], "version"=>1}
DEBUG [2020-06-01 13:50:48.62]: No selected tests, using testable 'MultiScanTestPlansUITests'
DEBUG [2020-06-01 13:50:48.62]: Getting tests from xctestrun file at 'DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun'
DEBUG [2020-06-01 13:50:48.71]: Found the following tests: MultiScanTestPlansTests/testExample
MultiScanTestPlansTests/testPerformanceExample
DEBUG [2020-06-01 13:50:48.77]: Found the following tests: MultiScanTestPlansUITests/testExample
MultiScanTestPlansUITests/testLaunchPerformance
DEBUG [2020-06-01 13:50:48.78]: Removing pre-existing xcresult bundles:
DEBUG [2020-06-01 13:50:48.78]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansTests-batch-1/report.xcresult
DEBUG [2020-06-01 13:50:48.78]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansUITests-batch-4/report.xcresult
DEBUG [2020-06-01 13:50:48.78]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansUITests-batch-3/report.xcresult
DEBUG [2020-06-01 13:50:48.78]: /Users/v.davydov/Downloads/MultiScanTestPlans/Build/MultiScanTestPlansTests-batch-2/report.xcresult
DEBUG [2020-06-01 13:50:48.80]: Fetching available simulator devices
DEBUG [2020-06-01 13:50:49.05]: Fetching available simulator devices
DEBUG [2020-06-01 13:50:49.27]: Cloning simulator
DEBUG [2020-06-01 13:50:49.53]: Cloning simulator
DEBUG [2020-06-01 13:50:49.77]: Cloning simulator
INFO [2020-06-01 13:50:52.48]: Starting test run 1
INFO [2020-06-01 13:50:52.48]: Starting test run 2
```
In my opinion there is two important things:
After building, found xctestrun files ["DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun", "DerivedData/Build/Products/MultiScanTestPlans_TestPlan2_iphonesimulator13.5-x86_64.xctestrun"] **(choosing 1st)**- it looks like 1st .xctestrun file is always selectedGetting tests from xctestrun file at 'DerivedData/Build/Products/MultiScanTestPlans_TestPlan1_iphonesimulator13.5-x86_64.xctestrun'- it is the same string in both messages. It looks like it always gets tests from the same .xctestrun file.
vdavydovHH
on 1 Jun 2020
Oh, there are two xctestrun files, one for each test plan. How devious of Apple.
lyndsey-ferguson
on 2 Jun 2020
Ok, I've updated the code to use the xctestrun associated with the test plan. Please test and let me know. One thing to keep in mind, you haven't told me what the actual latest problem is.
😉
lyndsey-ferguson
on 2 Jun 2020
Could you tell me, how TestCollector seeks test plans?
On my main project I get next message:
DEBUG [2020-06-03 16:44:01.58]: After building, found xctestrun files ["DerivedData/Build/Products/Scheme_TestPlan1_iphonesimulator13.5-x86_64.xctestrun", "DerivedData/Build/Products/Scheme_TestPlan2_iphonesimulator13.5-x86_64.xctestrun"] (choosing 1st)
DEBUG [2020-06-03 16:44:01.58]: Removing report files generated by the build
DEBUG [2020-06-03 16:44:01.58]: Build/report.junit
DEBUG [2020-06-03 16:44:01.67]: Looking in Scheme '/Users/v.davydov/ios/Applicant/Applicant.xcodeproj/xcshareddata/xcschemes/Scheme.xcscheme' for any testplans
DEBUG [2020-06-03 16:44:01.67]: TestCollector found testplans: []
DEBUG [2020-06-03 16:44:01.67]: using :testplan option, TestPlan2, using found one:
DEBUG [2020-06-03 16:44:01.67]: Getting tests from xctestrun file at 'DerivedData/Build/Products/ApplicantHH_TestPlan1_iphonesimulator13.5-x86_64.xctestrun'
On the sample app TestCollector finds test plans and all works correct. I can't modify sample so that TestCollector doesn't find test plans
vdavydovHH
on 3 Jun 2020
TestCollector gets the test plans from your scheme here
config = FastlaneCore::Configuration.create(
Fastlane::Actions::TestplansFromSchemeAction.available_options,
{
workspace: options[:workspace],
xcodeproj: options[:project],
scheme: options[:scheme]
}
)
testplans = Fastlane::Actions::TestplansFromSchemeAction.run(config)
FastlaneCore::UI.verbose("TestCollector found testplans: #{testplans}")
So, apparently, the Scheme it has does not have test plans in it.
You can test it directly by running this in a fastlane lane:
puts testplans_from_scheme(
workspace: <workspace>, # workspace OR xcodeproj
xcodeproj: <xcodeproj>,
scheme: 'Scheme'
)
@vdavydovHH have you had a chance to look at the Scheme that is being referenced? Any chance you could include it?
lyndsey-ferguson
on 5 Jun 2020
Hi @lyndsey-ferguson !
I'm so sorry about the so big delay, I was very sick last week :(
Today I'll try to understand where is the problem and I'll keep you up the date
vdavydovHH
on 8 Jun 2020
I am sorry to hear that you were very sick, but the fact that you're writing that you "were" means that you are better: that is great news.
Whatever you can send me for the scheme or the run of the testplans_from_scheme action would help me a lot. I want to make sure that this issue is fixed before merging this to master and releasing it.
Also, I saw the ❤️ sponsorship!! Thank you so much! Send me a DM tweet with your address so that I can send you the stickers on July 8th!
lyndsey-ferguson
on 8 Jun 2020
Okay
I guess all my problems with test plans running were because of my bad
But it seems there is some problem with .xcresult reports when parallel_testrun_count is enabled -reports are not created
Could you check it?
vdavydovHH
on 9 Jun 2020
Ohhh...
It is again my fault
So
I believe you may merge this branch to master
If I find other problems I'll let you know
Thank you, great job! 👍
vdavydovHH
on 9 Jun 2020
Fixed in v3.11.0
lyndsey-ferguson
on 10 Jun 2020
Related issues
ryanpato
·
4Comments
karthick-nagendran
·
10Comments
zacoid55
·
6Comments
jineshqa
·
9Comments
mykola-mokhnach
·
10Comments