Fastlane-plugin-test_center: Collating test_result bundles are deleted after completion with Xcode 10.3
New Issue Checklist
- [ ] Updated
fastlane-plugin-test_centerto the latest version (3.8.7) - [ ] I read the README.md
- [ ] I reviewed the example(s) for the action(s) I am using
- [ ] I have removed any sensitive data such as passwords, authentication tokens, or anything else I do not want to world to see
Issue Description
The parallelization of multi_scan is deleting test_result bundles when it cleans up. These result bundles are stored in the intermediate batch directories for each simultaneous run of tests.
When the test starts it generates 6 different folders (running 6 parallel tests). However, when the test finished it cleans all 6 of these bundles (folders).
If we add option collate_reports:false the folders not cleaned but we have folders also for re-runs
And we have only 23 test cases but without collate, multiscan calculate the re-run tests
Environment
- Xcode
10.3 - Fastlane
2.134.0 - Ruby
2.6+ - fastlane-plugin-test_center
3.8.7
🙅♂️ wontfix
moatazeldebsy
All 26 comments
Hello @moatazeldebsy, thank you for the bug report. A couple things I want to clarify with you.
multi_scan by default, will collate test_result bundles when given the option to generate them. Collation means that all the test_result bundles are merged into one final test_result bundle. This means that the others are removed.
So, it is a feature to remove the intermediate test_result bundles. Those are _intermediate_ bundles. This is intended behavior.
lyndsey-ferguson
on 18 Oct 2019
When you provide the collate_reports:false option, the test_result bundles are _not_ collated and the intermediate files are _not_ removed because they are not merged into the final test result.
lyndsey-ferguson
on 18 Oct 2019
Unless I misunderstand, this is not an actual bug unless you are asking for a new feature.
lyndsey-ferguson
on 18 Oct 2019
@lyndsey-ferguson Thank you for your replay and actually the problem that after multi_scan is collating the results from different folders and removed them, the last folder is including only the test output from last clone (in our case clone 6).
Xpretty html report contains the correct number of tests with the correct status but
we are using Allure-reportto generate reports with screenshots and graphs after multi_scan is finished we run this command
allure generate fastlane/test_output/UISmokeTests.test_result
and allure generate the report for us with these results:
Parse file fastlane/test_output/UISmokeTests.test_result/TestSummaries.plist
Parse file fastlane/test_output/UISmokeTests.test_result/Info.plist
Report successfully generated to allure-report
Then
allure open allure-report and we found only 4 tests from 23 tests, and after we check that we found that the last clone number 6 was including 4 tests and this is our problems currently.
moatazeldebsy
on 18 Oct 2019
Can you include a log of your fastlane run (please use the --verbose flag)?
lyndsey-ferguson
on 18 Oct 2019
this is for collate report part
DEBUG [2019-10-18 22:38:24.66]: TestCenter::Helper::HtmlTestReport::Report.collate_report to report:
<html>
<head>
<meta charset='utf-8'/>
<title>Test Results | xcpretty</title>
<style type='text/css'>
body { font-family:Avenir Next, Helvetica Neue, sans-serif; color: #4A4A4A; background-color: #F0F3FB; margin:0;}
h1 { font-weight: normal; font-size: 24px; margin: 10px 0 0 0;}
h3 { font-weight: normal; margin: 2px; font-size: 1.1em;}
header { position: fixed;width: 100%;background: rgba(249, 254, 255, 0.9);margin: 0;padding: 10px;}
header:before, header:after { content:""; display:table;}
header:after { clear:both;}
a:link { color: #A1D761;}
footer { clear: both;position: relative;z-index: 10;height: 40px;margin-top: -10px; margin-left:30px; font-size:12px;}
table { width:100%; border-collapse: collapse;}
tr td:first-child { width:7%}
.left { float: left; margin-left:30px;}
.right { float: right; margin-right: 40px; margin-top: 0; margin-bottom:0;}
.test-suite { margin: 0 0 30px 0;}
.test-suite > .heading { font-family:Menlo, Monaco, monospace; font-weight: bold; border-color: #A1D761; background-color: #B8E986; border-width: 1px;}
.test-suite.failing > .heading { border-color: #C84F5E; background-color: #E58591;}
.test-suite > .heading > .title { margin-top: 4px; margin-left: 10px;}
.tests { overflow: scroll;margin: 0 30px 0 60px;}
.test, .test-suite > .heading { height: 30px; overflow: hidden; margin: 0 30px;}
.test, .test-suite > .heading { border-width: 1px; border-collapse: collapse; border-style: solid; }
.test { margin-left: 30px; border-top:none;}
.test.failing { border-color: #C84F5E; background-color: #F4DDE0;}
.test.passing { border-color: #A1D761;}
.test.failing { background-color: #E7A1AA;}
.test.passing { background-color: #CAF59F;}
.test.failing.odd { background-color: #EEC7CC;}
.test.passing.odd { background-color: #E5FBCF;}
.details.failing { background-color: #F4DDE0; border: 1px solid #C84F5E;}
.details.passing { background-color: #E5F4DC; border: 1px solid #A1D761;}
.test .test-detail:last-child { padding-bottom: 8px;}
.test .title { float: left; font-size: 0.9em; margin-top: 8px; font-family: Menlo, Monaco, monospace;}
.test .time { float: left;margin: 4px 10px 0 20px;}
.test-detail { font-family:Menlo, Monaco, monospace; font-size: 0.9em; margin: 5px 0 5px 0px;}
.screenshots { height: auto; overflow: hidden; padding: 4px 4px 0 4px; background-color: #B8E986; border: #A1D761; border-width: 0 1px; border-style: solid; }
.screenshots.failing { border-color: #C84F5E; background-color: #E58591; }
.screenshot { max-height: 60px; float: left; transition: max-height 0.2s; margin: 0 4px 4px 0 }
.screenshot.selected { max-height: 568px; }
#test-suites { display: inline-block; width: 100%;margin-top:100px;}
#segment-bar { margin-top: 10px;margin-left: 14px;float:right;}
#segment-bar a:first-child { border-radius: 9px 0 0 9px; border-right: none;}
#segment-bar a:last-child { border-radius: 0 9px 9px 0; border-left: none;}
#segment-bar > a { color: #565656; border: 2px solid #7B7B7B; width: 80px; font-weight: bold; display:inline-block;text-align:center; font-weight: normal;}
#segment-bar > a.selected { background-color: #979797; color: #F0F3FB;}
#counters { float: left;margin: 10px;text-align: right;}
#counters h2 { font-size: 16px; font-family: Avenir, sans-serif; font-weight: lighter; display:inline;}
#counters .number { font-size: 20px;}
#fail-count { color: #D0021B; margin-left:10px;}
@media (max-width: 640px) {
h1, #counters, #segment-bar { margin: 5px auto; text-align:center;}
header, #segment-bar { width: 100%; position: relative; background:none;}
.left, .right { float:none; margin:0;}
#test-suites { margin-top: 0;}
#counters { float:none;}
}
</style>
<script type='text/javascript'>
var hide = function(element) { element.style.display = 'none';}
var show = function(element) { element.style.display = '';}
var isHidden = function(element) { return element.style.display == 'none';}
var isSelected = function(element) { return element.classList.contains("selected");}
var deselect = function(element) { return element.classList.remove("selected");}
var select = function(element) { return element.classList.add("selected");}
var toggle = function(element) { isHidden(element) ? show(element) : hide(element);};
var toggleTests = function(heading) { toggle(heading.parentNode.children[1]);};
var toggleDetails = function(detailClass) {
var details = document.querySelectorAll('.' + detailClass);
for (var i = details.length - 1; i >= 0; i--) { toggle(details[i]);};
};
var hideAll = function(collection) {
for (var i = collection.length - 1; i >= 0; i--) { hide(collection[i]); };
}
var showAll = function(collection) {
for (var i = collection.length - 1; i >= 0; i--) { show(collection[i]); };
}
var selectSegment = function(segment) {
if (isSelected(segment)) return;
var segments = document.querySelectorAll('#segment-bar > a');
for (var i = segments.length - 1; i >= 0; i--) { deselect(segments[i]);};
select(segment);
if (segment.id == "all-segment") {
showAll(document.querySelectorAll('.test-suite'));
showAll(document.querySelectorAll('.test'));
} else if (segment.id == "failing-segment") {
hideAll(document.querySelectorAll('.test.passing'));
showAll(document.querySelectorAll('.test.failing'));
hideAll(document.querySelectorAll('.test-suite.passing'));
showAll(document.querySelectorAll('.test-suite.failing'));
} else if (segment.id == "passing-segment") {
hideAll(document.querySelectorAll('.test.failing'));
showAll(document.querySelectorAll('.test.passing'));
hideAll(document.querySelectorAll('.test-suite.failing'));
showAll(document.querySelectorAll('.test-suite.passing'));
}
}
var toggleScreenshot = function(suiteName, index) {
var screenshot = document.getElementById("screenshot-" + suiteName + "-" + index);
isSelected(screenshot) ? deselect(screenshot) : select(screenshot);
}
</script>
</head>
<body>
<header>
<section class='left'>
<h1>Test Results</h1>
</section>
<section class='right'>
<section id='counters'>
<h2 id='test-count'><span class='number'>23</span> tests</h2>
<h2 id='fail-count'><span class='number'>3</span></h2></section>
<section id='segment-bar'>
<a class='selected' id='all-segment' onclick='selectSegment(this);'>All</a><a id='failing-segment' onclick='selectSegment(this);'>Failing</a><a id='passing-segment' onclick='selectSegment(this);'>Passing</a>
</section>
</section>
</header>
<section id='test-suites'>
<section class='test-suite passing' id='VoloUITests.CartCalculationUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.CartCalculationUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testCartSubtotalCalculation');'>
<td>
<h3 class='time'>62.072s</h3>
</td>
<td><h3 class='title'>testCartSubtotalCalculation</h3></td>
</tr>
</table>
</section>
</section>
<section class='test-suite passing' id='VoloUITests.ProductsManipulationUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.ProductsManipulationUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testDecreaseProductQuantity');'>
<td>
<h3 class='time'>48.792s</h3>
</td>
<td><h3 class='title'>testDecreaseProductQuantity</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testIncreaseProductQuantity');'>
<td>
<h3 class='time'>53.714s</h3>
</td>
<td><h3 class='title'>testIncreaseProductQuantity</h3></td>
</tr>
<tr class='test passing ' onclick='toggleDetails('testRemoveProduct');'>
<td>
<h3 class='time'>47.312s</h3>
</td>
<td><h3 class='title'>testRemoveProduct</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testRemoveProductWhenCountGoesToZero');'>
<td>
<h3 class='time'>45.467s</h3>
</td>
<td><h3 class='title'>testRemoveProductWhenCountGoesToZero</h3></td>
</tr>
</table>
</section>
</section>
<section class='test-suite passing' id='VoloUITests.ProfileEditingUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.ProfileEditingUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testChangeMobileNumber');'>
<td>
<h3 class='time'>80.475s</h3>
</td>
<td><h3 class='title'>testChangeMobileNumber</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testChangePassword');'>
<td>
<h3 class='time'>102.391s</h3>
</td>
<td><h3 class='title'>testChangePassword</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.RestaurantMenuUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.RestaurantMenuUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testAddProductsToCart');'>
<td>
<h3 class='time'>51.064s</h3>
</td>
<td><h3 class='title'>testAddProductsToCart</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testAddProductsWithVariationsToCart');'>
<td>
<h3 class='time'>53.810s</h3>
</td>
<td><h3 class='title'>testAddProductsWithVariationsToCart</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.PaypalHPPUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.PaypalHPPUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testNewPayment');'>
<td>
<h3 class='time'>21.040s</h3>
</td>
<td><h3 class='title'>testNewPayment</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testTokenizedPayment');'>
<td>
<h3 class='time'>11.127s</h3>
</td>
<td><h3 class='title'>testTokenizedPayment</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.VoucherUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.VoucherUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testCalculationWithAmountVoucher');'>
<td>
<h3 class='time'>104.826s</h3>
</td>
<td><h3 class='title'>testCalculationWithAmountVoucher</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testCalculationWithFreeDeliveryVoucher');'>
<td>
<h3 class='time'>88.721s</h3>
</td>
<td><h3 class='title'>testCalculationWithFreeDeliveryVoucher</h3></td>
</tr>
<tr class='test passing ' onclick='toggleDetails('testCalculationWithPercentageVoucher');'>
<td>
<h3 class='time'>81.618s</h3>
</td>
<td><h3 class='title'>testCalculationWithPercentageVoucher</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing ' id='VoloUITests.PaypalSDKUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.PaypalSDKUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testNewUser');'>
<td>
<h3 class='time'>117.503s</h3>
</td>
<td><h3 class='title'>testNewUser</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing ' id='VoloUITests.RegistrationUITest'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.RegistrationUITest</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testUserWithEmail');'>
<td>
<h3 class='time'>40.223s</h3>
</td>
<td><h3 class='title'>testUserWithEmail</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.AddressUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.AddressUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('test01UserCanSearchAddressAndGetRestaurants');'>
<td>
<h3 class='time'>65.152s</h3>
</td>
<td><h3 class='title'>test01UserCanSearchAddressAndGetRestaurants</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.LoginUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.LoginUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testUserWithEmail');'>
<td>
<h3 class='time'>56.840s</h3>
</td>
<td><h3 class='title'>testUserWithEmail</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite passing' id='VoloUITests.ReorderUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.ReorderUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testQuickReorderOrderPlaced');'>
<td>
<h3 class='time'>63.615s</h3>
</td>
<td><h3 class='title'>testQuickReorderOrderPlaced</h3></td>
</tr>
</table>
</section>
</section><section class='test-suite failing' id='VoloUITests.AddressBookUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.AddressBookUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test failing ' onclick='toggleDetails('testUserCanAddNewAddressWithEditAndDeletion');'>
<td>
</td>
<td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
</tr>
<tr class='details failing testUserCanAddNewAddressWithEditAndDeletion'>
<td/>
<td>
<section class='test-detail reason'>XCTAssertTrue failed</section>
<section class='test-detail snippet'/>
<section class='test-detail'>AddressBookUITests.swift:45</section>
</td>
</tr>
</table>
</section>
</section><section class='test-suite failing' id='VoloUITests.CashOnDeliveryUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test failing ' onclick='toggleDetails('testNewUserWithEmail');'>
<td>
</td>
<td><h3 class='title'>testNewUserWithEmail</h3></td>
</tr>
<tr class='details failing testNewUserWithEmail'>
<td/>
<td>
<section class='test-detail reason'>No matches found for Find: Elements matching predicate '"closeXButton" IN identifiers' from input {(</section>
<section class='test-detail snippet'/>
<section class='test-detail'><unknown>:0</section>
</td>
</tr>
<tr class='test failing odd' onclick='toggleDetails('testReturningUserEmail');'>
<td>
</td>
<td><h3 class='title'>testReturningUserEmail</h3></td>
</tr>
<tr class='details failing testReturningUserEmail'>
<td/>
<td>
<section class='test-detail reason'>XCTAssertTrue failed</section>
<section class='test-detail snippet'/>
<section class='test-detail'>CashOnDeliveryUITests.swift:33</section>
</td>
</tr>
<tr class='test passing ' onclick='toggleDetails('testReturningUserFacebook');'>
<td>
<h3 class='time'>77.157s</h3>
</td>
<td><h3 class='title'>testReturningUserFacebook</h3></td>
</tr>
</table>
</section>
</section></section>
<footer>Report generated with <a href='https://github.com/supermarin/xcpretty'>xcpretty</a></footer>
</body>
</html>
DEBUG [2019-10-18 22:38:24.69]: collating testsuite
<section class='test-suite passing' id='VoloUITests.AddressBookUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.AddressBookUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testUserCanAddNewAddressWithEditAndDeletion');'>
<td>
<h3 class='time'>79.024s</h3>
</td>
<td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
</tr>
</table>
</section>
</section>
DEBUG [2019-10-18 22:38:24.69]: updating testcase
<tr class='test failing ' onclick='toggleDetails('testUserCanAddNewAddressWithEditAndDeletion');'>
<td>
</td>
<td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
</tr>
DEBUG [2019-10-18 22:38:24.71]: after collation exiting testsuite
<section class='test-suite passing ' id='VoloUITests.AddressBookUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.AddressBookUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testUserCanAddNewAddressWithEditAndDeletion');'>
<td>
<h3 class='time'>79.024s</h3>
</td>
<td><h3 class='title'>testUserCanAddNewAddressWithEditAndDeletion</h3></td>
</tr>
</table>
</section>
</section>
DEBUG [2019-10-18 22:38:24.73]: collating testsuite
<section class='test-suite passing' id='VoloUITests.CashOnDeliveryUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testNewUserWithEmail');'>
<td>
<h3 class='time'>88.714s</h3>
</td>
<td><h3 class='title'>testNewUserWithEmail</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testReturningUserEmail');'>
<td>
<h3 class='time'>60.746s</h3>
</td>
<td><h3 class='title'>testReturningUserEmail</h3></td>
</tr>
<tr class='test passing ' onclick='toggleDetails('testReturningUserFacebook');'>
<td>
<h3 class='time'>65.932s</h3>
</td>
<td><h3 class='title'>testReturningUserFacebook</h3></td>
</tr>
</table>
</section>
</section>
DEBUG [2019-10-18 22:38:24.77]: updating testcase
<tr class='test failing ' onclick='toggleDetails('testNewUserWithEmail');'>
<td>
</td>
<td><h3 class='title'>testNewUserWithEmail</h3></td>
</tr>
DEBUG [2019-10-18 22:38:24.78]: updating testcase
<tr class='test failing odd' onclick='toggleDetails('testReturningUserEmail');'>
<td>
</td>
<td><h3 class='title'>testReturningUserEmail</h3></td>
</tr>
DEBUG [2019-10-18 22:38:24.79]: updating testcase
<tr class='test passing ' onclick='toggleDetails('testReturningUserFacebook');'>
<td>
<h3 class='time'>77.157s</h3>
</td>
<td><h3 class='title'>testReturningUserFacebook</h3></td>
</tr>
DEBUG [2019-10-18 22:38:24.80]: after collation exiting testsuite
<section class='test-suite passing ' id='VoloUITests.CashOnDeliveryUITests'>
<section class='heading' onclick='toggleTests(this);'>
<h3 class='title'>VoloUITests.CashOnDeliveryUITests</h3>
</section>
<section class='tests'>
<table>
<tr class='test passing ' onclick='toggleDetails('testNewUserWithEmail');'>
<td>
<h3 class='time'>88.714s</h3>
</td>
<td><h3 class='title'>testNewUserWithEmail</h3></td>
</tr>
<tr class='test passing odd' onclick='toggleDetails('testReturningUserEmail');'>
<td>
<h3 class='time'>60.746s</h3>
</td>
<td><h3 class='title'>testReturningUserEmail</h3></td>
</tr>
<tr class='test passing ' onclick='toggleDetails('testReturningUserFacebook');'>
<td>
<h3 class='time'>65.932s</h3>
</td>
<td><h3 class='title'>testReturningUserFacebook</h3></td>
</tr>
</table>
</section>
</section>
DEBUG [2019-10-18 22:38:24.99]: Collating test_result bundles ["/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-3/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-2/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-5/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests-2.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-4/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests-2.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests.test_result"]
DEBUG [2019-10-18 22:38:25.05]: result bundles are NOT of format version 3
INFO [2019-10-18 22:39:21.91]: Finished collating test_result bundle to '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests/FoodpandaUISmokeTests.test_result'
DEBUG [2019-10-18 22:39:30.49]: Results for each test run: [false, true, true, true, true, false]
+-------------------+-----------------------------------------------------------------------+
| multi_scan results |
+-------------------+-----------------------------------------------------------------------+
| result | false |
| total_tests | 23 |
| passing_testcount | 23 |
| failed_testcount | 0 |
| failed_tests | [] |
| total_retry_count | 8 |
| report_files | ["/fastlane/test_output/report.junit" |
| | , |
| | "/fastlane/test_output/report.html", |
| | "/pd-mob-b2c-ios/fastlane/test_output/FoodpandaUISmo |
| | keTests.test_result"] |
+-------------------+-----------------------------------------------------------------------+
INFO [2019-10-18 22:39:30.61]: Cruising back to lane 'ios ta_foodpanda_smoke' 🚘
INFO [2019-10-18 22:39:30.61]: -------------------
INFO [2019-10-18 22:39:30.61]: --- Step: slack ---
INFO [2019-10-18 22:39:30.61]: -------------------
INFO [2019-10-18 22:39:31.28]: Successfully sent Slack notification
INFO [2019-10-18 22:39:31.29]: Successfully generated documentation at path '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/README.md'
+------+-----------------------------------------+-------------+
| fastlane summary |
+------+-----------------------------------------+-------------+
| Step | Action | Time (in s) |
+------+-----------------------------------------+-------------+
| 1 | Verifying fastlane version | 0 |
| 2 | default_platform | 0 |
| 3 | Switch to ios build_run_ta_testing lane | 0 |
| 4 | multi_scan | 886 |
| 5 | slack | 0 |
+------+-----------------------------------------+-------------+
DEBUG [2019-10-18 22:39:31.30]: All plugins are up to date
INFO [2019-10-18 22:39:31.31]: fastlane.tools just saved you 15 minutes! 🎉
DEBUG [2019-10-18 22:39:31.31]: All plugins are up to date
INFO [2019-10-18 22:39:31.31]: Deleting iPhone Xʀ Clone 1 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:38.78]: Deleting iPhone Xʀ Clone 2 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:46.04]: Deleting iPhone Xʀ Clone 3 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:39:53.07]: Deleting iPhone Xʀ Clone 4 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:40:00.02]: Deleting iPhone Xʀ Clone 5 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
INFO [2019-10-18 22:40:06.97]: Deleting iPhone Xʀ Clone 6 TestCenter::Helper::MultiScanManager::SimulatorHelper<70339986019380>
Ah, okay. Yes, collate_test_result_bundles will not properly collate pre-version 3 format bundles. With Xcode 11, it _will_. It is a lot of work to collate pre-Xcode 11 based test_results properly and Xcode 11 does it with one command, so I have decided to not do the work.
Please update to Xcode 11 and this will begin working.
lyndsey-ferguson
on 18 Oct 2019
I will close this issue as I don't intend on fixing it. If you feel strongly about it and want to fix that action to work pre-Xcode-11, I can give you guidance on making the changes.
lyndsey-ferguson
on 18 Oct 2019
@moatazeldebsy just think, if you do fix this for Xcode 10 people, you'll become famous in the test_center community!! You'll be the unsung hero!
Let me know if you'd like to tackle this. It is not complicated work, just a series of things that you need to do that take some time. Time I don't have to do it by myself.
lyndsey-ferguson
on 18 Oct 2019
I would review this PR where I fixed collation for HTML test results. I would suggest the same approach. Make wrapper classes for each file of a Xcode 10 test result with methods to "collate" each file.
lyndsey-ferguson
on 18 Oct 2019
@lyndsey-ferguson ok will try to do that. I need to check this PR or need to do something else
moatazeldebsy
on 18 Oct 2019
🚀1
🎉1
👍1
@lyndsey-ferguson I updated to Xcode 11 and the results are in xcresult format without screenshots and logs.
inside test_results
inside scheme folder
insdie data folder
DEBUG [2019-10-19 15:17:51.59]: adding testsuite
#<TestCenter::Helper::HtmlTestReport::TestSuite:0x00007fa932ee9c50>
DEBUG [2019-10-19 15:17:51.61]: adding testsuite
#<TestCenter::Helper::HtmlTestReport::TestSuite:0x00007fa932ee9bb0>
DEBUG [2019-10-19 15:17:51.74]: Collating test_result bundles ["/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-3/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-2/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-5/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-4/FoodpandaUISmokeTests.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-6/FoodpandaUISmokeTests-4.test_result", "/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests-batch-1/FoodpandaUISmokeTests-2.test_result"]
DEBUG [2019-10-19 15:17:51.74]: result bundles are of format version 3
INFO [2019-10-19 15:17:51.81]: xcrun xcresulttool merge /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1kblznm.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-14t8sg5.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1yjpx97.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1nkujtl.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-4.test_result20191019-99406-1dvadtf.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-2.test_result20191019-99406-17356zd.xcresult --output-path /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:17:51.81]: $ xcrun xcresulttool merge /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1kblznm.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-14t8sg5.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1yjpx97.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests.test_result20191019-99406-1nkujtl.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-4.test_result20191019-99406-1dvadtf.xcresult /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/FoodpandaUISmokeTests-2.test_result20191019-99406-17356zd.xcresult --output-path /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:17:59.14]: ▸ [v3] Merged to: /var/folders/pw/hkgjc9kx7dn9v7xswlzy0llxdh0jc7/T/d20191019-99406-1itt0gc/FoodpandaUISmokeTests.test_result
INFO [2019-10-19 15:18:00.49]: Finished collating test_result bundle to '/Users/m.eldebsy/pd-mob-b2c-ios/fastlane/test_output/VoloUITests/FoodpandaUISmokeTests.test_result'
DEBUG [2019-10-19 15:18:02.60]: Results for each test run: [true, true, true, true, true, false]
Environment
Xcode 11.1
Fastlane 2.134.0
Ruby 2.6+
fastlane-plugin-test_center (3.8.10)
with fastlane-plugin-test_center (3.8.7) all folders for batches already exists and including xcresult only
moatazeldebsy
on 19 Oct 2019
@moatazeldebsy are you saying that the original test result bundles have screenshots and then when the xcresulttool performs the merge, the screenshots are no longer there? Can you please post sample test result bundles that demonstrate the failing merge?
(My Xcode 11 install is broken and I cannot get tests to run on my local computer at the moment.)
lyndsey-ferguson
on 21 Oct 2019
Yes during running the tests the folders have screenshots and logs but in the collate step (last step) all removed and I found only the xcresult files.
And this is the final result for each batch and folders exist for all batches without logs or screenshots
we have 1 folder including index and JUnit report
moatazeldebsy
on 21 Oct 2019
Ok. Please post sample test result bundles so that I can test that out. Once you do that, I'll update this Issue and reopen it.
lyndsey-ferguson
on 21 Oct 2019
sample test result bundles from the logs or the folders?, I didn't get it
moatazeldebsy
on 21 Oct 2019
I need sample result bundles. If you could provide your call the multi_scan the option collate_reports: false, it will generate uniquely named test result bundles. Please zip those up and and attach them to this Issue (make sure no sensitive data is provided).
lyndsey-ferguson
on 21 Oct 2019
@lyndsey-ferguson I would like to add that .xctestresult files are altogether missing. I use xchtmlreport to generate xcode type looking result html. This plugin will soon be upgraded to support Xcode 11 and would need .xcresultfiles to generate html. Could you not delete .xcresult file or collate 1 .xcresult file from multiple simulator runs?
jineshqa
on 21 Oct 2019
From my observation, the xcresult file is just a test_result bundle with the xcresult file extension.
lyndsey-ferguson
on 21 Oct 2019
So far the new version of the plugin is not released. As soon as it is released, I'll test it out and let you know. In worst case could we get a collated .xcresult file instead of bundle?
jineshqa
on 21 Oct 2019
You _may_ have to make a symbolic link ending with .xcresult that points to your resulting test_result bundle and provide that the the xchtmlreport plugin, but I really don't know. Play with it and let me know 😄
lyndsey-ferguson
on 21 Oct 2019
@lyndsey-ferguson test_results is big file but like you said test_result bundle with the xcresult file extension without any screenshot or logs even I disable collate this is the results from 1 clone like the following image
moatazeldebsy
on 21 Oct 2019
@moatazeldebsy can you upload the bundles to a service like Dropbox? I need to run the code to see where it is failing and what is in the intermediate test_result bundles.
lyndsey-ferguson
on 22 Oct 2019
Hi @lyndsey-ferguson can I know with which Xcode version you fixed this issue Fixes issue with multi_scan when collating test_result?
moatazeldebsy
on 25 Nov 2019
Sorry, I don't remember.
lyndsey-ferguson
on 25 Nov 2019
I believe I only started using Xcode 11 around Oct 18th.
lyndsey-ferguson
on 25 Nov 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
jineshqa
·
9Comments
BohdanOrlov
·
9Comments
lyndsey-ferguson
·
5Comments
karthick-nagendran
·
10Comments
bpoplauschi
·
9Comments
Most helpful comment
@lyndsey-ferguson ok will try to do that. I need to check this PR or need to do something else