Karate: in Karate the surefire XML report created appends and merges test steps for all examples one after another in next test executed

[kraul25]

am running following scenario example:

Scenario Outline: Basic test for user
When url 'xyz'
And path 'abc/efg'
And params id =
When method get
Then status 200

Example:
|userID|
|1|
|2|
|3|
|4|
Note: I am passing userID from examples to scenario name and also params now after this is executed the xml report generated in surefire for above test looks something like this:

name="xyz.feature"
tests="4"
time="8.5">
name="Basic test for user 1"
time="1.20"

When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '1'...............passed
When method get...................passed
Then status 200...................passed

name="Basic test for user 2"
time="2.10"

When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '1'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '2'...............passed
When method get...................passed
Then status 200...................passed

name="Basic test for user 3"
time="3.21"

When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '1'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '2'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '3'...............passed
When method get...................passed
Then status 200...................passed

name="Basic test for user 4"
time="3.56"

When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '1'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '2'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '3'...............passed
When method get...................passed
Then status 200...................passed
When url 'xyz' ...................passed
And path 'abc/efg'................passed
And params id = '4'...............passed
When method get...................passed
Then status 200...................passed


Please see how for first test case with user id 1 there are 4 steps printed as passed which is correct. Now for second test case the steps of first test case are appended to steps of second testcase. For 3rd testcase steps of testcase1 and testcase2 are shown along with testcase3 steps.This creates an issue in CI pipelines. Ideally in xml report all tescase should have only 4 steps but that is not the case.

Wanted to check on what could be the issue here. I am using parallel runner as: https://github.com/intuit/karate/blob/master/karate-demo/src/test/java/demo/DemoTestParallel.java

Logback file as : https://github.com/intuit/karate/blob/master/examples/jobserver/src/test/java/logback-test.xml

let me know if any more details required.

Stackoverflow question link: https://stackoverflow.com/questions/63747415/in-karate-the-surefire-xml-report-created-appends-and-merges-test-steps-for-all

[ptrthomas]

@kraul25 sorry, I specifically asked for a sample project. can reopen once that is available: https://github.com/intuit/karate/wiki/How-to-Submit-an-Issue

[kraul25]

Nothing special that I am doing... This could be reproduced with any existing karate projects available. . I am using parallel runner as: https://github.com/intuit/karate/blob/master/karate-demo/src/test/java/demo/DemoTestParallel.java... Upto you to fix it or not or keep it closed/invalid. Difficult for me to create a new project and upload as I am using office machine only which blocks everything. Again will not push for you to fix this.

[ptrthomas]

@kraul25 we will look at this in #1281 but no guarantees without a way to replicate. I do wish more people contributed code instead of complaining, but that's just me