Azure-pipelines-tasks: Publish Test Result: JUnit result with multiple test suites not displayed properly

[domenico404]

Hi there,

we're encountering problems using the "Publish test results" tasks under following environment:

• TFS on-premises: 15.117.27024.0
• Private Agent: 2.122.1 running on Windows 7

The test results file (JUnit) contains multiple testsuite elements, which again can contain multiple test cases. The publishing runs without exception. But the "Tests" overview does not group the results by testsuite. Instead it produces a single "parent node", which contains all testcases of all testsuites.

<?xml version="1.0" encoding="utf-8"?> <testsuites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <testsuite name="Testsuite1" tests="2" failures="2" errors="0" time="0" timestamp="2018-07-06T14:27:44.3258474+02:00" id="0"> <testcase name="Case1" time="0"> <failure>Fail1</failure> </testcase> <testcase name="Case2" time="0"> <failure>Fail2</failure> </testcase> </testsuite> <testsuite name="Testsuite2" tests="2" failures="2" errors="0" time="0" timestamp="2018-07-06T14:27:44.3258474+02:00" id="1"> <testcase name="Case1" time="0"> <failure>Fail1</failure> </testcase> <testcase name="Case2" time="0"> <failure>Fail2</failure> </testcase> </testsuite> </testsuites>

The image shows the single parent named "Unspecified". I would expect two test suites (Testsuite1 and Testsuite2).

Greetings

[divais]

Hi,

We don't support Test suites today and thus you see this. Can you please clarify if having test suites as tags meet your need to filter results by Test suites?

[domenico404]

Hi,

I've found a workaround for this "problem". If I put a single inside a -tag, it'll work (when grouped by test run). But I have to produce a single file for each testsuite I want to create.

It would help avoiding this workaround (and multiple result files), if I were able to put multiple inside a -tag.

[shanehsu]

Is this going to be supported in the future, or is it on the roadmap?

[kachkaev]

Same problem here. If junit.xml looks like this, testsuite name is ignored.

<testsuites name="jest tests" tests="123" failures="0" time="42">
  <testsuite name="suite 1" errors="0" failures="0" skipped="0" timestamp="2018-12-20T18:58:06" time="6" tests="3">
    <testcase classname="a" name="a" time="1">
    </testcase>
    <testcase classname="b" name="b" time="2">
    </testcase>
    <testcase classname="c" name="c" time="3">
    </testcase>
  </testsuite>
  <testsuite name="suite 2" errors="0" failures="0" skipped="0" timestamp="2018-12-20T18:58:12" time="15" tests="3">
    <testcase classname="x" name="x" time="4">
    </testcase>
    <testcase classname="y" name="y" time="5">
    </testcase>
    <testcase classname="z" name="z" time="6">
    </testcase>
  </testsuite>
</testsuites>

[kachkaev]

Real-world public example: https://www.npmjs.com/package/prettier-plugin-elm

Contents of junit.xml:

<testsuites name="jest tests" tests="18" failures="0" time="18.103">
  <testsuite name="/dist/cache.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="14.348" tests="1">
    <testcase classname=" correctly deals with cache" name=" correctly deals with cache" time="13.494">
    </testcase>
  </testsuite>
  <testsuite name="/dist/index.test.js" errors="0" failures="0" skipped="0" timestamp="2018-12-23T11:16:25" time="16.507" tests="17">
    <testcase classname=" formats fixture broken-code.elm" name=" formats fixture broken-code.elm" time="0.902">
    </testcase>
    <testcase classname=" formats fixture broken-code.md" name=" formats fixture broken-code.md" time="0.547">
    </testcase>
    ...
    <testcase classname=" formats fixture only-import.md" name=" formats fixture only-import.md" time="0.417">
    </testcase>
  </testsuite>
</testsuites>

Task:

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: JUnit
      testResultsFiles: $(System.DefaultWorkingDirectory)/junit.xml
    displayName: "Publish test results"
    condition: and(variables.REPORT_TEST_RESULTS_AND_COVERAGE, succeededOrFailed())

Result (test suites are joined, total duration is broken):

HALP 🙏

[divais]

      Is this going to be supported in the future, or is it on the roadmap?

Yes its on the roadmap but not this quarter. It will be of great help if you can request for this feature @ https://developercommunity.visualstudio.com/content/idea/post.html?space=21 to help us prioritize it.

[iamarno]

@divais
I still have the same issue. Test suite names are just ignored, only test cases are shown when publishing the test result. Is there any ETA when this will be available?

[lindhe]

But it clearly says in your official documentation that you support test suites (e.g., under Test Results > Outcome):

Failed: if exists /Testsuites/testsuite/testcase/failure

[Additi]

Hi ,
We have considered this feedback and added this to our backlog. We'll take it up in the coming months.
I'll post the updates once its done.
Thanks.

[IanKemp]

Hi ,
We have considered this feedback and added this to our backlog. We'll take it up in the coming months.
I'll post the updates once its done.
Thanks.

Any update? It's nonsensical that named testcases are supported but named testsuites and/or testsuites are not!

[fdcds]

https://github.com/gotestyourself/gotestsum generates JUnit XML files that look like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite tests="3" failures="0" time="1.910000" name="full/package/name">
        <properties>
            <property name="go.version" value="go1.13.7 linux/amd64"></property>
        </properties>
        <testcase classname="full/package/name" name="TestA" time="0.130000"></testcase>
        <testcase classname="full/package/name" name="TestB" time="0.550000"></testcase>
        <testcase classname="full/package/name" name="TestC" time="1.230000"></testcase>
    </testsuite>
    <testsuite tests="3" failures="0" time="1.850000" name="full/package/othername">
        <properties>
            <property name="go.version" value="go1.13.7 linux/amd64"></property>
        </properties>
        <testcase classname="full/package/othername" name="TestA" time="0.130000"></testcase>
        <testcase classname="full/package/othername" name="TestB" time="0.400000"></testcase>
        <testcase classname="full/package/othername" name="TestC" time="1.320000"></testcase>
    </testsuite>
</testsuites>

full/package/name and full/package/othername do not appear in the visual hierarchy displayed by Azure DevOps Pipelines. Instead it uses a flattened representation that shows only the JUnit XML filename and the name of the test (which can be ambiguous, as in my example).

This seems to contradict the documentation, which says that the "Test file" is taken from the /testsuites/testsuite/testcase/Attributes["classname"].Value element in the JUnit XML: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml#result-formats-mapping

[vercellone]

PublishTestResultsV1 outputs properly for me, but not PublishTestResultsV2.

[apis3445]

I want this feature working correctly, the name is not useful

[resried]

Any news, when this is planned?