Azure-pipelines-tasks: PublishTestResults

[ColinLaws]

This issue exists on AzureDevops

I am running tests in PHPUnit and logging the test to a JUnit xml file. My build pipeline is setup to publish test results using the PublishTestResults task.

Here is my XML file generated by PHPUnit in the JUnit format:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="2" assertions="4" errors="0" failures="0" skipped="0" time="1.765531">
    <testsuite name="Unit Tests" tests="2" assertions="4" errors="0" failures="0" skipped="0" time="1.765531">
      <testsuite name="Tests\Unit\DatabaseServiceTest" file="/home/vagrant/src/Website/tests/unit/DatabaseServiceTest.php" tests="2" assertions="4" errors="0" failures="0" skipped="0" time="1.765531">
        <testcase name="a_database_team_member_can_be_added" class="Tests\Unit\DatabaseServiceTest" classname="Tests.Unit.DatabaseServiceTest" file="/home/vagrant/src/Website/tests/unit/DatabaseServiceTest.php" line="35" assertions="2" time="1.656353"/>
        <testcase name="a_database_team_member_can_be_removed" class="Tests\Unit\DatabaseServiceTest" classname="Tests.Unit.DatabaseServiceTest" file="/home/vagrant/src/Website/tests/unit/DatabaseServiceTest.php" line="57" assertions="2" time="0.109178"/>
      </testsuite>
    </testsuite>
  </testsuite>
</testsuites>

Here is my build pipeline YAML:

# https://docs.microsoft.com/azure/devops/pipelines/languages/php
pool:
  vmImage: 'Ubuntu 16.04'

variables:
  phpVersion: 7.2

steps:
- script: |
    sudo update-alternatives --set php /usr/bin/php$(phpVersion)
    sudo update-alternatives --set phar /usr/bin/phar$(phpVersion)
    sudo update-alternatives --set phpdbg /usr/bin/phpdbg$(phpVersion)
    sudo update-alternatives --set php-cgi /usr/bin/php-cgi$(phpVersion)
    sudo update-alternatives --set phar.phar /usr/bin/phar.phar$(phpVersion)
    php -version
  displayName: 'Use PHP version $(phpVersion)'

- script: composer install --no-interaction --prefer-dist
  displayName: 'composer install'

- script: npm install
  displayName: 'npm install'

- script: npm run dev
  displayName: 'run mix (webpack)'

- script: phpunit --log-junit test-results.xml
  displayName: 'Run tests with phpunit'

# Publish Test Results
# Publish Test Results to Azure Pipelines/TFS
- task: PublishTestResults@1
  inputs:
    testRunner: 'JUnit'
    testResultsFiles: 'test-results.xml' 

I receive the following error when the PublishTestResults task runs:

[warning]Invalid results file. Make sure the result format of the file '/home/vsts/work/1/s/test-results.xml' matches 'JUnit' test results format.

[SadagopanRajaram]

Thank you for the feedback. Currently, the JUnit Parser does not support nested test suites in the XML. I am not completely familiar with php unit tests but is there a way for you not to nest test suites in the tests file?

[ColinLaws]

The PHPUnit binary accepts the --log-junit option, which outputs my test results in the nested fashion that the JUnit parser does not support. I have found a way to make this work in Azure DevOps (albeit not the best solution, but I got it working).

To make PHPUnit test results work in Azure DevOps with the current JUnit parser, I processed PHPUnit's output using an XSL transform.

To perform the XSL transform on my build agent in Azure DevOps, I downloaded the Java Archive (.jar) file for Saxon 9 Home Edition and included that in my project's source code. In my build pipeline, I run the XSL transform by calling the java binary and passing the saxon9he.jar file with the appropriate options for pointing it to my test results XML output by PHPUnit.

This is how my build pipeline is configured for running the Saxon 9 jar file.

# Navigate to the saxon folder and run the saxon jar file to transform the test results
# into a format that DevOps can consume
- script: cd saxon && java -jar saxon9he.jar -xsl:phpunit_to_junit.xsl -s:test-results.xml
  displayName: 'Perform XSL transform on test results'

# Publish Test Results to Azure Pipelines/TFS
- task: PublishTestResults@2
  displayName: 'Publish test results'
  inputs:
    testRunner: 'JUnit'
    testResultsFiles: '**/*-Test.xml' 
    searchFolder: '$(System.DefaultWorkingDirectory)'
    mergeTestResults: false

# Cleaning up the build by removing the saxon directory that contains the test
# results and the saxon 9 JAR file, as those files are not required for release.
- script: rm -r saxon
  displayName: 'Remove saxon and test results from the build'

This is an XSL transform file that formats PHPUnit output in a way that DevOps can parse it:

<?xml version="1.0" encoding="UTF-8"?>

<xsl:stylesheet
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="2.0">

<xsl:output method="xml"/>

<xsl:template match="/">
<xsl:for-each select="//testsuite[@file]">
    <xsl:variable name="filename" select="concat('TEST-',@name,'-Test.xml')" />
    <xsl:result-document href="{$filename}" method="xml">
        <xsl:copy-of select="." />
    </xsl:result-document>
</xsl:for-each>
</xsl:template>

</xsl:stylesheet>

[zanechua]

Thanks @ColinLaws for spending the effort to write the transform file.

I don't really like java and would rather not have to download and place a jar file in my repository so I came up with a php based solution since it's guaranteed that we would have php installed. It's a bit rough on the edges since I did it in like two hours last night but here it is.

This is specifically built for PHPUnit and Laravel Dusk, so if you notice it checks if a filename has a "dusk" keyword inside and it'll be of a different format.

My current yaml has the following:

- script: |
    vendor/bin/phpunit --log-junit TEST-phpunit-junit.xml
  displayName: 'Running Unit Tests'

- script: |
    php artisan dusk --log-junit TEST-dusk-junit.xml
  displayName: 'Running Browser Tests'

- script: |
    php parseJUnitToAzureJUnit.php TEST-phpunit-junit.xml TEST-phpunit-azure-junit.xml
    php parseJUnitToAzureJUnit.php TEST-dusk-junit.xml TEST-dusk-azure-junit.xml
  displayName: 'Parsing JUnit Files to Azure JUnit Supported Format'

# Publish Test Results to Azure Pipelines/TFS
- task: PublishTestResults@2
  inputs:
    testRunner: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
    testResultsFiles: '**/TEST-*.xml'
    searchFolder: '$(System.DefaultWorkingDirectory)/tests/Results' # Optional
    mergeTestResults: false # Optional

Usage:

php parseJUnit.php TEST-file.xml TEST-file-azure.xml

Here it is:

<?php

if (defined('STDIN')) {
    $inputFileName = $argv[1];
    $outputFileName = $argv[2];
}

$directory = './tests/Results/';

$data = simplexml_load_file($inputFileName);

$json_string = json_encode($data);

$result_array = json_decode($json_string, true);

if (strpos($inputFileName, 'dusk') !== false) {
    $mainResults = $result_array;
} else {
    $mainResults = $result_array['testsuite']['testsuite'];
}

$domCollection = flatten_xml_element($mainResults);

$outputFilePieces = explode(".", $outputFileName);
$outputFileNameOnly = $outputFilePieces[0];
$outputFileExtension = $outputFilePieces[1];

foreach ($domCollection as $key => $domSingle) {
    outputToFile($domSingle, $directory, $outputFileNameOnly . '-' . $key . '.' . $outputFileExtension);
}
//unlink($inputFileName);

function outputToFile($dom, $directory, $filename)
{
    if (!file_exists($directory)) {
        mkdir($directory, 0777, true);
    }
    $dom->encoding = 'UTF-8';
    $dom->formatOutput = true;
    $xml = $dom->saveXML();
    file_put_contents($directory . $filename, $xml);
}

function flatten_xml_element($dataArray)
{
    $domArray = [];

    foreach ($dataArray as $data) {
        //        $element = $dom->createElement('testsuite', '');
        //        // Add any @attributes
        //        if (! empty($data['@attributes']) && is_array($data['@attributes'])) {
        //            foreach ($data['@attributes'] as $attribute_key => $attribute_value) {
        //                $element->setAttribute($attribute_key, $attribute_value);
        //            }
        //        }
        //
        //        $dom->appendChild($element);

        //Check for Single Element only, otherwise assume multiple elements
        if (array_key_exists('testsuite', $data) && array_key_exists('@attributes', $data['testsuite']) && array_key_exists('testcase', $data['testsuite'])) {
            $dom = new DOMDocument();
            $testsuite = $data['testsuite'];

            $element = $dom->createElement('testsuite', '');

            // Add any @attributes
            if (! empty($testsuite['@attributes']) && is_array($testsuite['@attributes'])) {
                foreach ($testsuite['@attributes'] as $attribute_key => $attribute_value) {
                    $element->setAttribute($attribute_key, $attribute_value);
                }
            }

            //Loop over internal testcases
            $testcase = $testsuite['testcase'];
            $childElement = $dom->createElement('testcase', '');

            if (! empty($testcase['@attributes']) && is_array($testcase['@attributes'])) {
                foreach ($testcase['@attributes'] as $attribute_key => $attribute_value) {
                    $childElement->setAttribute($attribute_key, $attribute_value);
                }
            }

            $element->appendChild($childElement);
            $dom->appendChild($element);

            array_push($domArray, $dom);
        } else {
            foreach ($data['testsuite'] as $testsuite) {
                $dom = new DOMDocument();
                $element = $dom->createElement('testsuite', '');

                // Add any @attributes
                if (! empty($testsuite['@attributes']) && is_array($testsuite['@attributes'])) {
                    foreach ($testsuite['@attributes'] as $attribute_key => $attribute_value) {
                        $element->setAttribute($attribute_key, $attribute_value);
                    }
                }

                //Loop over internal testcases
                //Check for Single Element only, otherwise assume multiple elements
                if (array_key_exists('testcase', $testsuite) && array_key_exists('@attributes', $testsuite['testcase'])) {
                    $testcase = $testsuite['testcase'];
                    $childElement = $dom->createElement('testcase', '');

                    if (! empty($testcase['@attributes']) && is_array($testcase['@attributes'])) {
                        foreach ($testcase['@attributes'] as $attribute_key => $attribute_value) {
                            $childElement->setAttribute($attribute_key, $attribute_value);
                        }
                    }

                    $element->appendChild($childElement);
                } else {
                    foreach ($testsuite['testcase'] as $testcase) {
                        $childElement = $dom->createElement('testcase', '');

                        if (! empty($testcase['@attributes']) && is_array($testcase['@attributes'])) {
                            foreach ($testcase['@attributes'] as $attribute_key => $attribute_value) {
                                $childElement->setAttribute($attribute_key, $attribute_value);
                            }
                        }

                        if (array_key_exists('failure', $testcase)) {
                            $toddlerElement = $dom->createElement('failure', $testcase['failure']);
                            $childElement->appendChild($toddlerElement);
                        } elseif (array_key_exists('error', $testcase)) {
                            $toddlerElement = $dom->createElement('error', $testcase['error']);
                            $childElement->appendChild($toddlerElement);
                        }

                        $element->appendChild($childElement);
                    }
                }

                $dom->appendChild($element);
                array_push($domArray, $dom);
            }
        }
    }

    return $domArray;
}
?>

[prawalagarwal]

@ColinLaws The changes to support nested test suites in JUnit have been pushed to the PublishTestResults task and will be deployed mid next week.

[zanechua]

Sweet.

So I guess the script I made won't be of much use then.

[ColinLaws]

Awesome, this is going to help a lot of people developing using PHPUnit!

And @zanechua the script you made was still a step up from what I implemented. Ideally we shouldn't be placing a binary in source control, that's just a bad way to do it. Having native support for nested test suites will make it obsolete, but at least we've got it working in the meantime! Also, I can't take credit for the transform file here, I found it somewhere else on the internet and can't remember where.

[prawalagarwal]

Closing this issue. Please reopen if you have any more concerns around this. Thanks!

[zanechua]

Tested the fixed version and it works fine for the junit that Laravel Dusk generates however the phpunit tests failed to publish.

You can see the build here: https://dev.azure.com/invoiceneko/community/_build/results?buildId=297&view=logs

There were no errors and only a simple:
2019-01-01T09:39:36.9156784Z ##[warning]Failed to publish test results: The string must have at least one character.

Would a sample xml of the junit file generated be helpful?

[prawalagarwal]

@zanechua Yes please attach the junit file causing this issue. @vagisha-nidhi please take a look.

[zanechua]

Please see the attached file.

Renamed the xml to log

TEST-phpunit-junit.log

[vagisha-nidhi]

@zanechua The changes to fix the bug have been pushed. The fix will be available with the next release of the agent which you can use to unblock yourself. You can check here for agent release dates.

[zanechua]

@vagisha-nidhi

Sweet. Thanks for getting it sorted.

[ColinLaws]

I don't know if it's worth mentioning here, but the docs for the PublishTestResults task do not reflect this change. It may help to document this for devs reading the docs.

[vagisha-nidhi]

@ColinLaws PHPUnit does not need to be called out in the docs since it uses the existing JUnit results format. We have fixed it on our end for the seamless support.