Phpunit: Junit logger produces report which Jenxins xunit-plugin considers invalid.

[bdsl]

| Q | A
| --------------------| ---------------
| PHPUnit version | 8.4.2
| PHP version | 7.3
| Installation Method | Composer

Summary

Report generated with --log-junit includes an attribute 'warnings' of element 'testsuite'. According to https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/phpunit-2.0.xsd#L89 this is not an allowed attribute.

Depending on build config, this can cause a build failure in Jenkins, with a message like the following showing up in the log:

At line 3 of file:/var/lib/jenkins/workspace/myproject/build/logs/report.xml:cvc-complex-type.3.2.2: Attribute 'warnings' is not allowed to appear in element 'testsuite'

However I'm not sure if there is any really authoritative xsd for the junit format available, so potentially this could something to fix in the xunit-plugin instead or as well as fixing here. I have also made a PR to xunit-plugin.

This issue is new in PHPUnit 8.4.2 , apparently introduced in https://github.com/sebastianbergmann/phpunit/commit/e7b8576b0588191869d62d8c354126cf87f15169 .

Current behavior

report.xml file is generated with line as shown below:
<testsuite name="Test Suite" tests="201" assertions="221" errors="0" warnings="0" failures="0" skipped="0" time="0.498413">

How to reproduce

Run ./bin/phpunit --log-junit report.xml

Expected behavior

report.xml to be generated without the warnings attribute, as in PHPUnit 8.4.1:

<testsuite name="Test Suite" tests="201" assertions="221" errors="0" failures="0" skipped="0" time="0.498413">

[sebastianbergmann]

The "X" in "XML" stands for "Extensible". This means, at least IMO, that any consumer of the JUnit XML produced by PHPUnit must ignore attributes it does not handle.

There is no (official) standard for JUnit XML and there is no official XSD for JUnit XML. See #3162 for more details. I am sick and tired of this situation and am strongly thinking about abandoning "JUnit XML" and instead create PHPUnit's XML logfile format for test results.

[sebastianbergmann]

CC @celeryclub @DanielRuf

[theseer]

Technically, we already have that with the PHPUnit coverage xml.

[sebastianbergmann]

Yes, we "just" need to make it possible to create a logfile using that format when no code coverage is collected/processed.

[theseer]

When @localheinz and I are done with the Event subsystem, we can have a look at that :)

[bdsl]

Thanks @sebastianbergmann.

For now it looks like our options are:

1. Untick _Post Build Actions_ / _publish xUnit test result report_ / _Stop and set the build to 'failed' status if there are errors when processing a result file_ in Jenkins job config.
2. Pin PHPUnit to 8.4.1

[bdsl]

The PR to xunit-plugin has been merged, so this should be fixed in a future release of that.

[markdwhite]

Just a nudge to notify watchers that the xunit-plugin has been updated and this is now working for me using phpunit 8.5.2

[lstrojny]

@sebastianbergmann what do you think about having an integration test against Jenkins XSDs? https://github.com/jenkinsci/xunit-plugin/tree/master/src/main/resources/org/jenkinsci/plugins/xunit/types

[sebastianbergmann]

I am tired of "playing catchup" with various "forks" of JUnit XML. Just because the Jenkins CI project has an XSD does not make this XSD official. There is no "official JUnit XML". This has been discussed in #3162, for instance.

To be honest, I would like to eventually deprecate and remove PHPUnit's ability to generate "JUnit XML" logfiles once #4321 has been implemented.