Junit5: Define a new XML report schema that supports the JUnit Platform

[sbrannen]

Overview

The XmlReportWriter in the junit-platform-console currently generates XML that is compliant with the de facto standard for JUnit 4; however, this XSD is very limiting and does not support many of the features of the JUnit Platform.

Goals

Define a new XML report schema that supports the features of the JUnit Platform.

The following is only a partial list of features needed beyond the JUnit 4 de facto standard.

• unique IDs
• display names
• tags
• test sources
• custom reporting entries
• nested test classes
• skipped vs. aborted execution

Additional Considerations

• Provide a mechanism for converting between the JUnit 4 XSD and JUnit Platform XSD (e.g., via XSLT).
• Provide a JSON based format compatible with the XML format.

Related Discussions

• https://github.com/ota4j-team/opentest4j/issues/9
• #86
• #350
• #372

[jensim]

I thought this might be a good place to come with some input on the current state of the junit5 report schema.

The surefire-reports are not valid in junit5.
We had a problem running JUnit plugins in jenkins because the XML validation would fail.
The difference between junit4 and junit5 is the XML tag "xsi:noNamespaceSchemaLocation" has changed to "xsi:schemaLocation".
I've manually validated both junit4 and junit5 reports, and junit4 succeeds where junit5 reports fail.

[marcphilipp]

How did you generate the XML reports? Using Maven or using Gradle/ConsoleLauncher?

[marcphilipp]

Using Gradle/ConsoleLauncher seems to work fine for us:
https://junit.ci.cloudbees.com/job/JUnit5/job/master/91/testReport/

[jensim]

Maven
I made a focused project to showcase this
https://github.com/jensim/junit5-bad-report

[marcphilipp]

@jensim The XML report is generated by the maven-surefire-plugin, not by JUnit. Did you also switch to a later version of the maven-surefire-plugin?

[jensim]

I was starting out from scratch
Now I've tested all released versions of the surefire plugin against junit-5.0.1.
I've discovered that the ONLY version this will both not crach and build a valid report is 2.20.
Version 2.20.1 craches.
Version 2.19.x are invalid. (surefires 'fault')
Versions prior to 2.19.x craches.

The good thing is, there is one version that is compatible with junit5, yay!

[jensim]

Anyway, this is nolonger a matter of XML formatting, which was the reason i brought it up in this thread from the first place.
Thank you @marcphilipp

[marcphilipp]

As stated in the User Guide, Surefire 2.20 has a memory leak. But as long as it works for you, I guess it's okay. 😉

[mkobit]

As of right now, is the only way to consume the ExtensionContext.publishReportEntry from extensions/what not is to implement a TestExecutionListener? Can that be used from Gradle?

[sbrannen]

As of right now, the only way to consume the ExtensionContext.publishReportEntry from extensions/what not is to implement a TestExecutionListener?

Yes, AFAIK, that's correct.

Can that be used from Gradle?

AFAIK, you cannot register a custom TestExecutionListener within a Gradle build script, but @marcphilipp can perhaps correct me if I'm wrong.

In any case, you can have your custom TestExecutionListener registered automatically. Details here: https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom

[marcphilipp]

Gradle has its own TestListener that you can register on a Test task: https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:addTestListener(org.gradle.api.tasks.testing.TestListener)

It does not currently report events for intermediate nodes in the test tree, though.

[bitcoder]

@marcphilipp , how are you? hope fine :)
Any inputs on this? As you mentioned that adding support for this is dependent on maven surefire supporting it (also mentioned on #1630).. I guess that you need to somehow also modify JUnit's code in order to pass that info into surefire, right?
We wouldn't really wanted to write a custom TestExecutionListener which would create a JUnit XML report with some additional custom tags.. I don't think it makes sense, don't you agree?
Thus, what is the best way to move forward? :)

1. are you aiming to build anytime soon a new XML report schema, more complete, that support additional information that is currently missing in the current JUnit XML report?
2. are you willing to enhance the current JUnit XML report and workout the remainings in the surefire integration so that is supports additional information, such as the tags?
3. or you don't want to evolve this at all?

Note: we have many customers using JUnit interested on overcoming the current limitation, which as I've said does not happen with other competitor frameworks.

[marcphilipp]

@bitcoder Yes, we still want to define a new schema that supports all Platform features. It's probably not going to make 5.5 but will be our top priority for 5.6.

[bitcoder]

thanks @marcphilipp for the feedback; that means that you will be able to make whatever changes will be necessary on the surefire plugin, if necessary, in order to generate that report?

[marcphilipp]

We'll work with the Surefire team to make it happen.

[bitcoder]

Then, from "Xray Test Management for Jira" team side will make the integration happen ASAP right afterwards :)

[tobymurray]

Reading this today, ironically while trying to integrate with XRay, is it accurate to say there's no out-of-the-box way to have @DisplayName contents appear in the XML produced by a JUnit 5 execution? I can use reflection and log it to system out, which is captured in <system-out>, but so is a lot of other stuff.

[bitcoder]

There are so many people waiting for a new, more complete format; I hope it comes soon so we can adopt it and bring it to everyone.
Eventually @tobymurray you may need to have a custom report to have it that would duplicate the existing Junit XML generation code (?) but probably a better solution is to wait for the format. Maybe @marcphilipp and the team can provide more insights at this point

[baev]

@tobymurray for now you can try use Allure as middleware.

[tobymurray]

I've never looked at Allure before (assuming it's this). I'll see what it offers for what I'm looking at - thanks for the tip. I'll update if it comes together with something usable that might be workable for others.

[baev]

I've never looked at Allure before (assuming it's this).

yep

We even have a plugin for XRay https://github.com/allure-framework/allure2/tree/master/plugins/xray-plugin

As a bonus you'll not only get support for JUnit5 tests, but we also provide integrations for majority of test frameworks (not only for Java, but also for Python, JavaScript, C# and others).

Advertising

Allure has paid version https://qameta.io/ that extends open source report and integrations, and it also provides XRay integration.

[adrian-skybaker]

Please consider a clear mechanism to link to attachments. This is supported in NUnit, as well as Allure 2.

[bitcoder]

Unfortunately, this keeps getting postponed :( we have many customers asking for this kind of capability

[marcphilipp]

Any arguments against using JSON instead of XML for the new format?

[bitcoder]

Hi @marcphilipp , we don't see any problems with adopting JSON instead of XML. For us, it will be a new format we have to support and if it's more flexible/easier in the future to make changes in it, ok.
As long as JUnit team provides a kinda of schema for it, it's ok 👍 We've seen some JSON formats out there that don't have a clear definition which in the end leads to some confusion around implementations.

[signed]

Would be nice to have support for #271 in the report as well.

Any arguments against using JSON instead of XML for the new format?

If you think about changing the format did you consider yaml?

[bitcoder]

@signed , yaml is not suited for reporting IMO.