Spock: Wrong total number of tests - Parameterized Tests with table

Created on 26 Feb 2021  路  4Comments  路  Source: spockframework/spock

Expected is 2 runs.
Actual is 3.

Problem is with the version 2.0-M4-groovy-3.0 reports 3 runs - wrong.
The old version 2.0-M2-groovy-3.0 reports 2 runs.

The where clause defines two data sets i.e. (1, 2) and (-5, 2), not three.

        where:
        a  | b || expectedResult
        1  | 2 || 3
        -5 | 2 || -3

I am Apache developer of the Surefire project and the Spock with JUnit5 appears in our integration tests and the documentation.
Due to the Spock was not capable to run on JDK 15+ last year, I have decided to upgrade the version of Spock and found this issue.
Updated the Spock Core and the integration test failed because the total number of runs of the test was three instead of two (with version 2.0-M4-groovy-3.0). Upgrade the version of spock-core.
https://github.com/apache/maven-surefire/tree/master/surefire-its/src/test/resources/junit5-spock

Environment: Windows, OpenJdk 14 and OpenJdk 15.

Most helpful comment

Pls let me dig into this problem. We have some plan in the Surefire roadmap which has the priority but I have to say that the next version 3.0.0-M6 is the one which is going to rework the XML report and this issue can be included there. Not sure it is doable, we will see soon.

All 4 comments

Three is the correct result in JUnit 5 Platform semantics.
There are three kinds of nodes in the hierarchy, CONTAINER, TEST and CONTAINER_AND_TEST.
The latter should be used when test nodes are generated programmatically as is the case for data-driven features. This is especially important when errors happen during the generation of the test cases, as with CONTAINER the error would not be reported properly and in a way a user can locate the problem properly.
So you have the feature itself as CONTAINER_AND_TEST and then the two iterations as TEST which then results in three.

These are two different views. One is the technical how the JUnit5 works and the second is the report needed by Surefire.
We can consider this as a bug in Surefire and suppress container messages, or use it but then the Total number of tests should be 2 and the third report should belong to container/class which means that the XML report will be extended somehow.

But this is more like a general JUnit 5 Platform support topic, than a Spock specific topic I'd say.
Spock just provides a JUnit 5 Engine like it is supposed to do. :-)

Pls let me dig into this problem. We have some plan in the Surefire roadmap which has the priority but I have to say that the next version 3.0.0-M6 is the one which is going to rework the XML report and this issue can be included there. Not sure it is doable, we will see soon.

Was this page helpful?
0 / 5 - 0 ratings