Codeception: The report.xml file structure provided by phpunit and codeception is different

[sapzape]

The format of the report.xml file obtained after testing with phpunit differs from the report.xml file obtained after code execution.

report sample after running phpunit

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="75" assertions="164" failures="0" errors="0" time="25.978013">
    <testsuite name="Feature Tests" tests="60" assertions="149" failures="0" errors="0" time="24.027709">
      <testsuite name="Tests\Feature\AgeCategoryTest" file="/home/travis/build/tititoof/sas_judo/tests/Feature/Admin/AgeCategoryTest.php" tests="6" assertions="19" failures="0" errors="0" time="2.349255">
        <testcase name="testIndexStatus" class="Tests\Feature\AgeCategoryTest" file="/home/travis/build/tititoof/sas_judo/tests/Feature/Admin/AgeCategoryTest.php" line="40" assertions="2" time="0.416466"/>
        <testcase name="testStoreStatus" class="Tests\Feature\AgeCategoryTest" file="/home/travis/build/tititoof/sas_judo/tests/Feature/Admin/AgeCategoryTest.php" line="50" assertions="6" time="0.503893"/>
        ...
      </testsuite>
      <testsuite name="Tests\Feature\AlbumTest" file="/home/travis/build/tititoof/sas_judo/tests/Feature/Admin/AlbumTest.php" tests="5" assertions="13" failures="0" errors="0" time="3.937930">
        <testcase name="testIndexAction" class="Tests\Feature\AlbumTest" file="/home/travis/build/tititoof/sas_judo/tests/Feature/Admin/AlbumTest.php" line="28" assertions="2" time="0.333698"/>
        ...
      </testsuite>
      ...
  </testsuite>
</testsuite>

report sample after running codeception

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="unit" tests="41" assertions="93" errors="0" failures="0" skipped="0" time="0.285144">
        <testcase file="/home/www/test/test/tests/unit/case/model/modelTagUnitCest.php" name="testGetById" class="modelTagUnitCest" assertions="2" time="0.045005"/>
        <testcase file="/home/www/test/test/tests/unit/case/model/modelTagUnitCest.php" name="testGetAll" class="modelTagUnitCest" assertions="1" time="0.001119"/>
        <testcase file="/home/www/test/test/tests/unit/case/model/modelTagUnitCest.php" name="testInsert" class="modelTagUnitCest" assertions="1" time="0.003246"/>
        <testcase file="/home/www/test/test/tests/unit/case/model/modelTagUnitCest.php" name="testUpdate" class="modelTagUnitCest" assertions="2" time="0.032069"/>
    <testcase file="/home/www/test/test/tests/unit/case/service/Reports/serviceReportsRunUnitCest.php" name="testUpdateReportVersion" class="serviceReportsRunUnitCest" assertions="2" time="0.085060"/>
        <testcase file="/home/www/test/test/tests/unit/case/service/Tags/serviceTagsCommonUnitCest.php" name="testGetAllTag" class="serviceTagsCommonUnitCest" assertions="2" time="0.017860"/>
        <testcase file="/home/www/test/test/tests/unit/case/service/Tags/serviceTagsCommonUnitCest.php" name="testUpdateTag" class="serviceTagsCommonUnitCest" assertions="2" time="0.001434"/>
        ...
    </testsuite>
</testsuites>

When executed with codeception, the entire result is output as a unit under suite unit.
I am trying to link the test result to sonarqube, but it seems that the format is wrong and the failure occurs.
If I am using something wrong, I would like to ask phpunit what to do to get the results.

Below is the configuration information.

• Codeception version: 2.4.1
• PHP Version: 7.0.22
• Operating System: Window 10, CentOS 7.4.1708
• Installation type: Composer
• Suite configuration:

paths:
    tests: tests
    output: tests/_output
    data: tests/_data
    support: tests/_support
    envs: tests/_envs
    log: tests/_output
actor_suffix: Tester
coverage:
    enabled : true
    include:
      - cont/*
    # url of file which includes c3 router.
    remote_context_options:
        ssl:
            verify_peer: false
settings:
  bootstrap: bootstrap.php
  log: true
  strict_xml: true
extensions:
    enabled:
        - Codeception\Extension\RunFailed

[lightswitch05]

I'm having the same issue when importing into sonarqube.

WARN: Test cases must always be descendants of a file-based suite, skipping : ClientCest.itCreatesANewFolder in functional

Manually setting the file attribute on the testsuite element allows sonarqube to parse and show the results. In the above example, you can see that it generates a new testsuite element per file and nests them. So an equivalent version would keep the parent <testsuite name="unit" element, but then breakup children testcase elements into more testsuite elements with file attribute being the only attribute.

For example, this codeception results:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="functional" tests="15" assertions="26" failures="0" errors="0" time="7.850769">
        <testcase file="/code/tests/functional/ClientCest.php" name="itDoesNotAlreadyHaveTheNewFolder" class="ClientCest" feature="it does not already have the new folder" assertions="5" time="1.216989"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itCreatesANewFolder" class="ClientCest" feature="it creates a new folder" assertions="1" time="1.339019"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itListsTheNewFolder" class="ClientCest" feature="it lists the new folder" assertions="2" time="0.449247"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itUploadsAFile" class="ClientCest" feature="it uploads a file" assertions="1" time="2.104713"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itListsTheNewFile" class="ClientCest" feature="it lists the new file" assertions="2" time="0.297534"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itSearchesTheNewFile" class="ClientCest" feature="it searches the new file" assertions="2" time="0.425515"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itDoesNotFindFakeFiles" class="ClientCest" feature="it does not find fake files" assertions="2" time="0.273783"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itDeletesTheNewFolder" class="ClientCest" feature="it deletes the new folder" assertions="1" time="0.750843"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itNoLongerHasTheNewFolder" class="ClientCest" feature="it no longer has the new folder" assertions="4" time="0.925299"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNames" class="ClientCest" feature="it sanitizes file names" assertions="1" time="0.008238"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNamesForHiddenFiles" class="ClientCest" feature="it sanitizes file names for hidden files" assertions="1" time="0.007309"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNamesWithMultipleDots" class="ClientCest" feature="it sanitizes file names with multiple dots" assertions="1" time="0.006771"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFolderNames" class="ClientCest" feature="it sanitizes folder names" assertions="1" time="0.007201"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itBlocksWindowsFileNames" class="ClientCest" feature="it blocks windows file names" assertions="1" time="0.031064"/>
        <testcase file="/code/tests/functional/ClientCest.php" name="itBlocksWindowsFileNamesWithExtensions" class="ClientCest" feature="it blocks windows file names with extensions" assertions="1" time="0.007244"/>
    </testsuite>
</testsuites>

into

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="functional" tests="15" assertions="26" failures="0" errors="0" time="7.850769">
    <testsuite file="/code/tests/functional/ClientCest.php" time="7.850769">
      <testcase file="/code/tests/functional/ClientCest.php" name="itDoesNotAlreadyHaveTheNewFolder" class="ClientCest" feature="it does not already have the new folder" assertions="5" time="1.216989"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itCreatesANewFolder" class="ClientCest" feature="it creates a new folder" assertions="1" time="1.339019"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itListsTheNewFolder" class="ClientCest" feature="it lists the new folder" assertions="2" time="0.449247"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itUploadsAFile" class="ClientCest" feature="it uploads a file" assertions="1" time="2.104713"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itListsTheNewFile" class="ClientCest" feature="it lists the new file" assertions="2" time="0.297534"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itSearchesTheNewFile" class="ClientCest" feature="it searches the new file" assertions="2" time="0.425515"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itDoesNotFindFakeFiles" class="ClientCest" feature="it does not find fake files" assertions="2" time="0.273783"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itDeletesTheNewFolder" class="ClientCest" feature="it deletes the new folder" assertions="1" time="0.750843"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itNoLongerHasTheNewFolder" class="ClientCest" feature="it no longer has the new folder" assertions="4" time="0.925299"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNames" class="ClientCest" feature="it sanitizes file names" assertions="1" time="0.008238"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNamesForHiddenFiles" class="ClientCest" feature="it sanitizes file names for hidden files" assertions="1" time="0.007309"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFileNamesWithMultipleDots" class="ClientCest" feature="it sanitizes file names with multiple dots" assertions="1" time="0.006771"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itSanitizesFolderNames" class="ClientCest" feature="it sanitizes folder names" assertions="1" time="0.007201"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itBlocksWindowsFileNames" class="ClientCest" feature="it blocks windows file names" assertions="1" time="0.031064"/>
      <testcase file="/code/tests/functional/ClientCest.php" name="itBlocksWindowsFileNamesWithExtensions" class="ClientCest" feature="it blocks windows file names with extensions" assertions="1" time="0.007244"/>
    </testsuite>
  </testsuite>
</testsuites>

Notice the extra testsuite with the file attribute set. One note, I also had to add the time attribute to the extra testsuite element because it was causing a null pointer exception in the sonarqube scanner, but that issue should be fixed on their side

[telefoontoestel]

Up-voted. Same issue when importing with Infection mutation tests.

[mikacalvo]

does anybody have a workaround ? Will it be fixed soon ?

[Naktibalda]

Implementing file-based test suites requires quite major change to a way Codeception executes tests.

Also Codeception shuffles tests by default, it would result in almost every test being in a separate file-based testsuite with non-unique filename and it would probably cause other issues for Sonarqube.

How about fixing Sonarqube instead?

[Naktibalda]

I had second thoughts.

We could implement solution specific to JUnit logger.

It would keep the current filename in a property and it would close and start file-based test-suites when the filename changes.

[telefoontoestel]

I believe something like the latest proposal would fix the issue. The problem isn't with the execution of the test rather than that the junit logfile has a different format that cannot be handled by several other applications.

[Naktibalda]

This issue was fixed in Codeception 2.5.2, please upgrade and run Codeception with parameter --phpunit-xml=report.xml instead of --xml.

[telefoontoestel]

I've just tested the issue. In my use case I tried to run codecept generating the report. I wanted to use the report with the php infection package, but it unfortunately still failed. I've pinpointed the problem to the name of each test suite that seems to be missing:

<testsuite file="/path/to/tests/unit/MyTest.php" tests="1" assertions="1" errors="0" failures="0" skipped="0" time="0.001">

should be

<testsuite name="tests\unit\MyTest" file="/path/to/tests/unit/MyTest.php" tests="1" assertions="1" errors="0" failures="0" skipped="0" time="0.001">

When I manually add this name attribute to the testsuite items the file is processed correctly.

[Naktibalda]

Must the name match the path?
Does setting name to MyTest work?

[telefoontoestel]

I've looked into it and it seems that the name attribute contains the fully qualified class name (FCQN) to the test. So it contains the namespace with the classname.

[Naktibalda]

Using namespace tests\unit; is quite unusual.

[telefoontoestel]

I know. It was not my idea to have them named liked this in our codebase. (tests\unit\*)

[Naktibalda]

Another problem - Cept format has no class name per file.

[telefoontoestel]

Can't it be retrieved by get the fcqn from the test class that is being run at that moment, or are you referring to a template in which you should add this attribute?

[Naktibalda]

@telefoontoestel I released a new version of codeception/phpunit-wrapper, please update and try again.

[telefoontoestel]

I've tested the update and it is working as expected now. The file structure is the same as the original phpunit.

[Naktibalda]

Great, thanks for confirmation.