Coverlet: How to merge several result files together?

[AArnott]

Due to https://github.com/AArnott/Library.Template/issues/2 and the various doc and product bugs linked from it, it seems the best workaround for now would be if we could merge several coverage.cobertura.xml files into just one in order to pass it to the PublishCodeCoverageResults task.

Does coverlet offer a means of doing that?

Any other suggestions?

[MarcoRossignoli]

AFAIK it's a bit tricky today...we can only merge coverlet json format...so we need to generate all json result and merge togheter and at the end merge and change format.
For coverlet repo for instance:

C:\git\coverlet (master -> origin)
λ dotnet test test\coverlet.collector.tests /p:CollectCoverage=true /p:CoverletOutput=C:\git\coverlet\resultcollectors.json
Test run for C:\git\coverlet\test\coverlet.collector.tests\bin\Debug\netcoreapp2.2\coverlet.collector.tests.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Test Run Successful.
Total tests: 15
     Passed: 15
 Total time: 1,9769 Seconds

Calculating coverage result...
  Generating report 'C:\git\coverlet\resultcollectors.json'

+--------------------+--------+--------+--------+
| Module             | Line   | Branch | Method |
+--------------------+--------+--------+--------+
| coverlet.collector | 74,59% | 55%    | 77,27% |
+--------------------+--------+--------+--------+
| coverlet.core      | 7,99%  | 6,49%  | 17,39% |
+--------------------+--------+--------+--------+

+---------+---------+--------+---------+
|         | Line    | Branch | Method  |
+---------+---------+--------+---------+
| Total   | 21,11%  | 12,5%  | 38,55%  |
+---------+---------+--------+---------+
| Average | 10,555% | 6,25%  | 19,275% |
+---------+---------+--------+---------+


C:\git\coverlet (master -> origin)
λ dotnet test test\coverlet.core.tests /p:CollectCoverage=true /p:CoverletOutput=C:\git\coverlet\resultcore.xml  /p:CoverletOutputFormat=cobertura /p:MergeWith=C:\git\coverlet\resultcollectors.json
Test run for C:\git\coverlet\test\coverlet.core.tests\bin\Debug\netcoreapp2.0\coverlet.core.tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 16.1.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:01.2539208]     Coverlet.Core.Tests.Instrumentation.ModuleTrackerTemplateTests.HitsOnMultipleThreadsCorrectlyCounted [SKIP]
  ! Coverlet.Core.Tests.Instrumentation.ModuleTrackerTemplateTests.HitsOnMultipleThreadsCorrectlyCounted [1ms]
[xUnit.net 00:00:03.5351530]     Coverlet.Core.Instrumentation.Tests.InstrumenterTests.TestCoreLibInstrumentation [SKIP]
  ! Coverlet.Core.Instrumentation.Tests.InstrumenterTests.TestCoreLibInstrumentation [1ms]

Test Run Successful.
Total tests: 114
     Passed: 112
    Skipped: 2
 Total time: 4,4710 Seconds

Calculating coverage result...
  Generating report 'C:\git\coverlet\resultcore.xml'

+--------------------+--------+--------+--------+
| Module             | Line   | Branch | Method |
+--------------------+--------+--------+--------+
| coverlet.core      | 77,12% | 67,93% | 87,57% |
+--------------------+--------+--------+--------+
| coverlet.collector | 74,59% | 55%    | 77,27% |
+--------------------+--------+--------+--------+

+---------+--------+---------+---------+
|         | Line   | Branch  | Method  |
+---------+--------+---------+---------+
| Total   | 76,62% | 66,33%  | 83,93%  |
+---------+--------+---------+---------+
| Average | 38,31% | 33,165% | 41,965% |
+---------+--------+---------+---------+

You'll need to run commands in a sequential way to avoid race.

@tonerdo do you have better solution?I'm not fluent yet on "reports strategies".

[danielpalme]

Alternative:
ReportGenerator can merge several _Cobertura_ files and generate one result file.
It also generates HTML reports which might be useful for visualization.

[MarcoRossignoli]

Nice!
Thank's @danielpalme!

[AArnott]

Thanks!
Ironically the HTML reports produced by ReportGenerator appears to be what Azure DevOps is already using internally and showing off in the build coverage tab. It's too bad it doesn't use that tool to also merge the files for the summary tab. Maybe they'll respond to my feedback with that resolution. :)

[danielpalme]

@AArnott
Correct, Azure DevOps is already using it internally.
But you can do the following:

• Install the _ReportGenerator_ extension: https://marketplace.visualstudio.com/items?itemName=Palmmedia.reportgenerator
• Configure the _ReportGenerator_ task to produce HTML and Cobertura (this is the default)
• Configure the _Publish code coverage results_ task and point it to the HTML files directory and the merged Cobertura file generated by _ReportGenerator_

Currently the _Publish code coverage results_ task regenerates the HTML report.
To avoid that, you have to create a new environment variable:
disable.coverage.autogenerate: 'true'

[AArnott]

Thanks, @danielpalme. I'm definitely seeing improvements here in https://github.com/AArnott/Library.Template/pull/6 where code coverage from multiple target framework test runs is now merged in ADO. :smile:

But I have a twist on the overall requirement: I run tests on 3 agents (Mac, Linux and Windows). Since ADO only shows one code coverage report, I want all these merged. I've arranged for all these to be merged by a "fan in" job in my pipeline, but the problem I'm seeing is that each agent's reports include full paths to source files, and the merging agent, while it has the source files, the paths aren't the same and the merging/report generator process warns that files are missing.

Analyzing 2 classes
Creating report 1/2 (Assembly: Library, Class: Library.Calculator)
File '/Users/vsts/agent/2.153.2/work/1/s/src/Library/Calculator.cs' does not exist (any more).
File 'D:/a/1/s/src/Library/Calculator.cs' does not exist (any more).
Creating report 2/2 (Assembly: Library, Class: ThisAssembly)
File '/home/vsts/work/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).
File '/Users/vsts/agent/2.153.2/work/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).
File 'D:/a/1/s/obj/Library/Release/netstandard2.0/Library.Version.cs' does not exist (any more).

Any suggestions?

[AArnott]

It looks like for now I'll have to settle for publishing a report from just one of the agents.

[AArnott]

Is there a way to get the report files to contain only relative paths to source files instead of absolute paths? If so, I could get this to work and merge reports across agents.

[MarcoRossignoli]

For cobertura we've an open issue https://github.com/tonerdo/coverlet/issues/408#issuecomment-501222651

Do you mean use this format https://issues.jenkins-ci.org/secure/attachment/23019/coverage.xml? Check sources and filename fields.

[AArnott]

Yes, the filename field (and I guess the sources field, although that's empty on my local machine). I'm sure absolute filenames are useful in some contexts, but if we can use relative paths then we can unify across agents.

The cobertura issue you linked to looks like it's refining absolute paths rather than switching to relative. Or am I reading it wrong?

[MarcoRossignoli]

When you say relative path...relative to what?
I mean if I understood well cobertura reports needs source+filename to get "full path"...I don't understand what you mean for "relative".
Another example https://gist.github.com/apetro/fcfffb8c4cdab2c1061d

[AArnott]

relative to what?

The repo root (or whatever I might specify). Some common root that can be reproduced across machines.

So imagine changing it like this:

-<source>/Users/apetro/code/github_jasig/uPortal/uportal-war/target/generated-sources/annotations</source>
+<source>uPortal/uportal-war/target/generated-sources/annotations</source>

Then when passing the report to the tool, /Users/apetro/code/github_jasig would either be the working directory or specified via a command line switch so that the tool knows how to create absolute paths as necessary to access the files.

Since all the reports from all agents would have relative paths, then the machine merging the reports from all of them, and producing HTML files from the source files, could find all the files it needs to, even if each report may have originally had different absolute paths.

[danielpalme]

Relative path are not necessarily easy to achieve.

In your case a script should do the job to remove the absolute path(s):

Example:

(gc "coverage.xml") | % { $_ -replace "C:\full\path", "path" } | Out-File "coverage.xml" -Encoding UTF8

[MarcoRossignoli]

We could try to fix a rule which states that for reports with base+relative source path discovery we setup base on repo root.
But also with this rule...how we can calculate "repo root" for every source system(we can do it for git...but for other)?Should we add a new switch for reports --report-sourcefiles-base-path?In this way merge machine can "replace" bases and generate html or whatever, it's only more simple to fix but it doesn't solve the problem enterely.

I'm not sure that write a reports that expect a base+relative path discovery but with non "real" base is correct...I mean we could break tools that expect "autonomous" report file.

[AArnott]

Yes, that's a concern with putting relative paths in these files, to be sure. If the merge tool could take a separate base path argument for each report file, then it could internally recognize and replace each absolute path with a new absolute path appropriate for the target machine. Feel free to close this issue if you don't expect the merge tool will support such a feature internally.

For now, I'll try your approach of a search-and-replace of each file to enable this.

[MarcoRossignoli]

For now, I'll try your approach of a search-and-replace of each file to enable this.

After https://github.com/tonerdo/coverlet/issues/408 you'll have maybe a worste scenario i.e.

<sources>
                <!--win-->
        <source>C:\</source>
        <source>D:\</source>
....
                <!--unix-->
               <source>/<source>
....
<class name... filename="folder/file" line-rate="0.0" branch-rate="1.0" complexity="0.0">

You'll need to normalize not only filenames but also sources.

[AArnott]

hmmmm.. What is sources used for? What would I need to substitute?

[MarcoRossignoli]

@danielpalme can confirm or correct me(I'm not so fluent on reports yet)..but that sources are a list of base path, that seem cause some issue on Jenkins if it missing

[AArnott]

Well, if that is a collection of "base paths" (and if they can include directories) then if each report were originally created with its own base path, along with absolute paths everywhere else, then the reports wouldn't break any existing tooling. But the merge tool could then consider the base path for each input report and thereby convert each absolute path into a relative path at least long enough to merge the report and then turn it into an absolute path for the final merged result. Just an idea.

That said, I think the search-and-replace strategy is going to be successful. https://github.com/AArnott/Library.Template/pull/9

But rather than use -Replace in powershell, I'm using the string.Replace method. On Windows the -Replace syntax fails because it assumes the first argument is a regex expression so a Windows path which includes backslashes gets misinterpreted.

[MarcoRossignoli]

...Just an idea.

Yep should work!

[danielpalme]

@danielpalme can confirm or correct me(I'm not so fluent on reports yet)..but that sources are a list of base path, that seem cause some issue on Jenkins if it missing

That's also my interpretation.
_ReportGenerator_ iterates through the source paths and combines them with the filename to get the full path of the file on disk.

[AArnott]

Cool. So it sounds like this problem (which I just finished working around myself with search-and-replace) will Just Work soon, once those sources tags are filled in by the collector? Sweet.
If that's already tracked then by other issues, please feel free to close this one.

[MarcoRossignoli]

Cool. So it sounds like this problem (which I just finished working around myself with search-and-replace) will Just Work soon, once those sources tags are filled in by the collector? Sweet.
If that's already tracked then by other issues, please feel free to close this one.

Yes if we'll fix it in a correct way...I mean if we generate same "relative paths" and only different base for every agent script should only amend the bases. Something more complex than simple put root in sources(to simply fix Jenkins issue that need almost one source) and the rest on filename. We need to find the "highest common base path".

Close for the moment...feel free to reopen if need further discussion...thank's to all guys!