Coverlet: /p:MergeWith does not merge but adds

[pape77]

I'm testing using coverlet over a solution that has 3 test projects which i want to combine into a single .opencover.xml file

Since there's currently no direct solution for this. What I tried to do is generate the coverage for one of the test projects, output its result into an "converage.json" file and then apply a MergeWith param in the following dotnet test commands to merge into that same json file and set the output format to opencover

Problem is, that since the first project(ProjectName.Api) includes other two as references (ProjectName.Application, ProjectName.Data), the report is generated as following:

Test run for /build/test/ProjectName.Api.Tests/bin/Debug/netcoreapp2.0/ProjectName.Api.Tests.dll(.NETCoreApp,Version=v2.0)

Results File: /results/_8e5df6e97674_2018-09-28_11_10_36.trx

Total tests: 15. Passed: 15. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.4645 Seconds

Calculating coverage result...
  Generating report '/results/coverage.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
|ProjectName.Api         | 24.4%  | 28.6%  | 33.3%  |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
|ProjectName.Application | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+

So it includes two other coverages for the referenced projects with 0% coverage.
This wouldn't be a problem if /p:MergeWith param actually did a proper merge.

But when I run the next dotnet test (over the ProjectName.Application project) with the MergeWith param pointing to this recently generated coverage.json file, I get this:

Test run for /build/test/ProjectName.Application.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.Tests.dll(.NETCoreApp,Version=v2.0)

Starting test execution, please wait...
Results File: /results/_5d6c3e7d1c85_2018-09-28_11_10_47.trx

Total tests: 4. Passed: 4. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.7143 Seconds

Calculating coverage result...
  Generating report '/results/coverage.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
| ProjectName.Application | 100%   | 60%    | 100%   |
+----------------------+--------+--------+--------+
| ProjectName.Api         | 24.4%  | 28.6%  | 33.3%  |
+----------------------+--------+--------+--------+
| ProjectName.Data        | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+
| ProjectName.Application | 0%     | 0%     | 0%     |
+----------------------+--------+--------+--------+

So it just adds the recent coverage to the existing ones in coverage.json. Causing a duplicated coverage in ProjectName.Application and ProjectName.Data (because again, ProjectName.Application references ProjectName.Data, so it adds a coverage 0% report)

Any solutions to this? Or am I missusing something?

Or even a way for coverlet for avoid including referenced project coverage?

[pape77]

So my final expectation is to run this:

dotnet test ./tests/ProjectName.Api.Tests/ProjectName.Api.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

Which is equivalent to run dotnet test on each proj:

dotnet test ./tests/ProjectName.Api.Tests/ProjectName.Api.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage

 dotnet test ./tests/ProjectName.Application.Tests/ProjectName.Application.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./tests/ProjectName.Data.Tests/ProjectName.Application.Tests.csproj -l trx -r /results /p:CollectCoverage=true /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

 dotnet test ./${SolutionName} -l trx -r /results /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:CoverletOutput=/results/coverage /p:MergeWith='/results/coverage.json'

[pape77]

Update: @tonerdo Looking into the Merging code https://github.com/tonerdo/coverlet/pull/186/commits/45bc5bc8433d07df90cbda5a6f80bfe98898d83d

I see you merge by Module key, then by Document key, etc.
In this case the Module key will be different since the dll is referenced from different paths

So the coverage.json looks like:

"/build/test/ProjectName.Api.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.dll": {
    "/build/src/ProjectName.Application/Services/Provider.cs": {
      "ProjectName.Application.Services.Provider": {
                 "System.Void ProjectName.Application.Services.KeyProvider::.ctor(ProjectName.Data.Repositories.Contracts.IProjectRepository,Microsoft.Extensions.Caching.Memory.IMemoryCache)": {
          "Lines": {
            "18": 0,
            "19": 0,
            "20": 0,
            "21": 0,
            "22": 0
          },
          "Branches": []
        }
      }
  ...
 }
},

...

"/build/test/ProjectName.Application.Tests/bin/Debug/netcoreapp2.0/ProjectName.Application.dll": {
    "/build/src/ProjectName.Application/Services/Provider.cs": {
      "ProjectName.Application.Services.Provider": {
        "System.Void ProjectName.Application.Services.KeyProvider::.ctor(ProjectName.Data.Repositories.Contracts.IProjectRepository,Microsoft.Extensions.Caching.Memory.IMemoryCache)": {
          "Lines": {
            "18": 4,
            "19": 4,
            "20": 4,
            "21": 4,
            "22": 4
          },
          "Branches": []
        }
      }
   ...
  }
}

So module keys are different but they reference the same dll file. This is why they are just added and not merged.

Any chance it can look into same dll names for merging instead of the whole key? (via Path.GetFileName perhaps?)

[pape77]

Pr created for this https://github.com/tonerdo/coverlet/pull/204

[Tonvengo]

When this will be pushed to coverlet.msbuild nuget package?

[pape77]

@Tonvengo Hopefully after #211 is merged! haha (otherwise the mergeWith will break)

Hopefully @tonerdo sees this first :)

[pape77]

I will reopen until it's there on the pkg

[Tonvengo]

:) nice

[vlef]

Great, thanks for the feature, hopefully it gets a release soon.

However, I'm wondering, wouldn't it be an even better solution to extend this feature into some kind of /p:AggregateResultsAt that would create the first file if it doesn't exist, append all other results to it, and then finally convert it to other formats if required.

In my use case (VSTS build with multiple test projects), I could have a single step that would look like this:

dotnet test **/*[Tt]ests/*.csproj
/p:CollectCoverage=true
/p:AggregateResultsAt=/somepath/
/p:CoverletOutputFormat=opencover,cobertura

Thank you guys for your work!

[pape77]

@vlef without doubts. That was my next step, because if the file doesn't exist it fails. And it's annoying because when running a dotnet test on a solution you want it to also generate the file for the first test run.

I haven't got the time to look into that TBH. But maybe @tonerdo can take a look when he is free. Or if you find it you can also make a pr for it :)

Cheers

[tonerdo]

Definitely sounds like a good idea @vlef @pape77. For starters I could just make the MergeWith parameter ignore when the specified file doesn't exist and just assume it's on a first run

[pape77]

That would be a great addition to the pr i made and you already merged in @tonerdo :)
Looking forward to it!

[pape77]

Released

[obtuse-whoosh]

Where was this released? I'm not seeing a new version on the NuGet feed. Last version is 2.3.1, 5 days before this was merged. Can we get an updated package please? I would love to use this feature!

[pape77]

I use the last pkg and it merges correctly now. Still i made a pr for correcting some bad branch coverage output (see #225). Also, the feature for not failing when merging and file doesn't exist (see #220) was merged but not released.
I will give it a try again deleting all my local nuggets then

[obtuse-whoosh]

I'm waiting on #220 to be released

[pape77]

@obtuse-whoosh and then you probably have the issue at #225 xD

[obtuse-whoosh]

🤔 I'm not sure about that. The issue I'm having is the error that crops up while running a test with the MergeWith parameter when the output file doesn't exist yet. Specifically the issue solved by #220. #225 doesn't affect my use case 🤷

[pape77]

@obtuse-whoosh had you check that your merge is not decreasing your branch coverage as stated in the issue i opened and referenced in #225?
If you use different test projects that test projects that reference each other, there's a chance that happens to you too.
Merging is properly done, but branch coverage percentage won't be accurate and different than when testing each project separately.

[obtuse-whoosh]

Wow that's something I missed, thanks for the info @pape77 . Do you know when @tonerdo will release these changes?

[pape77]

@obtuse-whoosh no idea. Hopefully my pr will get merged first.
Then we have to wait for a release 😅

[pape77]

@obtuse-whoosh Seems like he won't release it :/

See #223

Still no reason why though

[Dragon160]

Any update here? I think it is quite a common use case to have more than one test project within the solution.
A way to aggregate it so a single output would be really nice.

[p4p3]

@dragon160 actually as I've seen it went worse because it's now adding coverage for dlls that are not even part of your project, but some other libraries (at least last time i checked).
As mentioned. I forked away long time ago with my own coverlet version. On the issue above was the change i did for this back then

[priyanka1112]

Hi, I am using coverlet 1.6.0 version and in my solution there are multiple test project but mergewith is not working

I am using below:

Coverlet "D:\Project\proj1.Testbin\Debug\netcoreapp2.2\proj1.Test.dll" --target "dotnet" --targetargs "test "D:\Project\proj1.Test" --no-build"
Coverlet "D:\Project\proj2.Testbin\Debug\netcoreapp2.2\proj2.Test.dll" --target "dotnet" --targetargs "test "D:\Project\proj2.Test" --no-build" --merge-with=”D:\ProjectCoverage.json”
Coverlet "D:\Project\proj3.Testbin\Debug\netcoreapp2.2\proj3.Test.dll" --target "dotnet" --targetargs "test "D:\Project\proj2.Test" --no-build" --no-build" --merge-with=”D:\ProjectCoverage.json” --format opencover

[ckarcz]

I have a unit-test-per-project structure and also having this issue. Coverlet replaces the correct coverage for an assembly with the coverage of the assembly in the current test assembly, instead of a merging coverage.

I am able to work around it by running coverlet per project and then letting report-generator do the merging, which it _seems_ it does correctly.

here is a gist for a powershell script that may help you https://gist.github.com/ckarcz/180a9ddfd9bfe33b1e68e1634c0e599e#gistcomment-3132943

cheers 🍻

[MarcoRossignoli]

Other that @ckarcz you can try with https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

[ckarcz]

@MarcoRossignoli i did that but still had the coverage results of a library replaced with a subsequent collection/merge, resulting in zero coverage for that library. same as @priyanka1112 describes. the merge should merge the result data, not just the result xml nodes in the files.

[MarcoRossignoli]

@MarcoRossignoli i did that but still had the coverage results of a library replaced with a subsequent collection/merge, resulting in zero coverage for that library

Can you provide a simple repro?I'm not sure to understand where is the problem.