Coverlet: Solution wide coverage

Created on 3 Mar 2019  路  28Comments  路  Source: coverlet-coverage/coverlet

Coverlet should be able to get coverage for all test projects in a specified solution file

related issues
https://github.com/tonerdo/coverlet/issues/165

feature-request
Source
picture tonerdo
👍28 🚀2

Most helpful comment

@IGx89 if you run 

dotnet test solution.sln /p:CollectCoverage=true /p:Exclude="[xunit*]*" /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1

it will create a single coverage file and will report the complete coverage after each test.

some things to consider with this approach:

  • You have to make sure to delete the coverage.json file after the run or the next run will re-use it
  • the path in CoverletOutput and MergeWith are relative to the test project being run
picture joemey on 8 May 2019
👍7

All 28 comments

FYI,
We use coverlet msbuild tool to run tests for all tests projects in a solution file with the next command:

dotnet test solution.sln /p:CollectCoverage=true /maxcpucount:1

The /maxcpucount:1 argument is necessary to prevent parallelizable code instrumentation. Parallelizable instrumentation lock files and block other instance of coverlet

pantosha picture pantosha on 17 Apr 2019

That'll produce individual test reports, but won't produce a single combined test report (my impression of what this enhancement is for).

IGx89 picture IGx89 on 17 Apr 2019

@IGx89 if you run 

dotnet test solution.sln /p:CollectCoverage=true /p:Exclude="[xunit*]*" /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1

it will create a single coverage file and will report the complete coverage after each test.

some things to consider with this approach:

  • You have to make sure to delete the coverage.json file after the run or the next run will re-use it
  • the path in CoverletOutput and MergeWith are relative to the test project being run
joemey picture joemey on 8 May 2019
👍7

Directory.Build.props:

<PropertyGroup>
    <CoverletOutput>./coverage/coverage.xml</CoverletOutput>
    <CoverletOutputFormat>opencover</CoverletOutputFormat>
</PropertyGroup>

Directory.Build.targets:

<Target Name="CoverRunPreparation" BeforeTargets="GenerateCoverageResult" Condition="'$(CollectCoverage)' == 'true'">
        <ItemGroup>
            <CoverletOutputPath Include="$(CoverletOutput)" />
        </ItemGroup>
        <PropertyGroup>
            <CoverletOutput Condition="'$(CoverletOutput)' != '' AND '$(TargetFrameworks)' != ''">%(CoverletOutputPath.RootDir)/%(CoverletOutputPath.Directory)/%(CoverletOutputPath.Filename).$(TargetFramework)%(CoverletOutputPath.Extension)</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' == ''">$(ProjectDir)coverage.xml</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' != ''">$(ProjectDir)coverage.$(TargetFramework).xml</CoverletOutput>
        </PropertyGroup>
</Target>
viceice picture viceice on 22 May 2019

@joemey how does MergeWith work with multiple CoverletOutputFormat?

iron9light picture iron9light on 16 Jul 2019

@joemey how does MergeWith work with multiple CoverletOutputFormat?

@iron9light take a look at https://github.com/tonerdo/coverlet/issues/474

MarcoRossignoli picture MarcoRossignoli on 16 Jul 2019

I have tried the solution suggested by @joemey and it works, but only for json, but it does not for opencover.

For this command dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover"
I received:

位 dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover"
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 4,8472 Seconds
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 2,5702 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 0,93%  | 0%     | 3,17%  |
+-----------------------------------+--------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 6,81% | 7,52%  | 7,86%  |
+---------+-------+--------+--------+
| Average | 7,06% | 5,93%  | 11,2%  |
+---------+-------+--------+--------+

Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,3510 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

So it counts coverage for the first project and saves it into json, but for the second project, it can not proceed.

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

lkurzyniec picture lkurzyniec on 31 Dec 2019

@lkurzyniec can you try to run same command but with /p:CoverletOutputFormat="opencover,json" parameter?Similar to https://github.com/tonerdo/coverlet/issues/262#issuecomment-569572522 and remember to use /maxcpucount:1

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

Yes I did same sample on our "sample" section https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md , you can clone repo and test solution, but you cannot run command "solution wide".

MarcoRossignoli picture MarcoRossignoli on 2 Jan 2020

For reference link msbuild task sync discussion https://github.com/microsoft/msbuild/issues/4898 https://github.com/microsoft/msbuild/issues/4954

MarcoRossignoli picture MarcoRossignoli on 2 Jan 2020
😕1

Unfortunately, it doesn't work

位 dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 5,0513 Seconds
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 3,1337 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 35,51% | 35,71% | 44,44% |
+-----------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 24,81% | 19,54% | 37,07% |
+---------+--------+--------+--------+
| Average | 24,35% | 23,78% | 31,83% |
+---------+--------+--------+--------+

Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,9084 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

What's interesting here is that it generate json two times:

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'
lkurzyniec picture lkurzyniec on 3 Jan 2020

Try with

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

Similar to https://stackoverflow.com/questions/53255065/dotnet-unit-test-with-coverlet-how-to-get-coverage-for-entire-solution-and-not/56281108#56281108 but with -m:1

For instance you can try with our samples on repo

D:\git\coverlet\Documentation\Examples\MSBuild\MergeWith
位 dotnet test /p:CollectCoverage=true  /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1
MarcoRossignoli picture MarcoRossignoli on 3 Jan 2020
👍1

@MarcoRossignoli That works fine! In my case, it gives me solution wide results.

Summary

cmd command

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

gives me the same results as following two commands

dotnet test ./test/HappyCode.NetCoreBoilerplate.Core.UnitTests/HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/

dotnet test ./test/HappyCode.NetCoreBoilerplate.Api.UnitTests/HappyCode.NetCoreBoilerplate.Api.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat="opencover"

image

What's more, your command dotnet test /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1 from https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md also works :)

lkurzyniec picture lkurzyniec on 8 Jan 2020
👍1

Glad to hear!

MarcoRossignoli picture MarcoRossignoli on 8 Jan 2020

Not sure if this should be a separate question, or asked here, but is it possible to do solution wide coverage using the coverlet .Net global tool? I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac). The task I'm running looks like this:

  - script: coverlet $(buildArtifact)/tests/core/My.Project.Tests/bin/$(buildConfiguration)/netcoreapp3.1/My.Project.Tests.dll --target "dotnet" --targetargs "test $(buildArtifact)/tests/core/My.Project.Tests --no-build" --format cobertura
    displayName: Run Coverlet to get code coverage

But could it target a directory and run all tests projects under that directory? Or target a solution file?

robertlarkins picture robertlarkins on 30 Jan 2020

I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac)

Why don't use collectors https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md? Why .NET tool?

Coverlet is cross platform and works cross plat for every "drivers" you're using https://github.com/tonerdo/coverlet#quick-start but the collectors is the preferred way to consume due to integration with vstest platform and you can run command on sln(it generates one coverage file for every test project you have in solution).

MarcoRossignoli picture MarcoRossignoli on 30 Jan 2020

At this stage I was only using coverlet as part of the Azure Pipeline, so didn't think the test projects should know or depend on it. Will try installing coverlet.collector as a NuGet package.

robertlarkins picture robertlarkins on 31 Jan 2020

So I added coverlet.collector as a NuGet package and got it going for a single test project with this:

  - task: DotNetCoreCLI@2
    displayName: 'Run tests for $(solution) with Cobertura'
    inputs:
      command: test
      projects: '$(buildArtifact)/tests/**/*.csproj'
      arguments: -c $(buildConfiguration) --collect:"XPlat Code Coverage"

  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage'
    inputs:
      codeCoverageTool: cobertura
      summaryFileLocation: '$(Agent.TempDirectory)/*/coverage.cobertura.xml'

But this produces a separate coverage.cobertura.xml file per test project (each in there own guid directory). In this Azure Pipelines context how would I get the individual test results merged together so that a single code coverage publish can be done?

robertlarkins picture robertlarkins on 31 Jan 2020

@robertlarkins my suggestion would be to read both this article and whole this thread. Then try to achieve solution-wide results locally. After that move your solution to Azure DevOps.

lkurzyniec picture lkurzyniec on 31 Jan 2020

@lkurzyniec Will attempt and see how I get on.

robertlarkins picture robertlarkins on 31 Jan 2020

At the moment I've removed from documentation Merge support for collectors because we're working on better approach https://github.com/tonerdo/coverlet/pull/704 (merge works well only for sequential tests), because "as is" doesn't work(vstest generates different filenames as you can see).

Actually that parameters is "hidden" but usable, so if you want be sure to not hit https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test and use merge with vstest you can use a trick of our contributor @p4p3 that with a script move generated coverage files for every test run https://github.com/tonerdo/coverlet/pull/225#issuecomment-573896446

If you don't hit issue above you can use msbuild driver and https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

MarcoRossignoli picture MarcoRossignoli on 31 Jan 2020
👍2

@lkurzyniec , @MarcoRossignoli Hi. I am trying to execute the above command but getting an error

PS > dotnet test /p:CollectCoverage=true /p:CoverletOutput=../.coveragedata/cover
age /p:MergeWith="../coverage.json" /p:CoverletOutputFormat=\"json,opencover\" -m:1
MSBUILD : error MSB1006: Property is not valid.
Key: opencover\
sungam3r picture sungam3r on 16 Apr 2020

@sungam3r you need to translate characters on powershell take a look https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#note-for-powershell--vsts-users %2c for ,

MarcoRossignoli picture MarcoRossignoli on 16 Apr 2020

Wow! Thanks. I read this section of the documentation, but for some reason I missed it.

sungam3r picture sungam3r on 16 Apr 2020

No prob

MarcoRossignoli picture MarcoRossignoli on 16 Apr 2020

Still having issues in getting merge results and allowing to run ReportGenerator from that. Following is the dotnet test command I am trying to run. Using coverlet 2.9.0 in my test projects. Getting the merge results works fine untill I add that last part regarding output format. But if I only output the default json I cannot get the ReportGenerator to produce anything valid. And if I use other formats in dotnet test, I cannot seem to be able to get results merged.

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/coverage.json' /p:CoverletOutputFormat=\"json%2copencover\"

C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]
C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]
C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonReader.ReadAndMoveToContent() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]
C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]
C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]
C:\Users\username\.nuget\packages\coverlet.msbuild2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]

aviita picture aviita on 21 Oct 2020

And I have tried different variations of this CoverletOutputFormat string, all resulting to error of some sort with command above:

/p:CoverletOutputFormat=\"json%2copencover\" 

/p:CoverletOutputFormat="json%2copencover"

/p:CoverletOutputFormat='json,opencover'

/p:CoverletOutputFormat='json%2copencover'
aviita picture aviita on 21 Oct 2020

And found my problem, so I am not supposed to put the json file, but only a path to CoverletOutput param. Defining json file likely caused coverlet xml to be written to the json file:

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/' /p:CoverletOutputFormat='\"json%2copencover\"'
aviita picture aviita on 21 Oct 2020

@aviita sorry for the delay(very busy time), glad to hear you solve your problem.

MarcoRossignoli picture MarcoRossignoli on 10 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  路  5Comments
Stefanie899 picture Stefanie899  路  3Comments
obtuse-whoosh picture obtuse-whoosh  路  6Comments
civspan picture civspan  路  6Comments
MarcoRossignoli picture MarcoRossignoli  路  6Comments