Coverlet: General questions about intended usage in .NET Core projects. Possible doc improvements.

[Hjortsberg]

Hi,

I've read practically all guides/issues I can find and tried many different ways of generating coverage reports with coverlet and it has left me somewhat confused.
I'm a newcomer to the whole .NET universe so my questions might be trivial.

Scenario:
The codebase is a multi project .NET core 3.1 solution, my intentions are to be able to generate coverage reports both local and in Azure DevOps pipelines.
Both DevOps agent and local machine runs Ubuntu and Fedora respectively.

Problems

Docs suggest to avoid MSBuild variant of coverlet due to bugs. This led me to install the .NET global tool instead. All project.test.csproj files has package reference to <PackageReference Include="coverlet.collector" Version="1.0.1" />.I've tried invoking coverlet through a lot of commands, here are some examples:

dotnet test /p:CollectCoverage=true

dotnet test path/to/someproject.test.csproj /p:CollectCoverage=true



md5-3db01699cba6c00a0a86a6c2012db86e



dotnet test someProject.sln --results-directory ../TestResults \
   "/p:CollectCoverage=true" \
   "/p:CoverletOutput=../TestResults/" \
   "/p:MergeWith=../TestResults/coverlet.json" \
   "/p:CoverletOutputFormat=\"json,cobertura\""



md5-a7a6719b56e7edc4cce97db179f0ee1b



Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.8588 Seconds

Calculating coverage result...
  Generating report 'pat/to/test/file/TestResults.json'

+----------------------+--------+--------+--------+
| Module               | Line   | Branch | Method |
+----------------------+--------+--------+--------+
| tested.file          | 25.33% | 18.18% | 26.38% |
+----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 25.33% | 18.18% | 26.38% |
+---------+--------+--------+--------+
| Average | 25.33% | 18.18% | 26.38% |
+---------+--------+--------+--------+

The only way I get this output is if I add the reference to coverlet.msbuild in the project.test.csproj files. Alternatively if I invoke coverlet directly on the test.dlls. However I don't get coverlet to work as a standalone command in DevOps pipelines so this is a not an viable alternative.

Questions regarding above:

• Is it correct to need coverlet.msbuild to get coverlet to work even if i invoke it by .NET global tool? Is this the best practice? Does the msbuild reference mean the test is run by msbuild instead of the global tool?
• Should it work with only coverlet.collector in the test.csprojs?
• Is it possible to give users any feedback regarding whats missing when invoking coverlet through dotnet test?

Hopefully my questions makes some kind of sense, if not, ask me and I will try to answer with more details. :)

[rcollina]

@Hjortsberg I'd chime in, if you don't mind. I've been trying to make coverlet work with our gitlab pipelines but no luck so far.

coverlet.collector reference:

    <PackageReference Include="coverlet.collector" Version="1.2.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>

runsettings.xml

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat Code Coverage">
        <Configuration>
          <Format>opencover,lcov,cobertura</Format>
          <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
          <Exclude>[coverlet.*]*,[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude>          
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

</RunSettings>

Powershell script to run dotnet test on an already built solution:

dotnet test --no-restore --no-build `
            --collect "XPlat Code Coverage" `
            --filter Category=Unit `
            --settings $runsettings_path `
            --logger "console;verbosity=detailed" `
            --results-directory $test_output_path `
            -o "$env:TESTS_DIR/**"

No coverage, no matter the format used.

[MarcoRossignoli]

@rcollina please open a new issue.

[rcollina]

I'm sorry, how is this not related to original issue?
Documentation is scattered across several files, and the provided guidance yields no results.

[MarcoRossignoli]

Docs suggest to avoid MSBuild variant of coverlet due to bugs. This led me to install the .NET global tool instead.

.NET global tool suffer of same issue as msbuild https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test
.NET global tool simply instruments->run you command-> and check for coverage so it's very similar to msbuild integration(tasks) but orchestrated by console app.

dotnet test /p:CollectCoverage=true
dotnet test path/to/someproject.test.csproj /p:CollectCoverage=true

This command is the same and uses msbuild version so if you don't add reference to coverlet.msbuild won't work. /p:CollectCoverage=true instrument through msbuild tasks.

All project.test.csproj files has package reference to ...

By design .net core test template add reference to coverlet.collector.dll because .net core platform doesn't have a way to produce xplat. So they decided to use coverlet that is xplat.
But to use coverage with collector(the preferred way due to known bugs, collectors are well used by vstest workflow) you need to pass --collect:"XPlat Code Coverage" and use "vstest integration" https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md

Is it correct to need coverlet.msbuild to get coverlet to work even if i invoke it by .NET global tool? Is this the best practice? Does the msbuild reference mean the test is run by msbuild instead of the global tool?

No if you use .net global tool(console app) the instrumentation and results will be orchestrated by console app, tool runs your command(dotnet test) after instrumenting you dll and when "your commad" exits(it's hosted by a custom process like a normal dotnet test) the same tool will read hits file and produce coverage file.

Should it work with only coverlet.collector in the test.csprojs?

collector is needed only and only if you're using vstest integration https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md
You'll see this reference inside every .net test template project(xunit) because is the "default" way to do coverage for every .net core app. This tool was chosed by MS and we're proud of it.

Is it possible to give users any feedback regarding whats missing when invoking coverlet through dotnet test?

Msbuild/.NET tool integration at the moment shows on console a table with coverage percentage plus generates xml coverage file(depends on chosed format)
VSTest collectors integration today doesn't show anything because it's pretty new and we're working to move all feature also on this integration(that is the preferred one).
For instance show to console(as you asked), merging reports and specify an absolute file path for generated file is not allowed due to vstest platform behaviour(today).

[MarcoRossignoli]

I'm sorry, how is this not related to original issue?

@rcollina it's a different issue...you're using collector and maybe your problem is a config problem not a problem related to "understand what driver use" this is our helloworld working example with collector https://github.com/tonerdo/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld take a look if you can to understand if you're missing something or feel free to open a new issue and post your complete csproj for test project and command line you're using.

[Hjortsberg]

Thank you for that explanation.
I think that we will stick with the msbuild option for now after reading this.

[MarcoRossignoli]

Thank you for that explanation.
I think that we will stick with the msbuild option for now after reading this.

Remember if you'll find strange exception or empty coverage take a look at known issue to understand if you're hitting known problem.

Feel free to close this if solved!

[Hjortsberg]

@MarcoRossignoli just a small update.
We quickly struck the bugs and strange behaviour (as you explained), now running with xplat and vstest instead and it now finds all tests (so far).
For anyone finding this issue, just go with Xplat/vstest straight away and save yourself some time. :)

[MarcoRossignoli]

Thank's for reporting this.