Coverlet: Show coverage summary on terminal

[MarcoRossignoli]

Today at the end of the tests msbuil/.net tool drivers show a table with coverage result on console.
For instance:

Calculating coverage result...
  Generating report '/home/vsts/work/1/s/test/coverlet.core.tests/coverage.opencover.xml'

+------------------------------------+--------+--------+--------+
| Module                             | Line   | Branch | Method |
+------------------------------------+--------+--------+--------+
| coverlet.tests.projectsample.empty | 0%     | 100%   | 0%     |
+------------------------------------+--------+--------+--------+
| coverlet.msbuild.tasks             | 0%     | 0%     | 0%     |
+------------------------------------+--------+--------+--------+
| coverlet.core                      | 83.65% | 78.2%  | 84.07% |
+------------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 71.74% | 69.83% | 61.9%  |
+---------+--------+--------+--------+
| Average | 27.88% | 59.4%  | 28.02% |
+---------+--------+--------+--------+

This is not supported at the moment for vstest collectors integration.

cc: @vagisha-nidhi @PureKrome

[patricksadowski]

Does VSTest signal the end of the entire test run (like an event)? Each data collector could then do one last action.

[MarcoRossignoli]

No AFAIK...this is the biggest issue for coverlet for a lot of features like merging...we don't have a way to synch between parallel tests run. I'm in contact with vstest team to undestand if there is a solution, but I think that at the moment the only place is inside dotnet cli out of vstest control, btw this is my speculation, we'll see.

[MarcoRossignoli]

For this issue I had a fast conversation with team member and seem that we could inject a custom logger https://github.com/microsoft/vstest-docs/blob/master/docs/report.md

[PureKrome]

Thanks @MarcoRossignoli for working on this issue ASAP. this is the biggest blocker for us, using the vstest collector.

[MarcoRossignoli]

@PureKrome why?Can you tell me how you use terminal data?

[PureKrome]

Sure, either when:

• doing localhost development (to get an idea of the coverage early on)
• CI/CD server (e.g. Azure DevOps or AppVeyor).

using the command line (eg. dotnet test ..) creates a file which I can't read easily. Just seeing the ascii art chart after that command is a great first and easy step into seeing some quick steps.

It's a blocker because it's slowing me down when coding. I don't want to have to push the report file up to codecov.io and _then_ log in to that to review the report.

[MarcoRossignoli]

Ok for local dev you can use ReportGenerator with simple script https://github.com/danielpalme/ReportGenerator

[mcallaghan-bsm]

This issue has strained on me for an hour. Due to https://github.com/coverlet-coverage/coverlet#vstest-integration-preferred-due-to-known-issue-supports-only-net-core-application, we have been exploring vstest instead of msbuild for code coverage.

To my dismay, same as op, there is seemingly no way today to get a simple reST (https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html) table summary output from this tool.

The REASON for this is for gitlab CI coverage integration and badges, see: https://docs.gitlab.com/ee/ci/yaml/#coverage, so for a gitlab job that runs dotnet test with coverage ... one would regex match like this:

    # opencover coverage report prints in TABLE reST format
    coverage: '/^\s*\|\s+Total\s+\|\s+(\d+\.?\d+\%)\s+\|/'

I tried the suggested workaround (to use an open-source report generator https://danielpalme.github.io/ReportGenerator/usage.html), however this cannot achieve it either. The closest I found was using the TextSummary

dotnet reportgenerator -reports:/SNIP/coverage.opencover.xml -targetdir:coveragereport -reporttypes:TextSummary

, this gives us something like:

(snip)
...
2020-09-21T15:37:46: Writing report file 'coveragereport/Summary.txt'
2020-09-21T15:37:46: Report generation took 0.3 seconds

so then we can at least get that data with:

$ sed -n '/Summary/,/^$/p' coveragereport/Summary.txt
Summary
  Generated on: 2020-09-21 - 3:33:00 p.m.
  Parser: OpenCoverParser
  Assemblies: X
  Classes: XX
  Files: XXX
  Line coverage: ##.#%
  Covered lines: XXXX
  Uncovered lines: XXXX
  Coverable lines: XXXX
  Total lines: XXXX

and we'd have to augment our regex match

[gitfool]

@mcallaghan-bsm I went through the same hoops - using a vstest "collector", then report generator for a text summary, and finally scrape the output for GitLab CI. AFAIK, this is the best way until coverlet collector hooks into collector logging to generate the text summary itself.

[PureKrome]

@mcallaghan-bsm and @gitfool - wow! as a workaround, that's great!!! painful ... but great!

[patricksadowski]

We use an another approach with the coverlet vstest collector on GitLab. Coverlet is configured to output the results as opencover format. After the test run a script extracts the coverage from the report and prints it to console.

PERCENTAGE=$(grep -m 1 -o -P '(?<=sequenceCoverage=")(\d+.\d+|\d+)' $DIRECTORY/coverage.opencover.xml)
echo "Line coverage: ${PERCENTAGE}%"