Coverlet: .net core 3.1 coverage usage not working anymore

[anaether]

Hello guys,

Has the release of .NET CORE changed something to coverlet?
Since i updated my projects/testprojects to .NET CORE 3.1

Used command:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Exclude='[xunit]'

Packages installed for 3.1:
coverlet.collector
coverlet.msbuild
Microsoft.NET.Test.Sdk

[MarcoRossignoli]

AFAIK no, can you provide more details?Can you provide a repro of issue?

[jhartmann123]

I also have that issue, but I only got it for one of our projects. It is an ASP.Net Core API project (using Microsoft.NET.Sdk.Web as project SDK, TargetFramework netcoreapp3.1). It works with a console project in the same solution (also netcoreapp3.1, TargetFramework Microsoft.NET.Sdk).

For the API project, I get the following error:

/root/.nuget/packages/coverlet.msbuild/2.7.0/build/coverlet.msbuild.targets(7,5): warning : [coverlet] Unable to instrument module: /builds/src/backend/xxx.Tests/bin/Debug/netcoreapp3.1/xxx.Api.dll because : Failed to resolve assembly: 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=3.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60' [/builds/src/backend/xxx.Tests/xxx.Tests.csproj]

I tried to create a repro, but unfortunatly it works perfectly fine in there and I couldn't really nail down the issue :( I'll let you know if I find more details.

[Edit] Testing framework is XUnit if that matters

[MarcoRossignoli]

@jhartmann123 you issue seem very similar to https://github.com/tonerdo/coverlet/issues/560 you could try the "manual copy trick" or try to use nightly build where there is an attempt to fix that issue https://github.com/tonerdo/coverlet/issues/560#issuecomment-561152895

[MarcoRossignoli]

This is also on our know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#4-failed-to-resolve-assembly-during-instrumentation

[jhartmann123]

Unfotunately it's not that. Tried it with coverlet.msbuild 2.7.48-g45d8d4ee42 and CopyLocalLockFileAssemblies.

• I used CopyLocalLockFileAssemblies when we were on core 2.2, as it was required there
• I removed it after upgrading to core 3.0. Coverage was working as expected
• The upgrade to core 3.1 basically killed the coverage for that one project (with the resolving issue mentioned above).

[MarcoRossignoli]

Unfotunately it's not that. Tried it with coverlet.msbuild 2.7.48-g45d8d4ee42 and CopyLocalLockFileAssemblies.

Can you retry and check logs https://github.com/tonerdo/coverlet/blob/master/Documentation/Troubleshooting.md#msbuild-integration

Ah one more thing...have you added required attributes <PreserveCompilationContext>true</PreserveCompilationContext>?If dll isn't found I expect and error like this https://github.com/tonerdo/coverlet/blob/3856c76a5627a8d38d2cf2eae9360f05f7d878c5/src/coverlet.core/Instrumentation/CecilAssemblyResolver.cs#L210

[anaether]

Its a console application with a separated project for tests. Which informations would be helpful?

[anaether]

When i run the following command:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover /p:Exclude='[xunit]'

I receive this error:
Path\Packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error MSB4044: Die Coverlet.MSbuild.Tasks.CoverageResultTask-Aufgabe hat keinen Wert für den erforderlichen InstrumenterState-Parameter erhalten.

[MarcoRossignoli]

Try to add <IsTestProject>false</IsTestProject> to console app https://github.com/tonerdo/coverlet/blob/master/test/coverlet.testsubject/coverlet.testsubject.csproj#L6 and you can remove /p:Exclude='[xunit*]*' it's unuseful with last version so

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

to be sure that is not a problem of "wrong instrumented project" launch this on root of xunit project.

[jhartmann123]

Sorry for hijacking the issue - found the bug that causes mine, looks to be pretty unrelated - I'll create a new issue.

[anaether]

nd you can remove /p:E

The console app is already IsTestProject = false and the test project has the flag to true.

With the command:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

I got this error again:
Packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error MSB4044: Die Coverlet.MSbuild.Tasks.CoverageResultTask-Aufgabe hat keinen Wert für den erforderlichen InstrumenterState-Parameter erhalten.

[anaether]

Who has also tried to use it in .net core 3.1 and has maybe the same issue?

[MarcoRossignoli]

@anaether at the moment you're the first signal this kind of issue

[anaether]

@MarcoRossignoli I will give it later a try and create a 2 new blank projects.

[MarcoRossignoli]

Yes thanks!Isolate a repro is the most useful thing to do!

[PureKrome]

@anaether - why are you referencing BOTH

• coverlet.collector
• coverlet.msbuild

in your testProject.csproj file? Shouldn't it be _one of the other_ ?

Then just do dotnet test --collect:"XPlat Code Coverage" as documented on the main README here?

[MarcoRossignoli]

@PureKrome collector is referenced by default by xunit project template, so I think will be pretty common if someone want to use msbuild(for historical reason) against "preferred" way of vstest integration, btw if this wasn't the case I agree that collector way is the best choice due to know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test

[anaether]

@PureKrome to use both packages was just a test. Before 3.1 upgrade i only used coverlet.msbuild package.

@MarcoRossignoli I will try it today the last day i had less time to create two new projects and test it there against.

[MarcoRossignoli]

@MarcoRossignoli I will try it today the last day i had less time to create two new projects and test it there against.

@anaether don't worry about me, take your time and do at your pace, no rush, we all have our commitments 😄

[anaether]

@PureKrome If my build would be only visual studio then it would be okey. I use coverlet test with my gitlab ci on my gitlab instance my projects and to extract analysis informations with sonarqube.

[PureKrome]

@anaether same here - I'm using AppVeyor and AzureDevOps for CI.

Looking at the documentation here, the command is dotnet test --collect:"XPlat Code Coverage" for using the VSTest integration (which to me is another name of "the Collector").

that dotnet test command has nothing to do with _Visual Studio_ per say.

so ... do you still need both, then?

Disclaimer: i may have all of this, wrong :(

Side note: I don't like the name VSTest integration because to me that implies it's a _visual studio_ dependency ... while looking at the command, it has nothing to do with VS? it's just using the dotnet SDK...

[MarcoRossignoli]

I'll try to clarify sorry for the confusion.

the reference to coverlet.collector is needed in case of dotnet test --collect:"XPlat Code Coverage" , as you can see it's a "collector", a way inside vstest engine https://github.com/microsoft/vstest to collect data from testing process. VsTest engine didn't have a way to do coverage in a xplat way, the only open product in the market was coverlet, so to simplify the user experience a "generic" collector name was chosen XPlat Code Coverage , actually with that collector name vstest engine load and uses coverlet engine in form of collector(there is a strict integration, embedded code inside vstest engine).

After the integration every new xunit project created with template will bring that reference because today coverlet is used as "default" xplat coverage engine for dotnet core.

coverlet.msbuild was the first and most used way to consume coverlet (before vstest integration) where instrumentation/report generation is injected throught msbuild Tasks https://docs.microsoft.com/en-us/visualstudio/msbuild/task-writing?view=vs-2019.
This libs will be used wiht dotnet test /p:CollectCoverage=true command line, that property(/p:) options will inject msbuild plumbing for instrumentation/coverage collect.

So for new xunit project coverlet.collector will be always present, instead you have to add reference to coverlet.msbuild if you want to use msbuild task integration.

Today if I want to use a msbuild task integration with a new xunit template project I'll have both reference in my csproj(due to default template), I could remove collector one, but it's not mandatory, simply collector won't be used.

I don't like the name VSTest integration

The VsTest integration is due to fact that code is embedded in repo https://github.com/microsoft/vstest Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console

The preferred way to do coverage is dotnet test --collect:"XPlat Code Coverage" because the strict integration/coordination with vstest engine solve some know issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test
This "coordination" is not possible with msbuild (/p:CollectCoverage=true) or .net tool(coverlet ...).

[PureKrome]

Wow - kewl! great reply ... there's a lot of info to try and understand so I'm going to have a go at summarising what I've just read to confirm my understanding.

• coverlet.collector is used to get coverage data when the --collect:"XPlat Code Coverage" argument is used with dotnet test
• you do NOT need to use the --collect parameter to collection coverage data. If that's the case, then you'll need to use the msbuild answer or the dotnet tool answer.
• preferred way is to use dotnet test --collect:"XPlat Code Coverage" which means you can stop using the msbuild answer.
• vstest is the magical testing engine that has the smarts to use a collector if one has been defined.

Question:

• what is the difference between dotnet test and vstest ? is this the same thing?

From the VSTest repo: Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.

... but dotnet test is a command line program that is outside of Visual Studio.

powers test explorer

Is this the Test Explorer in Visual Studio?

and vstest.console

what is this? Again, is this tied to Visual Studio?

I feel like the name vstest is not _really_ a visual studio thing, but a dotnet core thing and all of this can work on a machine without Visual Studio installed ... meaning the name is very misleading.

[MarcoRossignoli]

coverlet.collector is used to get coverage data when the --collect:"XPlat Code Coverage" argument is used with dotnet test

Yep

you do NOT need to use the --collect parameter to collection coverage data. If that's the case, then you'll need to use the msbuild answer or the dotnet tool answer.

when I say msbuild I mean msbuild tasks injected when you run dotnet test thank's to reference to coverlet.msbuild that imports props/targets to msbuild pipeline.

preferred way is to use dotnet test --collect:"XPlat Code Coverage" which means you can stop using the msbuild answer.

Yep

vstest is the magical testing engine that has the smarts to use a collector if one has been defined

Yep vstest is the general purpose way to say Visual Studio Test Platform that is the runner and engine that powers test explorer and vstest.console. Actually when you run dotnet test underneath you're running dotnet core version of vstest.console https://github.com/microsoft/vstest/blob/2eea2f100a7bf5108665b2aff3d7499d636210b9/src/Microsoft.TestPlatform.Build/Tasks/VSTestTask.cs#L20

what is the difference between dotnet test and vstest ? is this the same thing?

Are the same there is only a small difference https://github.com/microsoft/vstest/issues/637#issuecomment-291762855 and seem that will be merged in 3.1.1xx

https://github.com/microsoft/vstest-docs/pull/196
https://github.com/microsoft/vstest/issues/1453#issuecomment-540984756

[MarcoRossignoli]

@anaether any on this?

[anaether]

@MarcoRossignoli i will try your steps in the days of the next week. This week i dont have time to check it.

[PureKrome]

@MarcoRossignoli I'm wondering if the README should be slightly tweaked with some of this info. I mean, if I was totally confused (which is _very_ common for me) .. then so might others .. and your answers (above) are really really helpful. Maybe ... I could try a PR with some minor changes/additions to the README?

[MarcoRossignoli]

@PureKrome sure feel free to do a PR, we could update the "how it works" section below the workflow https://github.com/tonerdo/coverlet#How-It-Works

[HarelM]

I've tried to read this thread and understand what has changed since my upgrade to .net core 3.1 but got confused.
I'm using the following command to generate lcov coverage file:
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:Exclude="[IsraelHiking.Common]*" --logger trx
Can be found here:
https://github.com/IsraelHikingMap/Site/blob/ef8b4edf5cb55a23591ef70de0ea4bd6530b91bc/Scripts/TestAndPublishWeb.ps1#L7
Do I need to change the command line parameters now?
I have stopped getting the lcov file ever since I migrated my project to .net core 3.1, somewhere around this build:
https://ci.appveyor.com/project/IsraelHikingHost/site/builds/29765493/job/pt911sxw44hfxlsr
Project can be found here:
https://github.com/IsraelHikingMap/Site/
Thanks!

[MarcoRossignoli]

@HarelM I don't see any package ref to coverlet coverage in your csproj

https://github.com/IsraelHikingMap/Site/blob/master/Tests/IsraelHiking.API.Tests/IsraelHiking.API.Tests.csproj
https://github.com/IsraelHikingMap/Site/blob/master/Tests/IsraelHiking.DataAccess.Tests/IsraelHiking.DataAccess.Tests.csproj

If I search coverlet in your repo https://github.com/IsraelHikingMap/Site/search?q=coverlet&type=Code

I found only command switch but without ref doesn't do nothing, try to follow the guide for msbuild

https://github.com/tonerdo/coverlet#installation-1

BTW I advice to use vstest integration https://github.com/tonerdo/coverlet#vstest-integration-preferred-due-to-know-issue

[HarelM]

Thanks!! I swear it used to work without this reference, I have no clue how, but it did.
Anyway, adding the NuGet dependencies seems to solve the issue locally, I'll check-in and see if I get my coverage back.
I'll try to keep the vstest in mind in case I run into more issues.
Thanks for the super quick response and detailed answer. Keep up the good work!

[MarcoRossignoli]

I have no clue how, but it did.

Maybe some inherited deps props/targets

[MarcoRossignoli]

Close for stale conversation, @anaether when you'll have time to test feel free to re-open or open a new issue.

[MasterS4683]

I am also using the .net core 3.1, test coverage file is not getting generated please help/guide me for the same.

[MarcoRossignoli]

@MasterS4683 can you open I new issue and attach if possible your csproj test file and your commandline?Until now you can take a look at our hello world guide https://github.com/tonerdo/coverlet/tree/master/Documentation/Examples/VSTest/HelloWorld

[anaether]

@MarcoRossignoli The example is complete different. It uses netstandard2.0 but .net core 3.1 dont supports netstandard2.0

[MasterS4683]

Yes I understood, needed guidelines for it, but the problem got resolved by adding the package of coverlet.msbuild. Then it got generated.

I also used the xunit test projects, with all latest updated packages.

[MarcoRossignoli]

@MarcoRossignoli The example is complete different. It uses netstandard2.0 but .net core 3.1 dont supports netstandard2.0

netstandard2.0 is supported by .net 3.1 https://docs.microsoft.com/en-us/dotnet/standard/net-standard#net-implementation-support

netstandard is not a runtime but only an api signature(similar to a big interface to implement) implemented by runtimes.

[MarcoRossignoli]

Yes I understood, needed guidelines for it, but the problem got resolved by adding the package of coverlet.msbuild. Then it got generated.

You should use coverlet collectors https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md because other way suffer some known issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnownIssues.md#1-vstest-stops-process-execution-earlydotnet-test

[MarcoRossignoli]

Updated sample to 3.1 https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/VSTest/HelloWorld/XUnitTestProject1/XUnitTestProject1.csproj

[anaether]

is coverlet also working with NUnit?

[MarcoRossignoli]

is coverlet also working with NUnit?

Yep, coverlet simply instrument and gather hit information on code workflow, it's not related to any testing framework, it acts before and after tests.