Vstest: Add support for dotnet test --collect:"Code Coverage"

[harshjain2]

Description

Add support for code coverage through dotnet test on Windows OS machine.
dotnet test --collect:"Code Coverage"

[arpit-nagar]

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in https://github.com/Microsoft/vstest/issues/579#issuecomment-305374623 for code coverage support in linux. Using this comment to keep track of linux support request)

[nulltoken]

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

[ctro]

:+1: I'd very much love to have working code coverage within the confines of .NET Core.

[harshjain2]

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

[arpit-nagar]

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux.
Is there any way we can achieve code coverage on Linux for dotnet core project.

[andyjansson]

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

[rol-dave-overall]

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'
Object reference not set to an instance of an object.

[jstallm]

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

[edumserrano]

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

[RoCore]

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

[FastNinja]

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known?
It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds.
The Linux community would like to embrace dot net core on board - but look it just simply can't.

[ollyjshaw]

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

[DavidParks8]

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

[tracyhoward]

@harshjain2 Is there a timeframe for when this will be available?
Add support for dotnet test --collect:"Code Coverage" #981

[pvlakshm]

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017.
However, the following will be possible with .NET Core tests:

1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

[arpit-nagar]

@pvlakshm regarding point 2.
Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

[pvlakshm]

@arpit-nagar, we are working towards that.

[henryJack]

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

[ISemyanchuk]

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error :
"Failed to initialize client proxy: could not connect to test process."

[GrimSmiler]

Hi!

So as far as i understand code coverage can only be generated localy by VS Enterprise users?
There are no howtos for setting it up in TFS with a build agent (on premises) without installing VS enterprise on a build agent machine?

[arpit-nagar]

Do you have any timelines, By when it will be available for .net core on linux environment without VS installation ?

[henryJack]

Not promising that this was removed from the 15.5 milestone and not added to any other milestone. Could @pvlakshm at least add it to the Future milestone so it's not forgotten about?

[IvanAlekseev]

@GrimSmiler I was able to get coverage on VSTS, maybe it can help you https://github.com/Microsoft/vsts-tasks/issues/5066#issuecomment-329745668

[henryJack]

Thanks for keeping this on sprint @pvlakshm :)

[sadiamukhtar]

I am running xunit for my netcoreapp2.0
I am using the vsts task dotnet 2.* (preview) task. I am not able to get code coverage out of the task. Any update on this issue?
Is this the right place to ask this question?

Thanks!

[alexzaytsev-newsroomly]

@IvanAlekseev that only works when your build machine is on Windows. We got Enterprise VS, but we use linux build agents (as we deploy and do everything on Windows). From what i can see, still out of luck with code coverage on linux.

[SleeperSmith]

Guys, is there any intention to progress this on linux?

How are we supposed to take dotnet core on linux seriously when we can't compute code coverage on it? Compile and test on windows and then run on linux? Or test on linux and ignore the test coverage altogether?

[henryJack]

It is extremely frustrating that such a basic feature is not available...

[AdrienneCohea]

Can we remove the enhancement label for this issue? Code coverage is not an enhancement. The dotnet tooling is not feature complete until code coverage works on all supported platforms.

[lsthiros]

Commenting to track/+1 this issue as it is affecting the QA of projects that I am contributing to. Serious development cannot take place with C# or any CLI system on a non-windows platform without some code coverage metric, and every search for such functionality on linux or MacOs leads to this issue.

[thisisthekap]

TD;DR: Code coverage is desperately needed.

I totally agree with @lsthiros: Code coverage is key for serious development. I also think that the dotnet core tooling on non windows platforms is simply not complete without code coverage (as @AdrienneCohea already pointed out).

[FastNinja]

Need code coverage - cannot take Dot Net Core seriously if such a basic engineering task is getting pushed between different teams and no one is interested in implementing it.

[axelheer]

Oh, calm down please.

I'm waiting very badly for dotnet test --collect coverage too, but we have still an alternative with @OpenCover and @ReportGenerator (since the "beginning"). Even the .NET Core team uses this combination to analyse their tests, as far as I know.

[thisisthekap]

@axelheer Afaik, @OpenCover is windows only. Do you know of any alternative working on non windows systems (linux, macos)?

[alexl0gan]

I am currently working on an .NET Core SDK project where the corresponding SDKs are written in Python, Java and Javascript. All these languages have support for code coverage on linux. The fact that .NET Core is lagging behind means I get roasted daily!

[voronoipotato]

https://github.com/SteveGilham/altcover

Maybe contribute to altcover. It does the same thing as opencover but is cross platform.

[lionelarmanet]

There's also this project https://github.com/lucaslorentz/minicover, seems a bit young but worth testing.

[pkaramol]

So, to sum up, there is currently no way of running tests with cov support on linux for .NET core projects? Even with mono that runs .exe files on Linux from what I understand?

[alexl0gan]

Mono has coverage built in. For example to get coverage from NUnit tests you can run

mono --debug --profile=log:coverage,covfilter=+[MyProject]MyProject packages/NUnit.ConsoleRunner.3.7.0/tools/nunit3-console.exe --noh --inprocess Tests/MyTestProject.csproj

However this is limited as it only provides coverage of the methods you call, so tests that hit all lines in one method out of twenty would give you 100% coverage.

[pkaramol]

Minicover states it is based on this: https://github.com/gaillard/SharpCover

Has anyone tried it on Linux? Hasn't been updated for a couple of years though...

[lionelarmanet]

Did not try SharpCover, but minicover was easy to setup / launch. However, I had problems to integrate it with my sonar toolchain, so I chose to make coverage w/ OpenCover on a windows node (I'm working with a distributed ci).

This article may be a good start : https://automationrhapsody.com/net-core-code-coverage-linux-minicover/.

My 2 cents.

[anjmao]

This is so sad. I can't recommend .NET Core for anyone. How can someone decided that such basic but important feature like test coverage is not supported by default. It is 2018 and we are not able to run .NET Core tests coverage on Jenkins or Travis during CI build. Amazing. Just Amazing.

[alexl0gan]

Looks like this decision has come from a product manager level, who want code coverage to be a feature in visual studio enterprise only

[voronoipotato]

Gotta say, it seems like late-capitalism is the only thing holding us back from better software. Thankfully I have confidence in my OSS comrades to fight the planned obsolescence and ultimately save Microsoft from their own undoing.

[mralnis]

Common its 2 years and still no metrics ....

[RoCore]

well right now it has prio 1 ;)

[sesispla]

Maybe coverlet is the solution we've been waiting for...

https://www.hanselman.com/blog/AutomaticUnitTestingInNETCorePlusCodeCoverageInVisualStudioCode.aspx

Thx @tonerdo for the amazing job and @shanselman for the diffusion

[shanselman]

@tonerdo

[GrimSmiler]

Used coverlet's feature to export the results in OpenCover format as part of TFS build step.
Also managed to feed it to SonarQube server. Not sure about the validity of metrics, but i can assure that such pipeline is possible.

@tonerdo +1

[pvlakshm]

We are starting on enabling code coverage via dotnet test as described in this RFC.

[janaka]

@pvlakshm just wanted to vote for *.nix support. We (at Hudl) are moving our backend from .NET Framework on Windows to .NET Core on Docker on Linux. Having to generate coverage on Windows build agents which is not ideal. Coverlet and Altcover seem to be options but not sure how good the coverage reports are.

[zfilipov]

@pvlakshm The RFC states that support is expected to be in version 15.8 onward, expected to release Q3 2018. Is this still the case?

In the meantime, is there any way to get code coverage results into a CI/CD pipeline for .NET Core 2.0 test scenarios?

[mbenua-mparticle]

@zfilipov DotCover just released a patch as of maybe a week ago that is now correctly supporting .NET Core 2.0. You do have test on Windows, but at least you can collect coverage metrics without a ton of wrangling on CI/CD.

[ahaleiii]

@zfilipov We are using coverlet with our .NET Core 2.0 applications. It works with our VSTS CI/CD pipeline (automated).

[zfilipov]

@ahaleiii @mbenua-mparticle Thank you for your input!

@ahaleiii When you generate the code coverage JSON file, is there another task you need to add to the pipeline to upload the coverage results to TFS / VSTS?

[pvlakshm]

@janaka, noted your vote.
@zfilipov, yes it is very much the case. We should be shipping a "preview" to NuGet very soon.

[ahaleiii]

@zfilipov We are generating the coverage in Cobertura format. Yes, there is another task in VSTS used to upload the results (Publish Code Coverage Results).

dotnet test task arguments for working with coverlet:
--configuration $(BuildConfiguration) --no-build --filter "Category=Unit|Category=UnitTest" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutputDirectory=test-results

[pvlakshm]

We have shipped a preview version of the Test SDK with support for code coverage for .NET Core (Windows) projects generating portable/Full PDBs.

As called out in the RFC, this will emit a .coverage file and will require VS to view the results.
Enhancements to viewing the results, and support for Linux/Mac will be taken up separately.

Here is a sample that can be used to try out the above with MSTest, and xUnit, and NUnit based tests: https://github.com/pvlakshm/Samples/tree/master/NetCoreSampleForCC.

[eduherminio]

VS or VS Enterprise, @pvlakshm?

[pvlakshm]

VS Enterprise has the viewer.

[zfilipov]

@pvlakshm Thank you for the update! Will we be able to upload the code coverage results to VSTS/TFS as part of a CI/CD pipeline for viewing?

[kennethkryger]

@zfilipov I've got it running on TFS 2018 with code coverage results in the build reports.

I've added the .NET Core task to the process with the following properties:

• Version: 2.* (preivew)
• Command: test ;-)
• Path to project(s): **\*.Tests.csproj
• Arguments: --logger "trx;LogFileName=test-results.trx" --collect "Code Coverage"

After the task above I've got a Publish Test Results task with the following properties:

• Test results format: VSTest (even though my unit tests are written using XUnit!!!
• Test results files: **\*.trx
• Search folder: $(System.DefaultWorkingDirectory)
• Under "Advanced" the "Upload test results files" is checked

[Coder3333]

@kennethkryger I added those steps with TFS Online, and I get test results, but no code coverage. There is a warning in the test step, "Data collection : Could not find data collector 'Code Coverage'". Do you have something additional installed to get the code coverage?

[kennethkryger]

@Coder3333 That might be because the preview SDK isn't installed on the Hosted VSTS build agents? If you look at the logs of your builds, you should be able to see what version of the Test Execution Command Line Tool your build i using... Mine says: 15.8.0-preview-20180605-02

You might be able to install the preview SDK? See more info here (I haven't tried it!)

However, I see the same error as you do, if I don't add the version above to my Unit Test project - e.g. if I reference version 15.7.2...

[zfilipov]

@kennethkryger Thanks for the detailed steps, is this for the Microsoft preview SDK? Or are you using Coverlet for this?

[kennethkryger]

@zfilipov I'm using the Microsoft preview SDK.

[zfilipov]

@kennethkryger Thanks Kenneth. Are you able to publish your code coverage results to TFS as well?

[kennethkryger]

@zfilipov Yes I am - works perfectly! I haven't tried VSTS - only TFS (on-premise).

BTW: For the .NET Core task in my build definition I had to change the following property, in order to get my code coverage results into SonarQube:

Arguments: --logger "trx" --collect "Code Coverage" (removed ;LogFileName=test-results.trx)

[clairernovotny]

Very much looking forward to this, but I must add that we need support for embedded portable pdb's. All of our projects use those for user ease and it's critical that we get coverage for those too.

We've been relying on third party tools that have long supported this, but are eager to use the framework tools.

[mraigonsp]

Is it possible to generate the coverage file as @kennethkryger suggested with VS Professional? My idea is to generate the report using VS Professional installed in a agent, and then publish the results to SonarQube. I don't need the Enterprise Version to see the report, is it required to be able to generate it?

[RoCore]

@mraigonsp you can install vs enterprise edition on your CI machine. the agent do not require VS licence at all. do someone know when vs 15.8 will be released?

[mraigonsp]

Ok @RomanKernSW so we only require a license when "opening and using VS" right? so, isn't a licence necessary to use the build tools? That solution may work for us now, but I can not understand how such an essential feature as code coverage is not available in a Professional licensing.

[zfilipov]

@kennethkryger Kenneth, I was trying to follow your example using the preview Microsoft Test SDK but am getting same the error as @Coder3333 . My test executor shows: "Microsoft (R) Test Execution Command Line Tool Version 15.7.0" but I have installed the preview version of the SDK

[kennethkryger]

@zfilipov Sorry for not getting back before now...
I realize that I've might have forgotten a rather important step - the .NET Core Tool Installer

This is the first task in my build definition!

I've set the properties to:

• Package to install: SDK (contains runtime)
• Version: 2.1.400-preview-008975

[JohnGalt1717]

Any update to this working properly under the sdk on Linux? Right now you can't get code coverage if you're using docker because your tests run in docker on Linux.

[henryJack]

+1 on that. Desperate for this

[ollyjshaw]

I've been following the thread for over a year. All the well meaning work arounds require windows and / or VisualStudio. As @JohnGalt1717 suggests, we're done when you can get a coverage report using docker via a linux base image.

[latop2604]

Coverlet as global tool is available on nuget : https://www.nuget.org/packages/coverlet.console/
Same for reportgenerator https://www.nuget.org/packages/dotnet-reportgenerator-globaltool/

Perfect combo

[JohnGalt1717]

That's nice and all but it should be built into the dotnet sdk since this is a basic feature of devops and required for testing to be relevant. Hence why this is an issue for the dotnet team to implement properly instead of hacking with VS.net

[sinapis]

@kennethkryger when i try installing 2.1.400-preview-008975 on my agent i get this error:

Could not fetch download information for version 2.1.400-preview-008975. Please check if the version specified is correct. You can refer https://github.com/dotnet/core/blob/master/release-notes/releases.json

I tried installing the latest available .net core sdk version (2.1.302) but then I get the following error:

2018-07-25T15:38:37.1279471Z [command]C:\hostedtoolcache\windowsdncs\2.1.302x64dotnet.exe test D:a1\sStringCalculatorStringCalculatorTestsStringCalculatorTests.csproj --logger trx --collect "Code Coverage" --logger trx --results-directory D:a_temp
2018-07-25T15:38:37.3433537Z MSBUILD : error MSB1008: Only one project can be specified.

I deleted --logger "trx" from command line to dotnet test and then the error went away but no trx file was generated, even through command looks good:

[command]C:\hostedtoolcache\windowsdncs\2.1.302x64dotnet.exe test D:a1\sStringCalculatorStringCalculatorTestsStringCalculatorTests.csproj --collect "Code Coverage" --logger trx --results-directory D:a_temp

any help would be appreciated

--EDIT--
another piece of information:

Microsoft (R) Test Execution Command Line Tool Version 15.7.0
...
Data collection : Could not find data collector 'Code Coverage'

[li0nsar3c00l]

So aparently ths is only working with .NET Core SDK 2.400 and higher which is only in preview right now? @kennethkryger can you confirm?

[kennethkryger]

@sinapis: I can't really help with the debugging the installation process, as the ".Net Core Tool Installer" I mentioned above, simple did the trick for me... Maybe someone else reading this issue has some input on this?
But my "Test Execution Command Line Tool" shows version 15.8.0-preview-20180605-02, and not 15.7.0 as yours - so the tools hasn't been installed properly, that's for sure...

@li0nsar3c00l: You are correct - currently .NET Core SDK 2.1.400 is needed, which should include version 15.8 of the Test Execution Command Line Tool... and yes, it's currently only available in preview...

[rifaterdemsahin]

any update on linux for --collect:"Code Coverage" ?

[escalonn]

For those trying to use .NET Core SDK 2.400 in VSTS - the .NET Core Tool Installer may have been able to get that version a couple weeks ago but currently it only pulls from a releases.json file which does not include any recent preview builds.

[clairernovotny]

The VSTS task was just fixed and can install the preview builds again.

[mpetito]

I've installed the preview sdk and can get the binary coverage outputs, but is there a way to convert these to xml? I'm trying to import them into sonarqube which expects the .coveragexml format.

[epoyraz]

@mpetito i had the very same problem. I added an inline PowerShell Task after the Test Task to convert the binary coverage.

# copy and rename the *.coverage to the DefaultWorkingDirectory and rename it to TestCoverage.coverage
Get-ChildItem -Path $(System.DefaultWorkingDirectory) -Filter *.coverage -Recurse -ErrorAction SilentlyContinue -Force | %{Join-Path -Path $_.Directory -ChildPath $_.Name } | Copy-Item -Destination $(System.DefaultWorkingDirectory)\TestCoverage.coverage

# convert binary to xml
CodeCoverage.exe analyze /output:$(System.DefaultWorkingDirectory)\TestCoverage.coveragexml $(System.DefaultWorkingDirectory)\TestCoverage.coverage

This is just a workaround. It works for me. I am interested in a proper solution, too :)

[mpetito]

@epoyraz Thanks! I had explored using CodeCoverage.exe but unfortunately that requires an installation of Visual Studio Enterprise edition (which i do not have). I was hoping there is a way to do this with the .NET Core sdk directly, otherwise the preview sdk generates a very nice binary coverage file that I cannot use.

[codito]

CodeCoverage.exe is available in this nuget package: https://www.nuget.org/packages/Microsoft.TestPlatform/. I think it should work without VS installed.

Look for the relative path: tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe in the nuget package.

[SharpNoiZy]

Please support the ExcludeFromCodeCoverage Attribute.

Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

[smadala]

Please support the ExcludeFromCodeCoverage Attribute.
Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

Related: https://github.com/Microsoft/vstest/issues/1548

/cc: @pvlakshm @cltshivash @mayankbansal018

[clairernovotny]

Please do not close this issue. This issue should remain open until embedded PDB's are supported.

[henryJack]

@livarcocc - This is still an issue and should not be closed

[livarcocc]

I didn't really close this issue. github did when @smadala made a PR tagged as fixing it. @smadala can you re-activate it?

[Larswa]

@mpetito Codecoverage comes with the nuget package, and it comes with the VSTS "Visual Studio Test Platform Installer" task.
In my case I end up with this on my buildserver that doesn't have VS2017 installed btw ...
D:\VSTSAgent\_work\_tool\VsTest\15.8.0\x64\tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe

[vikasillumina]

@harshjain2 any timelines for providing this support? I am currently working on building the CI/CD for our .NET core based projects and we use docker for running the CI/CD and this feature is missing. If code coverage is natively supported with dotnet test command itself it will make it very easy to use this.

[mpetito]

@vikasillumina use coverlet for now.

I ended up using coverlet even in Windows build environments as the workarounds required above are less than ideal. The issues I ran into included:

• Output of 0 byte .coverage files without any warning or explanation.
• Binary coverage outputs that I couldn't use otherwise.
• Separate package references needed to download CodeCoverage.exe to run binary-to-xml conversion.

After spending hours trying to get this to work, I added coverlet and had my coverage imported into sonarqube in about 10 minutes.

[li0nsar3c00l]

@vikasillumina 2.1.400 is not in preview anymore, so the feature exist and can be used. I got it working in my pipeline. only downside is, that you only get the coverage report and not an output of the percentage.

[JohnGalt1717]

@li0nsar3c00l Does this work on Linux now?

[joymon]

@codito ,
I tried to install the Microsoft.TestPlatform nuget package to .Net standard test project. But the codecoverage.exe didn't showed up. Any ideas other than checking in the codecoverage.exe with in source

[vikasillumina]

@li0nsar3c00l I actually tried the 2.1.400 version and coverage option does work, however its only supported on Windows. Please correct me if I am wrong. I am looking to use the same command natively on docker running linux. :(
Thanks @mpetito for your inputs and lessons learnt on the way to use coverlet, I will give it a try.

[codito]

@joymon it seems to work for me, here's what I tried:

# create a new netstandard2.0 class lib
> dotnet new classlib
> dotnet add package Microsoft.TestPlatform
# see the csproj for classlib, it should have Microsoft.TestPlatform added now
> dotnet restore
> ls ~/.nuget/packages/microsoft.testplatform/15.8.0/tools/net451/Team\ Tools/Dynamic\ Code\ Coverage\ Tools/CodeCoverage.exe
# ~/.nuget is the default nuget package cache on my box. See global-packages details in https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-global-packages-and-cache-folders

Please don't check-in the exe in any case :) As an alternative, consider adding a step in your CI to download the nuget package directly. Like this in powershell script:

> Invoke-WebRequest "https://www.nuget.org/api/v2/package/Microsoft.TestPlatform" -OutFile Microsoft.TestPlatform.nupkg
> Expand-Archive Microsoft.TestPlatform.nupkg -DestinationPath ./test-pkg

[vikasillumina]

@mpetito Thanks once again for your input and I was able to get the Coverlet to work without any issues other than creating the Result directory to save the output file on Windows (few team members use windows for local development) On mac it worked like charm! I didn't need to do any conversion just used this command:
dotnet test %csproj file% /p:CollectCoverage=true /p:CoverletOutput='./Results/' /p:CoverletOutputFormat=opencover and I got the report in xml format without any conversion needed. Please note I used 2.2.1 version of the coverlet.msbuild library https://www.nuget.org/packages/coverlet.msbuild/2.2.1 in case it made the difference.

I tried opencover and cobertura and both options worked fine for me.

[StuffOfInterest]

Even with the 15.8 tools there appears to still be issues with code coverage running under some scenarios. I was able to get the command line to work on my local PC. When I tried adding it to a TFS build things didn't go so well. The command that ran was:

"C:\Program Files\dotnet\dotnet.exe" test B:\agent\_work\43\s\Something.Web.Tests\Something.Web.Tests.csproj --no-build --verbosity normal --collect "Code Coverage" --logger trx --results-directory B:\agent\_work\_temp

What came out was many, many instances of errors like this:

Failed   Index_view_convention_sets_ActivityName_to_View_All_XYZ
Error Message:
 Assembly Initialization method Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration threw exception. System.OutOfMemoryException: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.. Aborting test execution.
Stack Trace:
    at Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration(TestContext context)

This is being run on a build server with TFS on premises. The build server has Visual Studio 2017 Enterprise version 15.8 installed and the project being tests has the Microsoft.NET.Test.Sdk version 15.8.0 installed in it. If the "--collect Code Coverage" is removed the test run works fine.

[mayankbansal018]

@StuffOfInterest , we have seen similar issue, where OOM exception is thrown when Coverage is enabled.
To validate that your issue is similar to one we fixed internally, can you please provide following info

• Are you facing this issue consistently with Coverage on, or is it intermediate?
• Are your tests netcore of full CLR based?
• When you run things locally, can you please take a quick look, & check what is the approx peak memory consumption of "testhost(if fullCLR tests)" else check "child dotnet.exe(for netcore tests)" for process. Hoping your tests don't take too much time to run.

[StuffOfInterest]

@mayankbansal018 , I'm seeing the issue very consistently. All is happening with netcore. The memory footprint is not excessive.

First, a bit of background. I'm converting over an existing code base of libraries from .NET Framework to .NET Standard. There are currently six library projects. Each library project has a unit test project targeting .NET Core 2.1. Across the six test projects there are just over 1000 unit tests in total, which unfortunately only provide about 50% code coverage.

The .NET build step in the TFS Build service runs "dotnet test" individually for each unit test project. One of the test projects only has eight unit tests in it right now. To make the verification as easy as possible I setup a build to only run testing for this one project.

Both "dotnet build" and "dotnet restore" ran fine. When it came time for "dotnet test" to run the OOM exceptions happened for all eight tests. I had Task Manager running on the build server while the build was running. No processes went over 250 MB of memory consumption while this was happening.

The test project has version 15.8.0 of the Microsoft.NET.Test.Sdk referenced. I've installed version 2.1.402 of the .NET Core SDK on the build server. Version 15.8.0 of Visual Studio Enterprise is installed on the test server. Later this morning I plan to update Visual Studio to 15.8.4, but this will take some work being that the build server does not have direct Internet access.

[mayankbansal018]

@StuffOfInterest , apologies for getting so late on this. Can you please create a separate bug for your issue, & see if you can attach a repro project to it.
At the very minimum please do attach the diagnostics logs that you get when you generate Code Coverage for your projects.

[StuffOfInterest]

@mayankbansal018 , it looks like that may not be necessary. I updated to the recently released Microsoft.NET.Test.Sdk version 15.9.0 and updated the build server's Visual Studio to 15.8.6. Now, the tests are working!

One of those two appears to have solved the memory issue.

Thanks.

[mayankbansal018]

@StuffOfInterest , great to see it got resolved. 😄

[JohnGalt1717]

Still requires Visual studio and isn't a dotnet.exe function so it doesn't work in docker for instance. And using coverlet etc. is VERY slow and often fails.

This needs to be independent of Visual Studio and just work as part of dotnet.exe

[henryJack]

Still desperately needed for linux/Docker and is the only thing standing in the way before my corporation adopts .NET core.

[carlosrpg]

Not resolved to me, If enable code coverage I get the following error: (I'm using on TFS)
2018-10-16T20:56:55.6793227Z Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to initialize client proxy: could not connect to test process. 2018-10-16T20:56:55.6793768Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable1 sources) 2018-10-16T20:56:55.6794376Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

• I have the <DebugType>Full</DebugType> o both test and test project
• I have the nuget for <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" /> on the test project
• I Have Microsoft.TestPlatform -Version 15.9.0
• the vstest command is: [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" MyTestDLL.dll /EnableCodeCoverage /logger:trx /framework:.NETCoreApp,Version=v2.0

It works fine locally but not on the TFS.

If I remove the /EnableCodeCoverage it runs all the tests just fine

[mayankbansal018]

@JohnGalt1717 , the coverage support is not longer tied to Visual Studio, & is available with dotnet.exe. Can you please try it again with latest dotnet.exe, & make sure you also have latest version of Microsoft.Net.Test.SDK. Please let me know if you face any errors with it.

@henryJack As mentioned previously, it is part of our black log, we are exploring what all gaps we need to fill, to move our current implementation from windows only, & take it x-plat.

@carlosrpg, the error you have shared seems unrelated to code coverage. Can you please try setting env variable VSTEST_CONNECTION_TIMEOUT=200, & try again, also please enable diagnostics logs via --diag:, & share those with us.

[JohnGalt1717]

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

[lomholdt]

First of all, it seems that a requirement is the 2.1.403 SDK when running dotnet test --collect:"Code Coverage". Each test project will then have a coverage file. But What about multiple test projects? If I wan't to use eg. TFS to show an overview of coverage this is not supported for multiple files. Is it possible to create a combined code coverage file across multiple test projects in a solution?

[mayankbansal018]

@JohnGalt1717 I agree that you cannot run this on Linux docker, as I mentioned previously as well, we have already started investigating, that what will it take us to move from windows only implementation, to x-plat. I don't have a clear timeline yet for this. I can share a plan once we have finalized it internally.

@lomholdt if you are running CodeCoverage via dotnet in TFS, & use publish coverage task, it would merge the multiple coverage files generated, & would show a consolidated report. However for dotnet code coverage currently does not merge the coverage files for multiple projects across the run. Can you please open a bug for the same. I don't want it to get lost in this conversation.

[lomholdt]

@mayankbansal018 Thanks. I have created it here.

[codekoenig]

if you are running CodeCoverage via dotnet in TFS, & use publish coverage task, it would merge the multiple coverage files generated, & would show a consolidated report.

@mayankbansal018 I tried to add a "Publish Code Coverage Results" task to have the generated coverage files from different projects merged, but it shows the following warning:

##[warning]Multiple file or directory matches were found. Using the first match: [Path to .coverage file]

So it doesn't seem to be able to merge multiple files. I used the following minimatch pattern in the "Summary file" field to select all generated coverage files: **\TestResults\**\*.coverage

Also I noticed that the code coverage couldn't be uploaded even for the first match as ist seems to expect another format as dotnet test did generate:

##[warning]Failed to read [path to coverage file]: Data at the root level is invalid. Line 1, position 1

Is there any guidance how to make this work?

[RehanSaeed]

Using dotnet test --logger "trx;LogFileName=Foo.trx" --collect:"Code Coverage" produces some crazy long file names and sub folders:

  rehan.saeed_MY_MACHINE_NAME_2018-10-24.12_25_21.coverage
rehan.saeed_MY_MACHINE_NAME_2018-10-24_12_25_18
  In
    MY_MACHINE_NAME
      rehan.saeed_MY_MACHINE_NAME_2018-10-24.12_24_56.coverage

A few questions come to mind:

1. Is it possible to provide a custom name, like we can for .trx files above?
2. I'm not very familiar with .coverage files but why are there two of them per test project and why do we have nested folders with the machine name repeated?
3. I know you are going for unique-ness but do we really need the username and machine name in there? If I could provide a custom name, I suppose all of that could go away.

All of this could be a lot cleaner.

[mayankbansal018]

@acesiddhu Can you please help @codekoenig regarding publishing multiple coverage files via publish code coverage task

[mayankbansal018]

@codekoenig @lomholdt , I just checked with team, & if you are using dotnet task to run tests in vsts pipeline, it should publish test results, as well as code coverage files, & merge them as well. "Publish Code Coverage" task however does not work with ".coverage" files, it works with "Cobertura" & other coverage outputs.

If you still face issue with dotnet task to run & publish coverage results, please run your build under diagnostics via setting build env variable "system.debug=true", & share those logs with us.

[codekoenig]

@mayankbansal018 Thank you, that actually seems to work now! That makes everything so much easier. Big thanks to the team for working on this!

Although when it comes to Code Coverage, the results do not seem to be correct. First I noticed coverage is reported in the TFS build, but with a very low blocks/line count that is wrong:

More interestingly, when downloading the .coverage file from the TFS build and opening in Visual Studio, more accurate numbers are shown that differ from the reported values in TFS:

When looking into the hierarchy, I can see that there are still some projects missing. I'm investigating, might be my fault, but the difference in reported coverage on TFS and what's actually inside the .coverage file seems definitely odd.

Are there known issues with this feature?

Edit: I just tried another build and the coverage was suddenly spot on correct and TFS as well as Visual Studio reported the exact same block count:

I immediately started another build and then the coverage results were suddenly completely off again. No checkins were committed in the meantime, no changes to the build were made:

Seems there might be a race condition or something happening during merging the different .coverage files ... any ideas what could cause this?

[RehanSaeed]

Is there an ETA on getting "Publish Code Coverage" task working with.coverage` files?

[Azrrael]

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

We have the exact same issue. We need Code Coverage running on a dockerized Linux container. Please let us know when you have an ETA for this.

[mayankbansal018]

@codekoenig , I'm not sure why the VSTS, is reporting in-consistence numbers regarding coverage results, where as VS IDE shows correct.
Can you please share a coverage file from one such scenario, where number reported were incorrect. We could try to upload the file from our end, & see if it shows any discrepancy.

@RehanSaeed I'll float this though internally. It's possible I'm missing something, & there is no need to publish coverage task to upload .coverage files as well.

[mayankbansal018]

@PBoraMSFT , @cltshivash , @pvlakshm do we have an estimate for CodeCoverage on x-plat?

[RehanSaeed]

@mayankbansal018 Azure DevOps has a task to run tests and accept coverage files but I use Cake build to run dotnet test myself which generates .coverage files. There is a built in task to accept code coverage files but it does not support the .coverage format.

[Tonvengo]

Running dotnet test with --collect "Code coverage" includes in a code coverage dll libraries referenced in a test project. Does anyone has the same error?

[mayankbansal018]

@Tonvengo user can configure what all dll's should be reported in coverage, please check here

[codekoenig]

@mayankbansal018 The problem is two-fold. One one hand, some of the generated coverage reports shows inconsistent results in VSTS and Visual Studio. At the same time, it's clear that not all projects are included in those reports as these files are noticably smaller than they should be and I can see in Visual Studio that some projects are missing from the report.

While on other occassions, I get a coverage report file that seems to be complete and contains all projects. And in this case, the results are also the same in VSTS and Visual Studio.

I have two sample files, one for each scenario I could provide you, but I don't want to share it publicly as it's a confidential project for a customer. Is there a way I can provide them to you over a private channel? Thanks!

[valdasmisevicius]

@codekoenig @mayankbansal018 Any progress about different numbers in VSTS and Visual Studio?

[mayankbansal018]

Apologies for delay on this. I was OOF for entire dec.
@codekoenig you can share the file over maban at microsoft dot com

[steinmr]

@codekoenig Did you resolve your issue? We have the exact same scenario, but in TFS 2018.3. For those builds that show incorrect numbers, you can go into "Tests"-tab, download all coverage files and combine them in VS. The number shown there is correct, and different from the incorrect percentage shown in "Summary"-tab. So it appears that VSTS/TFS has all the files, but in some builds it calculates the total incorrectly, and also does not merge the files when you try to download them from the "Summary"-tab. I took a look in the TFS-database, and the files coverage data can be found there, but I was unable to find the total percentage, maybe it is calculated on the fly.

[codekoenig]

@steinmr I'm in (e-mail) contact with @mayankbansal018 and some of his colleagues about this issue and I sent them sample files and screenshots and will provide them any logs they might ask for. So no solution yet, but investigations are ongoing and I will report back as soon as we know more.

The problem you are having definitely sounds like mine, although in my case, whenever TFS reports incorrect numbers, I can also see missing projects when I download and open the coverage file in Visual Studio. So the coverage report in Visual Studio is incorrect too, just _differently incorrect_, so to say. But maybe these are two unrelated issues that just happen to appear at the same time. We have a lot of projects (about 30) in our solution.

[steinmr]

@codekoenig Ok, thanks for keeping us posted.
When you say that you download the coverage files, do you do that from the summary page (as a single file) or as attachments on the tests runs on the test page? Because in the first case, I also get inconsistent numbers and a lot of missing projects (seems to be to be a random file out of all the coverage files, but not sure). But in the latter case I see all the correct coverage files (as attachments to the test runs), identical to those generated on the build server.

[codekoenig]

@steinmr Ah, missed that in your post - yes I'm talking about the single/merged file from the summary page. I downloaded now the individual coverage files from the test page and you are right, if I merge them in Visual Studio manually, the results are correct. That might be an important find, I'll send the individual files to the team working on the issue too.

And, of course, looks like we have exactly the same issue now.

[nigurr]

@codekoenig we are acknowledging this bug in code coverage merge at Azure DevOps Service.
We are running into race condition where data might be shown incorrect if many code coverage files exist.

I will raise separate tracking item which can be tracked (don't want to extend this thread with this)

[Lobosque]

What is the current state of code coverage? I see that there is a nuget package with a new major version released a few days ago: https://www.nuget.org/packages/Microsoft.CodeCoverage/
Does this adds support for Linux code coverage? If not, is there an ETA?

[acesiddhu]

@Lobosque we are actively working on enabling CC on Linux. The latest package that was shipped doesnt have that support yet.
/cc: @abhishekkumawat23 @cltshivash @PBoraMSFT

[ghost]

@acesiddhu Is there any update in enabling CC on Linux? Does the 16.1.0 nuget have it?

[acesiddhu]

@accu-joeykarns as we speak the support is currently under development. We are integrating it with coverlet. Here is the documentation https://github.com/tonerdo/coverlet/pull/402 that has details on how this is going to work

[RamiSh]

@codekoenig we are acknowledging this bug in code coverage merge at Azure DevOps Service.
We are running into race condition where data might be shown incorrect if many code coverage files exist.

I will raise separate tracking item which can be tracked (don't want to extend this thread with this)

hey -- where is the separate tracking item for this? and where are we with merging multiple code coverage files?

[codekoenig]

@RamiSh https://github.com/microsoft/azure-pipelines-tasks/issues/9581

[gremlm]

When using code coverage with vstest.console.exe it is impossible to specify location of the output .coverage file! That problem makes usage of the code coverage with vstest.console.exe useless!

[steinmr]

@steinmr Ah, missed that in your post - yes I'm talking about the single/merged file from the summary page. I downloaded now the individual coverage files from the test page and you are right, if I merge them in Visual Studio manually, the results are correct. That might be an important find, I'll send the individual files to the team working on the issue too.

And, of course, looks like we have exactly the same issue now.

@codekoenig I added a workaround for our common problem in microsoft/azure-pipelines-tasks#9581

[giggio]

I just tested the coverlet integration and it works. Is this going to be ported over to VSTest or that will be the recommended approach? If so, this issue should be closed and the docs should be updated.
BTW, it works on Windows, Linux, Mac and anywhere .NET Core works.

[vikasillumina]

@giggio this is great news that it works on all the platform. Please can you share details of your cli command and version of dotnet core you used? I would like to try it.

[gremlm]

By the way, how it can be executed just in .Net, not under .net core?

Sincerely, Grigoriy Milman

On Aug 28, 2019 5:59 PM, vikasillumina [e-mail] wrote:

@giggiohttps://github.com/giggio this is great news that it works on all the platform. Please can you share details of your cli command and version of dotnet core you used? I would like to try it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/981?email_source=notifications&email_token=AMNNEQCEC2CQG7VMLPOGTXDQG4NOPA5CNFSM4DVX5Q6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5M4ULQ#issuecomment-525978158, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMNNEQFEFKGEOWCSKMX2DPLQG4NOPANCNFSM4DVX5Q6A.

[giggio]

@vikasillumina just follow the steps on their readme: https://github.com/tonerdo/coverlet#installation
It works on .NET Core and .NET Framework, and on .NET Core they have a 'msbuild' way and a 'vstest' way. Both work, but have different capabilities (and different bugs). :)

[vikasillumina]

@giggio I am actually already using coverlet something like this:
dotnet test $project \ /p:CollectCoverage=true \ /p:CoverletOutput='./TestResults/' \ /p:CoverletOutputFormat=cobertura \ /p:Exclude=someexclusions /p:CopyLocalLockFileAssemblies=true \ --configuration Release \
But I was curious to try any coverage cli options that can natively run with dotnet core itself like this issue asked for:
"dotnet test --collect:"Code Coverage""
But thanks for your reply.

[JesseNewman19]

Gentlemen I have the answer

Instead of:
dotnet test My.ProjectName.csproj --logger trx --collect:"Code Coverage"

Use this instead:
dotnet test My.ProjectName.csproj --logger trx /p:collect="Code Coverage"

If you're in Azure DevOps, do not use their checkbox for collecting test results and coverage, it will execute the first command.

[adaughterson]

Gentlemen I have the answer

Instead of:
dotnet test My.ProjectName.csproj --logger trx --collect:"Code Coverage"

Use this instead:
dotnet test My.ProjectName.csproj --logger trx /p:collect="Code Coverage"

If you're in Azure DevOps, do not use their checkbox for collecting test results and coverage, it will execute the first command.

Yea, this didn't work either. Still get this lovely message:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

[mtrtm]

@JesseNewman19
Your command line works for me, but only outputs a .trx file. In Azure, how are you getting coverage to tie in? Do you convert to one of the two supported formats?

[adaughterson]

EDIT: Ugh, I retract my premature retraction. The proposed solution totally works if I don't try to generically disregard certain assemblies based on attributes using a settings file (see https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis).

It seems that when I use the runtime settings, the problem rears its ugly head again.

Still a problem for me.

[svejnohatomas]

@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

[da-edwards]

@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us.

Thanks

[queen-of-code]

You can use coverlet to collect coverage on Linux. It appears to be the supported way. See:

https://github.com/tonerdo/coverlet/blob/master/README.md#vstest-integration-preferred-due-to-known-issue

On Mar 12, 2020, at 08:09, da-edwards [e-mail] wrote:


@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us.

Thanks

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[da-edwards]

You can use coverlet to collect coverage on Linux. It appears to be the supported way. See: https://github.com/tonerdo/coverlet/blob/master/README.md#vstest-integration-preferred-due-to-known-issue
…
On Mar 12, 2020, at 08:09, da-edwards @.*> wrote:  @nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux? Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us. Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Yeah, we are using Coverlet to output a Cobertura format coverage report but (happy to be wrong here) not been able to integrate Coverlet supported output formats with the Azure DevOps code coverage status policy.

Was hoping to run with something like this and get a .coverage file which should then integrate with Azure DevOps pull requests:

dotnet test --no-build --results-directory:$(System.DefaultWorkingDirectory)/VSTestTestResults/Coverage/ --logger trx /p:collect="Code Coverage"

[nohwnd]

@da-edwards I am sorry I don't have any updates on this issue, I am working on the rest of the pri:1 issues in this repo.

[danilobreda]

I really does not understand how a 2017 issue with 80+ participants and a lot of "reactions" is not taken seriously and with priority... .net core is crossplatform, why the features is not? This creates loopholes. @shanselman im waiting it for 2 years now :( and all my customers would like to use it.

As said before "this feature should be available for all target platforms. It should not require an installation of Visual Studio."

[vikassd2012]

I really does not understand how a 2017 issue with 80+ participants and a lot of "reactions" is not taken seriously and with priority... .net core is crossplatform, why the features is not? This creates loopholes. @shanselman im waiting it for 2 years now :( and all my customers would like to use it.

As said before "this feature should be available for all target platforms. It should not require an installation of Visual Studio."

I believe this is intentional. Microsoft even after moving towards cross platform support still wants people to use their products(Azure pipeline etc) over cross platform products, otherwise I don't see a reason why this issue which have been open for so long have not be prioritized by Microsoft.

[twsl]

How is this not pri:1 @nohwnd ?

[poke]

You do realize that this issue already is priority 1?

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

[vikassd2012]

You do realize that this issue already is priority 1?

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

Is it really a priority 1? It's certainly not being treated as P1 IMHO.
Regarding coverlet we are already using coverlet for last 2 years like most of us on this thread. But would have been nice to use --collect option natively with dotnet test cli.

[clairernovotny]

@vikassd2012

But would have been nice to use --collect option natively with dotnet test cli.

I'm genuinely curious what "native" means to you? Coverlet is a .NET Foundation project, just like .NET Core and many others. It can be invoked with dotnet test --collect:"XPlat Code Coverage", so what makes it not native?

[vikassd2012]

@vikassd2012

But would have been nice to use --collect option natively with dotnet test cli.

I'm genuinely curious what "native" means to you? Coverlet is a .NET Foundation project, just like .NET Core and many others. It can be invoked with dotnet test --collect:"XPlat Code Coverage", so what makes it not native?

@clairernovotny I had tried the dotnet cli with coverage options on a window box and I found the reports to be really informative. So was hoping to take advantage of those reports in linux side. Other concern with using coverlet (no offence to coverlet team, they have done a splendid job to provide us an option for linux world in absence of this feature) is that they have broke in past after upgrades. So if we have these coverage options available within dotnet cli, we would not have this risk. Also we don't need to manage upgrading of coverlet packages along with dotnet core updates. Not sure if this helps you to understand where I am coming from. If not, we can agree to disagree. :) Also if you feel coverlet is going to be the way for dotnet cli to work with on non windows environment, we should close this issue citing to use coverlet instead and it should be included in Microsoft official documentation. So that we have the right expectation for non windows users.

[poke]

Other concern with using coverlet is that they have broke in past after upgrades.

You need to treat it like any other dependency: Upgrades can break stuff so you should always test things when you upgrade. But you can also always stick to older working versions.

So if we have these coverage options available within dotnet cli, we would not have this risk.

As demonstrated above, Coverlet _is_ available within the CLI.

I assume you mean if it came bundled with the SDK. But as far as I am aware, that isn't true for the Windows coverage option either. It's part of the Test SDK.

Also we don't need to manage upgrading of coverlet packages along with dotnet core updates.

Why is this even an issue? You will have to update loads of dependencies anyway: Test SDK, MSTest, xUnit, Moq, …

You seem to assume that stuff from Microsoft is always “safe to use” and that it doesn't break but that simply isn't the case. And if it was perfect, then you will still need to understand that in order to get there they plan and test it properly before releasing updates. And that takes a lot of time and work, which is precisely what is slowing down issues like this.

Not sure if this helps you to understand where I am coming from.

Sorry, but to me this sounds like the common thinking that everything from Microsoft is magically vetted as good, and every alternative from within the community is a suboptimal solution by default. That is really a harmful behavior that needs to stop.

[danilobreda]

@poke

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

So if I have a test project (example) on windows using visualstudio, and i create a automated build (CI) on azure devops... i build on linux and windows for test, so, on windows i can run code coverage, on linux not. I will be forced to use coverlet. So my first question, why does a cross platform have a oneplatform feature?

Are you guys saying that it is better to throw away this tool and guide everyone to use coverlet?
if this is the case, lets update the documentation

_(Azure devops - Pipeline - oficial .net core Test Task 2.x)_

[poke]

If you are _currently_ working on a project that targets Windows and Linux, then yes, you should probably be using Coverlet to generate code coverage if you need that. Not everybody is required to run on Linux though. I know a lot of people, myself included, that still primarily target Windows platforms (for a multitude of reasons). So for those, they can just rely on the VSTest included mechanism here.

why does a cross platform have a oneplatform feature?

There are actually a few of those within .NET Core. WPF and WinForms come to mind but you can also create COM components that only work on Windows. _Why_ does the platform has those? Because it is also the replacement for the .NET Framework and as such requires to replace use cases of it. Not everything automatically works cross-platform then though. Some can be with additional work, e.g. this very issue, but some others probably will never make it to other platforms.

I think it’s wrong to expect platforms that are cross-platform to have every single functionality identical on every other platform. That actually isn’t true for most platforms and languages, where you always have some OS-specific things. Sometimes, those can be abstracted so that you won’t notice it as much, but some other times not.

Are you guys saying that it is better to throw away this tool

That’s probably not the “better” option. This code coverage functionality is also built into Visual Studio and many people rely on it during development. And there are _still_ plans to expand its functionality to other operating systems.

lets update the documentation

It might be a good idea to suggest a change to the dotnet Azure DevOps task, to include a separate option for built-in Coverlet support. So people using Coverlet could just tick a box and the task would pick up the results automatically. It probably doesn’t hurt to suggest that on the Azure Pipeline tasks repo. The only problem with that would be that Coverlet uses a different output format but they also own the PublishCodeCoverageResults task which supports Coverlet’s output.

[vikassd2012]

Other concern with using coverlet is that they have broke in past after upgrades.

You need to treat it like any other dependency: Upgrades can break stuff so you should always test things when you upgrade. But you can also always stick to older working versions.

So if we have these coverage options available within dotnet cli, we would not have this risk.

As demonstrated above, Coverlet _is_ available within the CLI.

I assume you mean if it came bundled with the SDK. But as far as I am aware, that isn't true for the Windows coverage option either. It's part of the Test SDK.

Also we don't need to manage upgrading of coverlet packages along with dotnet core updates.

Why is this even an issue? You will have to update loads of dependencies anyway: Test SDK, MSTest, xUnit, Moq, …

You seem to assume that stuff from Microsoft is always “safe to use” and that it doesn't break but that simply isn't the case. And if it was perfect, then you will still need to understand that in order to get there they plan and test it properly before releasing updates. And that takes a lot of time and work, which is precisely what is slowing down issues like this.

Not sure if this helps you to understand where I am coming from.

Sorry, but to me this sounds like the common thinking that everything from Microsoft is magically vetted as good, and every alternative from within the community is a suboptimal solution by default. That is really a harmful behavior that needs to stop.

This would have stopped 2 years ago if Microsoft had accepted coverlet as an official code coverage option for non windows environment and more _importantly_ close this issue with that explanation. Keeping this ticket open gave false expectations.

@poke you didn't respond to my last portion of comment. It seems you only picked and chose portion of my comments according to your perception about my view on this.

"Also if you feel coverlet is going to be the way for dotnet cli to work with on non windows environment, we should close this issue citing to use coverlet instead and it should be included in Microsoft official documentation. So that we have the right expectation for non windows users."

Anyways I am done following this issue. Will leave the experts here to decide what they want to do.

[danilobreda]

@poke its a good idea to close this issue and indicate that the way for cross plat is coverlet? I think its the better for all new programmers thats is facing this problem... i will open some documentation issues to solve the problem raised previously.

Maybe change the CLI to say that its not supported when the command is called, use coverlet...

Thanks for explanations and your time :D

[nohwnd]

@twsI @danilobreda It is taken seriously, and it already is pri1. I can also explain why this is not done. 🙂 I took over this area in December when I started in MSFT, and since then I am focusing on learning the platform, getting the infrastructure and tests into shape so we can ship easily and realiably, and solving the most pressing bugs. Unfortunately this a more of a feature than a bug, and so the progress here is slower, because it requires design work and high-level and in detail understanding of the platform. I totally understand this is a pressing issue, for you, but also an issue that we want to solve correctly, not just monkey patch it and then suffer with maintaining that for the rest of eterninty 😁

There are more people in our team now, that also pick up momentum, and more will hopefully join.

[da-edwards]

Thanks all for various replies, @poke good shout - I'll raise a github issue for the pipeline repo to see if they can support Coverlet.

Thanks also @nohwnd - just to be clear from my perspective, I'm perfectly happy using Coverlet, we have been for over a year without a single issue, the only thing I want to be able to do is take a view at a PR what level of unit test coverage there is on changed code and currently that's a feature only available on Windows. Will keep following this issue and hopefully you and the team can ship in the near future. At the moment, I'm contemplating moving our builds back on to a Windows box to take advantage of this PR code coverage feature but I would much prefer to stick on Linux.

[nohwnd]

@da-edwards I think this is definitely an important scenario to cover, tagging @jakubch1 so he can keep it in mind in his current research of code coverage.

[syedhassaanahmed]

We need this functionality in our current project. Any update?

[jakubch1]

Did you try coverlet?

[syedhassaanahmed]

@jakubch1 No. Should this issue be closed and indicated that coverlet is the answer, as @danilobreda also pointed out?

[gremlm]

Coverlet is not the answer. This feature should be compatible when executed at solution containing .Net Core and .Net Framework projects.
MS buils can build such solution, all projects.
dotnet - only projects related to .Net Core.
Coverlet only works for .Net Core, so again it is custom feature

[da-edwards]

Hey @nohwnd , @jakubch1 , just wondering if there are any timelines to this one? Keen to make use of the Azure Devops pull request code coverage feature and I'd rather keep our build hosts as Linux but if it's not likely to be fixed soon, I'll move to Windows for the builds, at least in the short term.

Thanks

[PureKrome]

I'm still confused to why people are complaining about _lack of_ code coverage support for XPlat when it is already here and has been for a while.

For example, this repo uses CI (on AppVeyor ... not Azure DevOps) that does linux + codecov:

dotnet test -c $env:CONFIGURATION 
            -v minimal 
            --no-build 
            --collect:"XPlat Code Coverage" 
            --settings coverlet.runsettings 
            --results-directory './CodeCoverageResults'

All the coverlet docs explain this.

In that repo, I added the following:

• coverlet.runsettings so I can be specific about what to EXCLUDE from coverage. For my personal self, I don't want coverage of any tests. I just want coverage of the _actual_ code itself.
• made sure my test projects included the coverlet dependency, which is now _default included_ to any new .net core app "xunit project" in visual studio. If you're not using VS (which is me sometimes + many others, like using VS Code) ... then just _manually_ include the nuget package in your test csproj.

As a bonus, I also (during my CI) upload the codecoverage results to an online codecoverage website. You can swap this out and instead use a code coverage reporting tool instead, on your CI machine.

---

TL;DR;

• there is .net core code coverage for linux and windows
• get familiar the CLI .net commands. e.g. dotnet test
• code coverage can work on any build system which has dotnet core SDK installed. Your own computer, AppVeyor, Azure DevOps and i'm assuming GitHub Actions.

Now, here's an awesome cat to make you smile and positive:

[da-edwards]

Hey @PureKrome ,

Just to be clear on the coverage, I'm very happy with using Coverlet, we've been using it for > 18 months now without issue. However, Azure DevOps doesn't support Coverlet for pull request diff code coverage policies, which we would find useful (blocking PRs with a diff code coverage < x%), as it's currently supported only on Windows. As our build infrastructure is Linux based, that poses a bit of an issue.

Thanks

[PureKrome]

Hi @da-edwards - ok, let me see if I understand the issue, so appologies if i'm getting all of this wrong.

• If codecov < x% (eg. 85%) then 'error' so the CI fails, which means the PR shouldn't be approved.
• CI system is Azure DevOps.

If this is correct, then CodeCov has the ability to error when a "threshhold" is not met.
REF: CodeCov 'threshold' docs.

That said, the threshold attribute is for the 'msbuild' nuget (option #2). I prefer the codecov option #1 'vstest' nuget. Now, the msbuild nuget seems to work on linux though, but i've not done it nor can I speak for it.

Q: Could more be done?
A: Yep

Q: Is it still a bit painful?
A: Yep :(

Q: Warts and all, _could_ it actually work?
A: Could! I've not tried it though but it _might_ work? Maybe post a question in the codecov issues about 'Threshold' argument and option #1 nuget. Maybe it might be a key/value setting in a runsettings file? (I use runsettings files with my codecov + option #1022

HTH.

[da-edwards]

Hey @PureKrome not quite, we're not (yet) interested in the overall code coverage, just on the diff, so if I was to update a method and refactor 4 lines but only 2 of those are covered with unit tests we would like to see a coverage of 50%, regardless of how many other unit tests have been added during the commit.

That's the (very specific) feature that's supported on Windows but not on Linux ... I think anyway :)

Does that make sense?

[PureKrome]

Yep - it does.... but .. I've never heard of that?! TIL!!!

So do you mean this?

[da-edwards]

That's the one ... did you manage that with Linux or is that on a Windows box?

Thanks

[PureKrome]

Windows :( I've not tried on linux (yet). Maybe we can move this convo over to the Codecov repo and re-ask there? (ping me into the issue you create?)

[thiagocamargos]

My two cents here: dotnet core and all its stack (vstest included) is supposed to be cross-platform and support linux.
It is a big issue if I have to change code (and build|release instructions/package management ARE code) if I decide to change platform and everyone tells me the framework /stack is x-plat.
That is misguiding.

When people here tells others to use coverlet instead vstest coverage for linux builds, you are also telling to use coverlet for windows and ignore the vstest feature. And if that is true, then this feature may be considered useless by those who appreciate a cross-platform framework. Thus, it is better to remove it from the application to minimize development complexity.

Telling people to use coverlet is only deprioritizing this issue, keeping it open forever and misguiding people into thinking this will be done. It has been almost 3 years since this was opened and there is still no clear answer from the core team. I have been following it and being notified since the beginning and I'm wondering if 3 years of a open issue is not enough yet.

@nohwnd , @jakubch1 and @AbhitejJohn Please do take this into consideration and give a clear answer. If coverlet should be the answer, please give a statement telling this, and making it clear that vstest IS NOT and WILL NOT be fully compatible with Linux. Also, a guide to cover this need (pun intended) in coverlet should be written/refered to. If Linux support is to be implement, please also make it clear.

just a quick ps: i've been reading this too.

[AceHack]

I would love to hear an update on this issue, it's very confusing on status and plans.

[JohnGalt1717]

Sorry I haven’t had time. Tomorrow morning I’ll try and do something to make it obvious.

Really I think the real solution is this:

https://github.com/microsoft/vscode/issues/97919#issuecomment-630849160

A real standardized test framework in vs code that can be hooked into to handle all of this stuff and give us parity with vs.net

Please thumbs up if you agree.

[pavelhorak]

We’ve been recently reviewing existing .NET code coverage solutions and decided to enhance our dynamic code coverage solution /also referred to as VS code coverage by some/ and make it cross platform. It will continue supporting code coverage for managed code on both .NET/.NET Core frameworks and for C++ on Windows. This would provide the community with a similar dynamic instrumentation solution, where only configured modules loaded at runtime are instrumented, as VS Code Coverage scenarios employed. As a proof of concept, @jakubch1 built a new prototype based on CLR Instrumentation engine profiler and made it run on Linux and Windows. We also plan to support non-binary report format(s) so it will be easily possible to display code coverage results, especially on Azure DevOps. As of now, we encourage community users continue using Coverlet as a cross platform code coverage solution. I hope to be able to share the timeline soon.

[PureKrome]

@pavelhorak (and others in that team):

[kendrahavens]

To add, Coverlet continues to provide an excellent cross-platform code coverage solution. Bringing the built-in dynamic code coverage solution cross-platform won’t replace it since Coverlet will still be the best option for those who need static code coverage (aka ahead-of-time compilation), which is particularly useful for testing services. This effort is an attempt to satisfy the customers who have specifically asked for a dynamic (on-demand instrumentation) cross-platform code coverage solution.

[da-edwards]

Great news! Thanks @pavelhorak and the whole team. Really appreciate your time and efforts on this.

[iyusuboff]

Coming from java, even the simplest tasks are a problem in dotnot

[mikeblakeuk]

We are running
dotnet 5.0 C#
xunit
AzureDevop - both windows and ubunt

We want:

• unit test results in devops,
• code coverage, excluding xunit dlls
• code coverage, results in devops (TBC)
• block the PR if the coverage is too low
• see ticks/crosses in the PR

any ideas what yaml we need to run? / runsettings we need?

[ElvenSpellmaker]

To add, Coverlet continues to provide an excellent cross-platform code coverage solution. Bringing the built-in dynamic code coverage solution cross-platform won’t replace it since Coverlet will still be the best option for those who need static code coverage (aka ahead-of-time compilation), which is particularly useful for testing services. This effort is an attempt to satisfy the customers who have specifically asked for a dynamic (on-demand instrumentation) cross-platform code coverage solution.

We use coverlet, but Azure DevOps's built in Code Coverage checker only supports .coverage and as we build on Linux we can't get the nice code coverage features without moving the build over to Windows... =|

[ElvenSpellmaker]

Is there a way to convert Coverlet Cobertura files into .coverage? I know the reverse is possible.

[IvanAlekseev]

@ElvenSpellmaker we use this on windows machines but I guess it will work on linux as well
You will not get all devops coverage features but at least you can fail build on insufficent coverage.
It uses another microsoft task to achieve that

      - task: DotNetCoreCLI@2
        displayName: Test Projects
        inputs:
          command: test
          projects: $(testProjects)
          publishTestResults: true
          arguments: '--no-build -c $(buildConfiguration) --collect:"XPlat Code Coverage"'

      - script: dotnet tool install --global dotnet-reportgenerator-globaltool --ignore-failed-sources
        displayName: Install ReportGenerator

      - script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:Cobertura
        displayName: Create Code coverage report

      - task: PublishCodeCoverageResults@1
        displayName: 'Publish code coverage'
        inputs:
          codeCoverageTool: Cobertura
          summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'

      - task: BuildQualityChecks@8
        displayName: 'Check code coverage'
        inputs:
          checkWarnings: false
          checkCoverage: true
          coverageFailOption: build
          coverageType: branches
          baseBranchRef: refs/heads/master
          allowCoverageVariance: true
          coverageVariance: 1
          coverageDeltaType: percentage

[ElvenSpellmaker]

@IvanAlekseev We'd really like to use this: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/codecoverage-for-pullrequests?view=azure-devops

But it seems it's not possible to trigger it on anything other than Windows as it needs the .coverage files to work. You'd think in 2020/2021 Microsoft wouldn't work on a feature for Azure DevOps that only worked with .coverage but hey ho.

Thanks for that task, I'll take a look.

[IvanAlekseev]

@ElvenSpellmaker I am with you on this one. It is pretty sad to see 0 progress in Azure DevOps code coverage handling for a year

[ahaleiii]

I am feeling a lot of the same pain points listed in this thread, and I agree they all seem to come back to Azure DevOps Pipeline integration. I think Coverlet is a perfectly viable solution for cross-platform code coverage. The pain isn't that Coverlet can't produce a binary coverage file, or even that vstest can't collect coverage on Linux; the pain is that Azure DevOps requires a binary coverage file in order to use so many of the features it's trying to develop. Unfortunately I do not see this issue resolving that.

I think @thiagocamargos's comment put it succinctly:

When people here tells others to use coverlet instead vstest coverage for linux builds, you are also telling to use coverlet for windows and ignore the vstest feature. And if that is true, then this feature may be considered useless by those who appreciate a cross-platform framework. Thus, it is better to remove it from the application to minimize development complexity.

Telling people to use coverlet is only deprioritizing this issue, keeping it open forever and misguiding people into thinking this will be done. It has been almost 3 years since this was opened and there is still no clear answer from the core team. I have been following it and being notified since the beginning and I'm wondering if 3 years of a open issue is not enough yet.

[jakubch1]

Sorry for delay in providing cross platform code coverage. We are still porting a lot of quite old C++ code to .NET Standard. I believe we are done in 80%. At the beginning we will be able to support Ubuntu and Alpine distributions. Do you use other distributions?

@ganesp @ShreyasRmsft is CC related logic in Azure Devops cross plat? We can already provide you cross plat library for analyzing .coverage file.

[jakubch1]

We added preview support for dynamic code coverage on Ubuntu and Alpine. To try it out you need:

1. Install dotnet sdk >= 5.0.200
2. Add latest Microsoft.CodeCoverage nuget package to your test project. For example: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210419-05&view=overview
3. On Alpine additionally you have to install few dependencies: apk add libxml2 libuuid libunwind

[ElvenSpellmaker]

Hey @jakubch1, thanks for the update. We'll give this a try thanks. ☺️

[ElvenSpellmaker]

Do we need to supply anything special on the build parameters?

[jakubch1]

You can use: dotnet test --collect "Code Coverage"

[ElvenSpellmaker]

Thanks @jakubch1, I'll give this a try. Also is it possible to output both "Code Coverage" and cobertura/open cover formats?

[ElvenSpellmaker]

Hey @jakubch1, I tried adding --collect "Code Coverage" and the package you linked does indeed generate .coverage files now, but it overrides the nice Cobertura output in the Code Coverage tab.

I just can't see us switching until we can have a nice Code Coverage tab as well.

Before the change:

After the change:

[davesmits]

First of all. Awesome that coverage files now work on linux! Big win we can use the PR validation thing of Azure DevOps again.

I just tried to combine coverlet and coverage file to have best of both world but unfortunately it doesnt solve the issue. Azure Devops keeps showing the download to the coverage file :(. Is there a way to have 'both'? the coverage file so the PR validation works and nice reports?

Is there also a easy way to exclude dlls and namespaces for coverlet adding the parameter or do i need to introduce the runsettings xml again?

[John0King]

@pavelhorak

We also plan to support non-binary report format(s) so it will be easily possible to display code coverage results, especially on Azure DevOps.

I'm wonder that will dotnet test support any non-binary report format(s) ? or the plan is only for azure devops.

IMO, the binary format do not have any advantage than text file, ~~and also unfriendly for "diff" in source controll ~~ , and unfriendly for human read.

[jakubch1]

@John0King @ElvenSpellmaker supporting non-binary format for Code Coverage is in our backlog: #2874

[ElvenSpellmaker]

Thanks, I'll be waiting! 🙂

[sergiocyklpoint]

@jakubch1 do you have any plan to release this change into sdk 3.1? Or only sdk 5 and above?

[jakubch1]

Some older versions of vstest.console on Unix have issue to print correctly summary of tests if there is an attachment. So you can try including latest Microsoft.CodeCoverage package to 3.1 project and try: dotnet test --collect "Code Coverage", but you can hit this issue.

[MarkusRodler]

At the beginning we will be able to support Ubuntu and Alpine distributions. Do you use other distributions?

@jakubch1 If Ubuntu is supported, Debian would be also nice

[jakubch1]

@ElvenSpellmaker @davesmits I had some talks with Azure DevOps team. We are still waiting for some features to be delivered in this area. Currently you can't publish .coverage file and cobertura together.

As workaround you can use coverage splitting by:

1) upgrade to: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210511-05&view=overview
2) create runsettings with:

<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
...
               <SplitCoverage>True</SplitCoverage>
...
            <CodeCoverage>
                    ....
             </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

or from command line

dotnet test --collect "CodeCoverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SplitCoverage="True"

You will see 2 attachments (covb and covx) for each module. On Azure DevOps this will enable such code coverage summary:

PR support is also supported when uploading .covx and .covb files.

[jakubch1]

@MarkusRodler we added support for more distributions in version: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210518-01&view=overview

We verified that dynamic code coverage works on:

OS | Version | Architectures | Dependencies | Lifecycle
--------------------------------------|-----------------------|---------------|-----------------------------|----------
Alpine Linux | 3.11+ | x64 | apk add libuuid libxml2 libunwind | Alpine
CentOS | 8+ | x64 | dnf install --setopt tsflags=nodocs --refresh -y dnf-plugins-core epel-release && dnf config-manager --set-enabled powertools && dnf install --setopt tsflags=nodocs -y libunwind | CentOS
Debian | 10+ | x64 | apt-get install libunwind8 libxml2 | Debian
Fedora | 32+ | x64 | | Fedora
Linux Mint | 19.3+ | x64 | apt-get install libunwind8 | Linux Mint
openSUSE | tumbleweed | x64 | zypper install libunwind libuuid1 | OpenSUSE
Ubuntu | 18.04+ | x64 | apt-get install libunwind8 libxml2 | Ubuntu

[Chainsaw-does-brr]

@jakubch1 , tested on ubuntu 20.04 with azure devops vs local Fine Code Coverage (Coverlet).
To be honest, I'm not sure if it's related to linux support specifically (don't have a win machine to compare), but:

• in FCC we got 72,2% coverage
• in Azure DevOps we got only 20,01% on the same solution.

The solution itself is a monorepo with a lot of dependencies between projects. Most of the projects have their own test projects. So this probably means that when you run dotnet test my.sln, coverage counts the lines of project dependencies, but does not append the results of tests of those dependencies neither during run, nor at the backend (Azure Devops).

[jakubch1]

@Chainsaw-does-brr could you please provide both coverage reports?

[Chainsaw-does-brr]

@jakubch1 i wish i could, but unfortunately can't.

You can simulate the setup with having both:

• project A being tested by project TA
• project B that depends on project A and being tested by TB.

I can provide more detail though. This is my 1st encounter with code coverage, so I'm rediscovering the wheel here, bear with me.
Apparently, all the methods generate separate coverage reports per project. All of those reports have missing data on coverage from dependencies, but. FCC under the hood uses ReportGenerator to merge reports from multiple projects.

Azure devops on the other hand, has 2 methods of uploading code coverage data:

• 'Publish code coverage results' that can't merge reports on it's on and will take only the 1st one that matches the filter.
• dotnet CLI - which, i suspect, has the same issue, but I'm not sure about this one.

So, basically, you are expected to use ReportGenerator to merge results from multiple test projects on your own if you want proper coverage data.

On a side note:
I've just tried merging .coverage files with ReportGenerator and, apparently, they must be converted to .xml 1st. Found #1939 and #2424 that request adding output formats and mention that formats will be a part of this issue.
Are there any updates on this by chance?

[jakubch1]

@Chainsaw-does-brr Azure DevOps pipelines are able to merge .coverage files. To make it working make sure that running dotnet CLI is producing trx file for each test project (add --logger trx and also --collect "Code Coverage"). Then add step to publish all your trx files:

    - task: PublishTestResults@2
      displayName: 'Publish Test Results **/*.trx'
      inputs:
        testResultsFormat: VSTest
        testResultsFiles: '**/*.trx'
      condition: succeededOrFailed()

Each trx file should contain link to .coverage file (you need latest stable .NET sdk). Azure DevOps will merge .coverage files. You can also configure PR coverage support: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/codecoverage-for-pullrequests?view=azure-devops

[Chainsaw-does-brr]

@jakubch1 It produces the same output as dotnet cli with publishTestResults: true. In my case it's 20.11%

While doing coverlet -> merge with ReportGenerator -> Publish Code Coverage result route yields: 52,64% (excluded test assemblies and AutoGeneratedAttribute).

ps: thanks for the tip on including coverage in pr, was wondering about this recently. :)

[jakubch1]

@Chainsaw-does-brr we can't investigate it without coverage reports. Feel free to generate feedback here: https://developercommunity.visualstudio.com/search?space=8.

[Keviento]

@jakubch1 That's some awesome news. Do you have any idea when the NuGet will be available on public nuget.org feed ? Thanks !

[jakubch1]

It's already available here: https://www.nuget.org/packages/Microsoft.CodeCoverage/16.10.0
But I recommend using this version: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210607-01 (contains fixes and support for more linux distributions). Both feeds are public

[RehanSaeed]

It's worth noting that Mac doesn't seem to be supported, you get this error which seems to fail the build:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows, Ubuntu and Alpine..
[xUnit.net 00:00:12.03]     [FATAL ERROR] System.AppDomainUnloadedException

https://ci.appveyor.com/project/RehanSaeed/schema-net/builds/39525934/job/51b68p0ks7aj9hpp

If the platform is unsupported, ideally it shouldn't fail the build.

[jakubch1]

@RehanSaeed I'm not sure if failure is related. Could you please run it with --diag log.txt and send us back all log* files which will be generated?

[daneshka]

When I run on this container mcr.microsoft.com/dotnet/sdk:5.0-alpine I get this error

$ dev/CodeCoverage/CodeCoverage.exe analyze   /output:coverage_result.xml $coverage_filename
dev/CodeCoverage/CodeCoverage.exe: line 1: MZ����@
��: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: $���: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚2�E�+�֚W�כ?�֚W�ӛ: not found
dev/CodeCoverage/CodeCoverage.exe: line 3: �֚W�қ4�֚W�՛1�֚i�ӛ:�֚i�қ:�֚/�қ9�֚/�כ.�֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 3: �ך��֚��ߛ: not found
dev/CodeCoverage/CodeCoverage.exe: line 4: syntax error: unexpected ")"

I used the exe file in this package
microsoft.codecoverage.17.0.0-preview-20210712-03.nupkg

[jakubch1]

CodeCoverage.exe is windows only. Please use this: https://github.com/microsoft/vstest/issues/2424#issuecomment-870313818

[ldwedari]

@ElvenSpellmaker @davesmits I had some talks with Azure DevOps team. We are still waiting for some features to be delivered in this area. Currently you can't publish .coverage file and cobertura together.

As workaround you can use coverage splitting by...

@jakubch1 I'm also looking for having cobertura and .coverage published at the same time. I tried the SplitCoverage setting but it only generates a report per dll. Clicking on the links downloads the coverage file for Visual Studio.
Is there a issue number that I can follow to know when this will be implemented?
From what you say: _"We are still waiting for some features to be delivered in this area"_ It seems that this will be implemented soon. Is that right?

[jakubch1]

My team is responsible for just generating coverage report from test run. Publishing and displaying is part of Azure DevOps team. You can try to find or create issues under repo: https://github.com/microsoft/azure-pipelines-tasks

I found few issues related:
1) https://github.com/microsoft/azure-pipelines-tasks/issues/13632
2) https://github.com/microsoft/azure-pipelines-tasks/issues/13448
3) https://github.com/microsoft/azure-pipelines-tasks/issues/13257