Vstest: Code coverage on .NET Core

[twsouthwick]

Description

Code coverage does not work in .NET Core

Steps to reproduce

Run vstest.console.exe /enablecodecoverage /framework:netcoreapp

Expected behavior

Runs the tests with code coverage enabled on .NET Core

Actual behavior

Error: Invalid .Net Framework version:netcoreapp11. Supported .Net Framework versions are Framework35, Framework40 and Framework45.

Environment

VS 2017 RTM
Windows 10 14393.693

[smadala]

Code coverage support not available in 15.0.0. Doc.

Please use /Framework:FrameworkCore10 for .Net core apps.

[smadala]

Error message should display as follows:
Error: Invalid .Net Framework version:netcoreapp11. Supported .Net Framework versions are Framework35, Framework40, Framework45 and FrameworkCore10.

[twsouthwick]

What is the timeline to support .NET Core? Is this different on the console vs in Visual Studio?

[codito]

@twsouthwick this will likely come in the next release (15.1 or 15.2 preview). We're still working out the details. Current thought process:

• In console, code coverage will show up as a data collector (spec is here). User may install the code coverage nuget package, and use it like dotnet test --collect:coverage
• In VS, it will show up as the existing Analyze Code Coverage in test explorer

We're first enabling the infrastructure (data collectors), this will allow the community to author code coverage extensions for dotnet test. VS code coverage support will use the same infra.

[robalexclark]

I was struggling with this, but found Microsoft.CodeCoverage started working when I added <debugtype>Full/>to the project containing the assembly to have coverage:

I'm using Microsoft.CodeCoverage 1.0.2, xunit 2.2.0, Ms.Test.sdk 15.0. The project under test is .net framework 4.5.1 and uses nuget packages for .net core 1.1.1 throughout

Can someone else please check and confirm?

  <PropertyGroup>
    <TargetFramework>net452</TargetFramework>
    <RuntimeIdentifier>win7-x86</RuntimeIdentifier>
    <PreserveCompilationContext>true</PreserveCompilationContext>
    <AssemblyName>ExtranetManager</AssemblyName>
    <OutputType>Exe</OutputType>
    <DebugType>Full</DebugType>
    <PackageId>ExtranetManager</PackageId>
  </PropertyGroup>

[twsouthwick]

@robalexclark your test is running on .NET Framework which is supported at the moment. The debugtype had to be changed because by default it builds with the new portable format, which is currently unsupported with vstest. If you switched your test to run as a .netcoreapp you won't see any coverage results.

uses nuget packages for .net core 1.1.1 throughout

This is probably not what you think it means. The nuget packages may support .NET Core 1.1.1, but are either targeting .NET Standard or have a .NET Framework version. These are then being selected when running on .NET Framework. The goal of .NET Standard is to allow the same binary to run on multiple runtimes (.NET Core and .NET Framework are two distinct runtimes).

If the project is one you can share, we can verify if this is the case based on the project files and packages.

[robalexclark]

Apologies yes you are right - I am running on .net 4.5.1 as runtime.

[twsouthwick]

Easy mistake to make in some projects!

[baor]

Hi all!
Do you have any plans about implementation code coverage for dotNetCore for Linux?

If you have any not very complex PR or subtasks, I am ready to help.

[nperno]

@codito I would like to try out the "dotnet test --collect:coverage" for my dotnet core VSTS builds. Do you have any insight/guidance as to when that will be available and we will need to do to get it working?

Thanks for your help!

[markvincze]

Hi all,

Is there any update on whether there will be .NET Core code coverage support for Linux too?

Thanks,
Mark

[HaroonSaid]

Any update - VS 2017 is expensive product one of key features is code coverage

[raffaeu]

Guys just yesterday I found a new Nuget Package:
Microsoft.CodeCoverage version 1.0.3

I updated the package into my XUnit test project and run Tests + Code Coverage in VS 2017 Ultimate.
The tests are executed but the Code Coverage is not even triggered:

So, what's going on? Any official Docs yet?

[raffaeu]

Also if I pass dotnet test --collect:coverage

This is what we get NOT IMPLEMENTED

MSBUILD : error MSB1001: Unknown switch.
Switch: --collect:coverage
For switch syntax, type "MSBuild /help"
Process completed with exit code 1.

[codito]

This feature hasn't shipped yet. We're pretty close though, will provide an update soon :)

[codito]

In 15.3 Preview 1, code coverage for .NET core is available in VS. Instructions are here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#coverage. Please try it out.

We're working on bringing coverage to dotnet test command line.

Sharing an alternate command line in case folks want to try it in CI. This will generate a .coverage file that can be published as reports.

> & "C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" /framework:".NETCoreApp,Version=v1.1" /collect:"Code Coverage" "G:\tmp\trial\bin\Debug\netcoreapp1.1\trial.dll

Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170427-09
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.6614583]   Discovering: trial
[xUnit.net 00:00:00.7762253]   Discovered:  trial
[xUnit.net 00:00:00.8162957]   Starting:    trial
[xUnit.net 00:00:00.9366491]   Finished:    trial

Attachments:
  G:\tmp\trial\TestResults\da20ba2e-2984-4d74-88d4-faf2bf018af0\armahapa_ARMAHAPA-SRV 2017-05-15 18_00_03.coverage

Total tests: 2. Passed: 2. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.1226 Seconds

[raffaeu]

@codito we just tried with these versions:

• Visual Studio 2017 15.2
• xunit 2.2.0
• Microsoft.NET.Test.Sdk" Version="15.3.0-preview-20170502-03"

Tests run and they are green but Code Coverage is not triggered.
Is it working only with MsTest? I hope we don't have to re-write all our tests 😢

[codito]

@raffaeu it is available in VS 2017 15.3 preview 1, you can get the build from https://www.visualstudio.com/vs/preview/. It works with all frameworks :)

[raffaeu]

Oh, thanks I will try then with the preview version
Thank you

[codito]

Currently known issues are tracked here: https://github.com/Microsoft/vstest/labels/exp%3A%20coverage

[cdie]

Hey, finally, great !
Nice to have it again with preview version. And I should also notice that this preview version (15.3) seems to works better than currently supported one (15.2) ...
Can't wait to use it every day

[twgraham]

@codito presumably the dotnet test version of the coverage runner you speak won't be the same as that vstest uses right? When you say something like that I get hopeful for those like me on mac/linux. Is there seperate issues for the dotnet test coverage support?

[codito]

@twgraham test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

[npasqua]

@codito Furthermore, is it the plan to include support of dotnet test code coverage in 15.3 / 2.0 initial versions or should I be exploring other avenues for the time being? If not the stated versions, is there a proposed time/release for this behavior? (so I may explain to others the decision to use 3rd party tools instead of wait for official MSFT tools)

Additionally - is this bug tracking adding the code coverage behavior for vstest AND dotnet test? Or is there another bug/feature for dotnet test specifically

[amattie]

@codito Do you have high-level timing you can share for when dotnet test with code coverage will be cross-platform? Q3? Q4? 2018? It'd be nice to know just for planning purposes.

[Alxandr]

I tried out dotnet test --collect:coverage on .NET CLI 2.1.0-preview1-006547, and this is what I got:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'

Seems someone needs to quote a path somewhere :P

[harshjain2]

Hi @Alxandr
I tried to repro this using .NET CLI 2.1.0-preview1-006567. It seems to be working as expected for me.

If it is a consistent repro, please share repro steps so that we can look into the issue.

[Alxandr]

Huh. That's weird. I'll try to do some more testing with it.

[dnelly]

I'm getting a similar error that that's giving me an object reference not set to an instance of an object.

 1>Done building target "ShowCallOfVSTestTaskWithParameter" in project "UnitTestProject1.csproj".
   Done executing task "CallTarget".
   Using "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task from assembly "C:\Program Files\dotnet\sdk\2.0.0-preview1-005977\Microsoft.TestPlatform.Build.dll".
   Task "Microsoft.TestPlatform.Build.Tasks.VSTestTask"

Object reference not set to an instance of an object.
Object reference not set to an instance of an object.

   Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask" -- FAILED.
 1>Done building target "VSTest" in project "UnitTestProject1.csproj" -- FAILED.

(VSTest target(s)) -- FAILED.

[harshjain2]

@dnelly Please consider upgrading to latest version of CLI.
@Alxandr also faced similar issues in 2.1.0-preview1-006547 which are not there in 2.1.0-preview1-006567.
Do let us know if you face these issues even after upgrading to latest version.

[Alxandr]

Still same error.

[dnelly]

@harshjain2 Yeah, it was still the same for me as well.

Here's the current version that I'm using 2.1.0-preview1-006632

[dnelly]

the error seems to come from this Test Execution Command Line Tool Version 15.3.0-dev

[harshjain2]

@dnelly @Alxandr Created an issue https://github.com/Microsoft/vstest/issues/902 for investigating this further.

[dnelly]

Thank you!

[mstephens-adig]

Is there a way to get the code coverage flags to work on VSTS? The best I can tell VSTS only has 15.0 available and not any of the pre-releases.
Everything works fantastic locally. Thanks for all the hard work!

[harshjain2]

Created a separate issue for providing support for dotnet test --collect:"Code Coverage" https://github.com/Microsoft/vstest/issues/981 as this issue is for code coverage support for vstest.console.exe

[Mardoxx]

Still doesn't work in VS2017 15.3 (incl preview 7) on framework projects. My coverage only shows for my test library, coverage for the tested library is not shown. I guess it didn't ship? Live unit testing works however!

Works with following workaround https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#working-with-code-coverage

[harshjain2]

@mstephens-adig VS 2017 15.3 is released now.
https://blogs.msdn.microsoft.com/visualstudio/2017/08/14/visual-studio-2017-version-15-3-released/
Upgrading to latest released version in VSTS should solve your problem.

[harshjain2]

@Mardoxx Thanks for trying it out. This workaround is required for code coverage to work on .NET Core

[harshjain2]

Please try code coverage and let us know if there is any feedback/issues.
I am closing this issue for now.

[cdie]

@harshjain2 Is the nuget package a temporary workaround or a permanent one ? I mean I don't want to spent a lot of time adding it now to remove it in two months...

[cdie]

It seems it's not working with .NET Core 2.0 ?
Or maybe it's with xunit 2.2.0 ?

[IvanAlekseev]

Coverage is still not working in VSTS, at least I can get the test results
https://github.com/Microsoft/vsts-tasks/issues/5066

[Mardoxx]

I would imagine this is because your agent is not up to date. Check what version of VS2017 your agent is using. Any >15.3 should work.

[IvanAlekseev]

@Mardoxx It is Hosted VS 2017, they updated to 15.3 recently

[Mardoxx]

Ah! Now I look, mine don't have coverage either!

[harshjain2]

@harshjain2 Is the nuget package a temporary workaround or a permanent one ? I mean I don't want to spent a lot of time adding it now to remove it in two months...
@cdie We are tracking the issue to remove need to manually add nuget package here.
https://github.com/Microsoft/vstest/issues/852.
We will prioritize it soon.

[a0z0ra]

Hi, we followed all above, and still no luck. Found different paths from log:

2017-08-24T22:58:11.1157918Z Attachments:
2017-08-24T22:58:11.1157918Z d:\a\1\s\TestResults\672524f6-3c0c-4dc4-86c9-4b58f01c9577\buildguest_FACTORYVM-AZ220 2017-08-24 22_58_06.coverage
2017-08-24T22:58:11.1157918Z
2017-08-24T22:58:11.1157918Z Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
2017-08-24T22:58:11.1157918Z Test Run Successful.
2017-08-24T22:58:11.1167892Z Test execution time: 4.6567 Seconds
2017-08-24T22:58:11.1508492Z Results File: d:\a\1\s\TestResults\buildguest_FACTORYVM-AZ220_2017-08-24_22_58_11.trx
2017-08-24T22:58:13.3673473Z ##[section]Async Command Start: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command End: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command Start: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command End: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command Start: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command End: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command Start: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command End: Telemetry
2017-08-24T22:58:13.3673473Z ##[section]Async Command Start: Publish test results
2017-08-24T22:58:13.8241527Z Publishing test results to test run '59'
2017-08-24T22:58:13.8241527Z Test results remaining: 1. Test run id: 59
2017-08-24T22:58:14.1399971Z ##[warning]Skipping attachment as it exceeded the maximum allowed size or not available on disk: d:\a\1\s\TestResults\buildguest_FACTORYVM-AZ220_2017-08-24_22_58_11\In\FACTORYVM-AZ220\buildguest_FACTORYVM-AZ220 2017-08-24 22_58_06.coverage

Could this be the case why codecoverage not published?

Also, once it's published, should it be visible under Code Coverage?

[harshjain2]

CC @nigurr for vstest-task

[nigurr]

Hi,

We are fixing this issue as part of this https://github.com/Microsoft/vsts-agent/pull/1149/files
Will update the thread once we release the fix.

Sorry for the trouble.

[xperiandri]

https://developercommunity.visualstudio.com/comments/84459/view.html

[Oceanswave]

Getting similar results as folks above

is referenced in my test project.

dotnet test --collect:coverage

Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program'

Object reference not set to an instance of an object.
Object reference not set to an instance of an object.

[davidreher]

even if I use vstest.console.exe like described in https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md#coverage like this:

vstest.console.exe --collect:"Code Coverage" --framework:".NETCoreApp,Version=v2.0" bin\Debug\netcoreapp2.0\some.tests.dll

I am getting:

Microsoft (R) Test Execution Command Line Tool Version 15.0.26929.2
Copyright (c) Microsoft Corporation.  All rights reserved.

Error: The test source file "--collect:Code Coverage" provided was not found.
Error: The test source file "--framework:.NETCoreApp,Version=v2.0" provided was not found.

I am running VS2017 Enterprise V15.4.1

[sbaid]

pl use the vstest.console.exe in testplatform folder for .NET core

C:\Program Files (x86)>dir vstest.console.exe /s
 Volume in drive C is OSDisk
 Volume Serial Number is 4445-DB68

 Directory of C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow

10/10/2017  09:42 AM           137,824 vstest.console.exe
               1 File(s)        137,824 bytes

 Directory of C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\**TestPlatform**

09/06/2017  09:43 AM           133,240 vstest.console.exe
               1 File(s)        133,240 bytes

[davidreher]

@sbaid: Thank you so much, this works now :)

[davidreher]

Is there a way to pass the --collect:"Code Coverage" switch to dotnet vstest instead of using vstest.console.exe?

[mundhras]

I get the below error. Has any one see/resolved this.

Test run will use DLL(s) built for framework NETCoreApp,Version=v2.0 and platform X86. Following DLL(s) do not match framework/platform settings.
.Test.dll is built for Framework 2.0 and Platform X86.

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatformvstest.console.exe" F:\Vsts\agent-1_work\14\a\Release\Deploy\Publications\\.Test.dll /EnableCodeCoverage /logger:trx "/TestAdapterPath:\"F:\Vsts\agent-1_work\14\a\"" /Framework:NETCoreApp,Version=v2.0 /logger:console;verbosity=normal

[BlitzkriegSoftware]

This is still an issue for XUnit on .NET Core 2.0 projects in the latest VS 2017. Why is this closed?

[Oceanswave]

Since Microsoft is unwilling or unable to provide a solution at this time, check out Coverlet:

https://github.com/tonerdo/coverlet

[IvanAlekseev]

Thats the latest update from Microsoft I believe - https://github.com/Microsoft/vstest-docs/blob/master/RFCs/0021-CodeCoverageForNetCore.md

[swati-jain13]

Is there a solution available now for dotnet test --collect:Code Coverage for linux containers?