Vstest: Microsoft.NET.Test.Sdk visual studio code coverage integration

[TonyTroeff]

Description

I had a hard time analyzing my code coverage using Visual Studio.
I thought that it is a local problem from my machine, but actually it wasn't.
All my projects, referencing Microsoft.NET.Test.Sdk version 16.3.0 and 16.4.0, cannot be analyzed.

However, after downgrading them to version 16.2.0, everything works fine.

I would expect to be able to upgrade the package and still be able to analyze my code coverage.

Environment

Windows 10
Visual Studio Enterprise
I am using NUnit for testing purposes if this is related to the issue.

[fdecaire]

I discovered this issue early this morning. Microsoft.NET.Test.Sdk version 16.3.0 and 16.4.0 does not work. 16.2.0 works.

Evironment

Windows 10
Visual Studio Enterprise
xUnit
.Net Core 2.2

[singhsarab]

This is a known issue.

When trying to run the tests via VS, VS runs the tests setting the architecture as x86 for AnyCPU projects.
You can try setting the Preferred Architecture for AnyCPU projects to x64 and then retry.

That said, we are making a fix to support code coverage for .Net core for x86 scenario.
@nohwnd We can use this issue for tracking the fix that we trying to make.

[nohwnd]

Related #2283

[amanda-barton]

I'm having the same issue. Downgrading SDK to 16.2.0 worked.

Environment
Windows 10
Visual Studio Enterprise 2019
xUnit
.Net Core 3.0

[koenigm-seattleu]

Same issue. UT on Xamarin Forms default project with nUnit tests fail to get CC under 16.3.0 or higher, but work under 16.2.0

[ManishJayaswal]

@nohwnd - we should try to fix this in 16.5. I have added the pri1 label to this.

[nohwnd]

I am not sure what the original repro here was, because this behavior depends on the pieces of test platform that are is shipped with VS. That said this was fixed and will work once that fix is released in VS. 🎉🎉🎉 Here are my results in the current stable public release of VS and the latest internal preview of VS. I am using older versions of test sdk, mstest and coverlet, but I see the same results as with the latest versions of those packages.

If anyone can describe how to reproduce this without updating VS I would be interested.

[AnthonySteele]

I see that Microsoft.NET.Test.Sdk V16.5.0 is out
Does this revert back to the behaviour in v16.2.0 ?

Or if the fix is elsewhere which version of which piece should I check? Particularly in ADO pipelines that use dotnet tasks, not Visual Studio.

[nohwnd]

@AnthonySteele 16.5.0 fixes the issue with collecting the CC on x86, but getting the fixed version is not as simple. The test platrform is shipped in multiple ways, and depending on where you run it from, different versions might run. I am not claiming to be an expert in this (yet 😁 ) so feel free to prove me wrong here, we will both learn:

In Visual Studio there is one version of test platform built-in, in the currently released stable that is 16.4.0, in our internal preview that is 16.5.0 because I inserted the new version last week. Once that is made available to the public you will be able to replicate what I did in the screenshot above. I am looking at Properties > Details> ProductVersion of "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe"

There is a version of test platform that ships with dotnet and when you run dotnet test, it currently uses version 16.3.0 (in dotnet sdk 3.1.101). This version will also update in the next public release to 16.5.0. I can confirm that running via dotnet test will use the built in version and will fail to run x86 libs on 16.3.0.

A third way that we are shipping test platform is the TestPlatform nuget package, that was updated and contains the vstest console that you can use to test your dlls.

So because you were able to break it and fix it by changing nuget package I assume that the task is using the test console from the nuget package that your app is referencing:

C:\projects\temp\mstest\mstest1> & "~\.nuget\packages\microsoft.testplatform\16.5.0\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" /platform:x86 C:\projects\temp\mstest\mstest1\bin\Debug\netcoreapp3.1\win-x86\mstest1.dll /collect:"Code Coverage"
Microsoft (R) Test Execution Command Line Tool Version >>>>>>>>>16.5.0<<<<<<<<<<
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  √ TestMethod1 [26ms]

Attachments:
  C:\projects\temp\mstest\mstest1\TestResults\1e530f52-3ebb-409d-9bbe-1af591e0b038\nohwnd_NOHWND-SURBOOK 2020-02-11 08_57_05.coverage
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,4649 Seconds

C:\projects\temp\mstest\mstest1> & "~\.nuget\packages\microsoft.testplatform\16.3.0\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" /platform:x86 C:\projects\temp\mstest\mstest1\bin\Debug\netcoreapp3.1\win-x86\mstest1.dll /collect:"Code Coverage"
Microsoft (R) Test Execution Command Line Tool Version >>>>>>>>>>16.3.0<<<<<<<<<<<<<
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
>>>>> No test is available in C:\projects\temp\mstest\mstest1\bin\Debug\netcoreapp3.1\win-x86\mstest1.dll. 
>>>>> Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Attachments:
  C:\projects\temp\mstest\mstest1\TestResults\44ed4457-e7d9-4630-8fd0-d85845752bd7\nohwnd_NOHWND-SURBOOK 2020-02-11 08_57_56.coverage
Additionally, path to test adapters can be specified using /TestAdapterPath command. Example  /TestAdapterPath:<pathToCustomAdapters>.
C:\projects\temp\mstest\mstest1>

So that was why I was asking how exactly you are are able to reproduce the problem because it's very easy to figure out what happens 🙂

[AnthonySteele]

It seems to be working for us now, in ADO pipelines with dotnet test, and using the package Microsoft.NET.Test.Sdk V16.5.0.

Thanks.

[nohwnd]

Cool, thanks for reporting back, it is nice to see that your issue was fixed. I will experiment with this a bit when I have time to get better understanding.

/fyi @AbhitejJohn I think you fixed this one, it's working 😀

[koenigm-seattleu]

Confirmed today that in nuget Microsoft.NET.Test.Sdk 16.5.0 the issue is NOT fixed.

Please release a 16.6.0 to fix the problem.

[fdecaire]

I tried version Microsoft.NET.Test.Sdk V16.5.0 and it did not fix my issue.

[AbhitejJohn]

@koenigm-seattleu , @fdecaire : Are these tests targeting X64? If not, you'd want to check if you have the right version of the dotnet runtime for x86 and x64 installed locally. dotnet --list-runtimes should give that to you.

[koenigm-seattleu]

@AbhitejJohn, I don't understand your suggestion.

When 16.5.0 Microsoft.NET.Test.Sdk is installed, code coverage is not generated, when downgraded to 16.2.0 and code coverage is generated.

No changes are needed to the targeting of x86 or x64 or such is needed reproduce the issue.

It is just the version of the Microsoft.NET.Test.Sdk that has the problem.

It works in 16.2.0 and fails in 16.3.0, 16.4.0, and 16.5.0.

[AbhitejJohn]

@koenigm-seattleu : Yes, that could be a little confusing. So the versions of the Test SDK 16.4 and later fixes a bug to honor the default processor architecture specified to it which is x86 by default VS 16.4 and earlier. Now version 16.2 of the Test SDK on the other hand always defaulted to the x64 runtime which was installed by default for older runtimes (2.* for instance). Now when you updated to use the latest Test SDK, the tests were rightly run against the architecture of your choice - x86 (even if you haven't explicitly set it) but it would fail to find the runtime for that architecture unless you go ahead and install it. (download link)

This developer community thread would help with a few more details. This should ideally just work with VS 16.5 preview-2 and later.

Hope that helps.

[AnthonySteele]

After expermenting some more with multiple projects and encountering errors, I have found that ability to use Microsoft.NET.Test.Sdk Version 16.5.0 in Visual Studio Test Explorer works sometime, but not others.

And I found a way to fix it.

Hint: it depends on which .NET Core SDKs you have installed under C:\Program Files (x86)\dotnet\shared, when of course they should be under C:\Program Files\dotnet\shared, due to the "default processor architecture" mentioned above.

Example

I have Visual Studio 2019, Version 16.4.4

Symptom: Open Visual Studio, compile the solution, open Test Explorer and run tests. It does not work. Tooltip at the bottom reads Test run aborted after < 1 ms. Total tests - 0 passed, 0 failed, 0 Skipped

There are more details, but you need to go to the Output tab, and select "Show output from: Tests" And you see output like:

Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: 
  Testhost process exited with error: It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '3.1.0' was not found.
  - No frameworks were found.

or

Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: 
  Testhost process exited with error: It was not possible to find any compatible framework version
The framework 'Microsoft.NETCore.App', version '2.1.0' was not found.
  - The following frameworks were found:
      3.1.1 at [C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App]
You can resolve the problem by installing the specified framework and/or SDK.

Although this framework does exist in C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App but not C:\Program Files (x86)\dotnet\shared

Fix 1:

In the tests.csproj file, change Microsoft.NET.Test.Sdk Version 16.5.0 to Microsoft.NET.Test.Sdk Version 16.2.0, rebuild, run tests, it works.

Fix 2:

Go to https://dotnet.microsoft.com/download/dotnet-core
Locate a SDK version appropriate to these tests, e.g. https://dotnet.microsoft.com/download/dotnet-core/3.1
download the Windows x86 version (not x64!) and install.

Note that it is now present under C:\Program Files (x86)\dotnet\shared\Microsoft.NETCore.App
If you have been deleting things in the C:\Program Files (x86)\dotnet\shared\ folder like I have, you may been to "run as administrator" and click "repair" to get it there.

Run tests, it works.

[nohwnd]

@AnthonySteele This sounds like it agrees with what Abhitej described above, when you have 16.4.0 then you need the appropriate x86 runtime installed for the tests to work. 16.5.0 fixes that, but if you just install the package in your project it is not honored by VS, nor dotnet test (unless you provide vstestconsole path). I would suggest we wait till the versions of VS and sdk that have 16.5.0 are publicly available and if that sill does not work, lets look at it in a new issue. Sounds good? 🙂

[AnthonySteele]

Yes, all good. I'm just documenting the workarounds and details that I dug up so that you don't get asked the same thing again. (and for my own reference next week :P )

[nohwnd]

Additionally the test host is loaded from the test.sdk package dependencies, so fixes that are specific to test host are directly consumable from the nuget package. That is why some of the behavior above changed after upgrading the test.sdk