Coverlet: Failing to collect code coverage when using .NET Core Fakes preview

[mateusdemboski]

Hello guys! I'm facing a problem here probably related to the coverlet.collector package.

I have a .NET Core project for unit tests using the Fakes preview and vstest, but when I try to collect the coverage from the application, an exception is thrown on the method that uses fakes.
Can this be an issue on the coverlet library?

output runing the vstest.console Extensions.Net.Http.UnitTests.dll /Collect:"Code Coverage" /Enablecodecoverage /Logger:trx command:

C:\Git\extensions\Extensions.Net.Http.UnitTests\bin\Debug\netcoreapp3.1>vstest.console Extensions.Net.Http.UnitTests.dll /Collect:"Code Coverage" /Enablecodecoverage /Logger:trx
Microsoft (R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
Data collector 'UnitTestIsolationV2' message: The data collector 'UnitTestIsolationV2' requested environment variable 'CORECLR_PROFILER' with value '{324F817A-7420-4E6D-B3C1-143FBED6D855}' to be set in test execution environment, but another data collector 'Code Coverage' has already requested same environment variable with different value '{E5F256DC-7959-4DD6-8E4F-C11150AB28E0}'..
Data collector 'UnitTestIsolationV2' message: The data collector 'UnitTestIsolationV2' requested environment variable 'CORECLR_PROFILER_PATH_32' with value 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Platform\InstrumentationEngine\x86\MicrosoftInstrumentationEngine_x86.dll' to be set in test execution environment, but another data collector 'Code Coverage' has already requested same environment variable with different value 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\covrun32.dll'..
Data collector 'UnitTestIsolationV2' message: The data collector 'UnitTestIsolationV2' requested environment variable 'CORECLR_PROFILER_PATH_64' with value 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Platform\InstrumentationEngine\x64\MicrosoftInstrumentationEngine_x64.dll' to be set in test execution environment, but another data collector 'Code Coverage' has already requested same environment variable with different value 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\amd64\covrun64.dll'..
  √ SettingAuthorizationWithNullHttpClientShouldThrowArgumentNullException [15ms]
  √ SettingAuthorizationWithNullAuthenticationHeaderValueShouldThrowArgumentNullException [114ms]
  √ SettingAuthorizationShouldReturnSameHttpClient [7ms]
  √ SettingAuthorizationShouldSetSameAuthorization [1ms]
  √ SettingBaseAddressWithNullHttpClientShouldThrowArgumentNullException [< 1ms]
  √ SettingNullBaseAddressShouldThrowArgumentNullException [< 1ms]
  √ SettingBaseAddressShouldReturnHttpClientWithSameBaseAddress [< 1ms]
  √ SettingBaseAddressShouldReturnSameHttpClient [< 1ms]
  √ SettingTimeoutWithNullHttpClientShouldThrowArgumentNullException [< 1ms]
  √ SettingTimeoutShouldReturnSameHttpClient [< 1ms]
  √ SettingTimeoutShouldReturnHttpClientWithSameTimeout [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException [4ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (00:00:00) [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (-10675199.02:48:05.4775808) [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (-00:01:00) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorWithNullHttpResponseMessageShouldThrowArgumentNullException [6ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenSuccessHttpResponseMessageShouldReturnExpectedHttpResponseMessage [1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException [6ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (BadRequest,,) [3ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (BadRequest,System.Net.Http.StringContent,Content BadRequest) [1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (InternalServerError,System.Net.Http.StringContent,Content InternalServerError) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (ServiceUnavailable,System.Net.Http.StringContent,Content ServiceUnavailable) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (Unauthorized,System.Net.Http.StringContent,Content Unauthorized) [< 1ms]
  X ReadingShouldReturnExpectedContent [71ms]
  Error Message:
   Test method Extensions.Net.Http.JsonHttpContentReaderTests.ReadingShouldReturnExpectedContent threw exception:
Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Failed to get profiler module handle 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\Extensions\Fakes\x64\Microsoft.QualityTools.Testing.Fakes.Instrumentation.dll'. The specified module could not be found. ---> System.ComponentModel.Win32Exception: The specified module could not be found.
  Stack Trace:
      at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.LibraryMethods.GetModuleHandle(String fileName)
   at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.LoadProfilerModule(String profilerPath)
--- End of inner exception stack trace ---
    at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.LoadProfilerModule(String profilerPath)
   at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
   at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
   at Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
   at Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
   at Extensions.Net.Http.JsonHttpContentReaderTests.ReadingShouldReturnExpectedContent() in C:\Git\extensions\Extensions.Net.Http.UnitTests\JsonHttpContentReaderTests.cs:line 19
   at Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.ThreadOperations.ExecuteWithAbortSafety(Action action)

  √ ReadingWithNullHttpContentShouldThrowArgumentNullException [9ms]
  √ CreatingContentShouldReturnObjectContentWithCorrectFormatter [22ms]
  √ CreatingContentShouldReturnObjectContentWithCorrectValue [< 1ms]
Results File: C:\Git\extensions\Extensions.Net.Http.UnitTests\bin\Debug\netcoreapp3.1\TestResults\Mateus_Demboski_W101573QV2_2020-09-25_18_29_56.trx

Attachments:
  C:\Git\extensions\Extensions.Net.Http.UnitTests\bin\Debug\netcoreapp3.1\TestResults\7e780545-dcf4-4375-894a-58e7d4942712\Mateus_Demboski_W101573QV2 2020-09-25 18_29_55.coverage
Test Run Failed.
Total tests: 25
     Passed: 24
     Failed: 1
 Total time: 1.8046 Seconds

output running the vstest.console Extensions.Net.Http.UnitTests.dll command:

C:\Git\extensions\Extensions.Net.Http.UnitTests\bin\Debug\netcoreapp3.1>vstest.console Extensions.Net.Http.UnitTests.dll
Microsoft (R) Test Execution Command Line Tool Version 16.7.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.
  √ SettingAuthorizationWithNullHttpClientShouldThrowArgumentNullException [12ms]
  √ SettingAuthorizationWithNullAuthenticationHeaderValueShouldThrowArgumentNullException [122ms]
  √ SettingAuthorizationShouldReturnSameHttpClient [5ms]
  √ SettingAuthorizationShouldSetSameAuthorization [1ms]
  √ SettingBaseAddressWithNullHttpClientShouldThrowArgumentNullException [< 1ms]
  √ SettingNullBaseAddressShouldThrowArgumentNullException [< 1ms]
  √ SettingBaseAddressShouldReturnHttpClientWithSameBaseAddress [< 1ms]
  √ SettingBaseAddressShouldReturnSameHttpClient [< 1ms]
  √ SettingTimeoutWithNullHttpClientShouldThrowArgumentNullException [< 1ms]
  √ SettingTimeoutShouldReturnSameHttpClient [< 1ms]
  √ SettingTimeoutShouldReturnHttpClientWithSameTimeout [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException [5ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (00:00:00) [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (-10675199.02:48:05.4775808) [< 1ms]
  √ SettingInvalidTimeoutShouldThrowArgumentOutOfRangeException (-00:01:00) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorWithNullHttpResponseMessageShouldThrowArgumentNullException [6ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenSuccessHttpResponseMessageShouldReturnExpectedHttpResponseMessage [1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException [8ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (BadRequest,,) [5ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (BadRequest,System.Net.Http.StringContent,Content BadRequest) [1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (InternalServerError,System.Net.Http.StringContent,Content InternalServerError) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (ServiceUnavailable,System.Net.Http.StringContent,Content ServiceUnavailable) [< 1ms]
  √ EnsuringSuccessStatusCodeWithErrorGivenErrorHttpResponseMessageShouldThrowHttpRequestStatusException (Unauthorized,System.Net.Http.StringContent,Content Unauthorized) [< 1ms]
  √ ReadingShouldReturnExpectedContent [50ms]
  √ ReadingWithNullHttpContentShouldThrowArgumentNullException [2ms]
  √ CreatingContentShouldReturnObjectContentWithCorrectFormatter [26ms]
  √ CreatingContentShouldReturnObjectContentWithCorrectValue [< 1ms]

Test Run Successful.
Total tests: 25
     Passed: 25
 Total time: 1.6128 Seconds

about the environment:

• Windows 10 Enterprise (1903)
• Visual Studio Enterprise 16.7.4

some of the libraries that the project are using:

• Microsoft.NET.Test.Sdk 16.7.1
• Microsoft.QualityTools.Testing.Fakes 16.7.4-beta.20330.2
• MSTest.TestAdapter 2.0.0
• MSTest.TestFramework 2.0.0
• coverlet.collector 1.3.0

[MarcoRossignoli]

Hi @mateusdemboski,

/Collect:"Code Coverage" is not the way to run coverlet the switch to run coverlet is --collect:"XPlat Code Coverage" https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/VSTestIntegration.md
The correct repo where fill the issue is https://github.com/microsoft/vstest/issues

@jakubch1 do you have any idea?

[mateusdemboski]

Thank you for the answer @MarcoRossignoli!
The same issue occurred when I change the collect parameter to "XPlat Code Coverage".
I will open an issue on the vstest repo!

[MarcoRossignoli]

Ok let us know if is something related to coverlet...but first glance seems not a coverage issue.

[jakubch1]

@mateusdemboski is it running without issues when code coverage is disabled?

Regarding dynamic code coverage (--collect "Code Coverage") we are building support for .Fakes currently. It should be available in 16.9 release of Visual Studio. You can check demo of this functionality here: https://devblogs.microsoft.com/visualstudio/visual-studio-2019-v16-8-preview-3-1/

Related feedback ticket: https://developercommunity.visualstudio.com/idea/1168657/support-running-code-coverage-for-fakes-tests-agai.html

[mateusdemboski]

Hello @jakubch1, thank you for your answer!

Until now we haven't any problem running vsteste when code coverage is disabled!
So we disabled the code coverage step from our pipeline and we will wait for the VS 16.9 to enable it again.

Tanks for the support of all! 😄

[jakubch1]

@mateusdemboski please try to reference from your test project https://www.nuget.org/packages/Microsoft.CodeCoverage/16.9.0-preview-20201123-03. It contains logic to get code coverage for Fakes .NET Core projects.

If it will not help you can try also updating your pipeline to install https://www.nuget.org/packages/Microsoft.TestPlatform/16.9.0-preview-20201123-03 (by adding Visual Studio test platform installer task).