Vstest: System.MissingMethodException Method not found: 'Void Microsoft.VisualStudio.CodeCoverage.Shim.1017596829__coverage_runtime_method__32412_19337__(UInt64)'.

[lhfiedler]

Description

When using the /Enablecodecoverage option on vstest.console.exe, we occasionally see a MissingMethodException about the injected Shim method for the code coverage. This appears during calls than span across a new AppDomain.

i.e.
System.MissingMethodException
Method not found: 'Void Microsoft.VisualStudio.CodeCoverage.Shim.2218451032__coverage_runtime_method__56282_18307__(UInt64)'.

Please this this related post we created about 4 years ago. This post is slightly different because it's dealing with IIS, but might be related due to dealing with AppDomains.
https://social.msdn.microsoft.com/Forums/vstudio/en-US/3f7415ed-7e17-496b-b705-3055c48a4406/iis-codecoverage-occasional-method-not-found-microsoftvisualstudiocodecoverageshim?forum=vsunittest

Steps to reproduce

Develop a unit test that makes a call across an app domain. You'll occasionally see this exception (not always).

Expected behavior

There should be no exception

Actual behavior

We see the MissingMethodException

Diagnostic logs

Example failed test stack trace:

Failed SaveTwitterSearchData

Test method UnitTests.SocialTests.SaveTwitterSearchData threw exception:
System.ServiceModel.FaultException`1[CompAnalytics.Contracts.Fault.ExecutionFault]: Exception has been thrown by the target of an invocation.
Stack Trace:

Server stack trace:
System.Reflection.TargetInvocationException
Exception has been thrown by the target of an invocation.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.Activator.CreateInstanceFromInternal(String assemblyFile, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo)
at System.AppDomain.InternalCreateInstanceFromWithNoSecurity(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
at System.AppDomain.InternalCreateInstanceFromWithNoSecurity(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityAttributes)
at System.Activator.CreateInstanceFrom(AppDomain domain, String assemblyFile, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at CompAnalytics.Execution.Isolation.SandBox..ctor(ExecutionContext context) in C:\agent_work\1\s\Product\CompAnalytics.Execution\Isolation\SandBox.cs:line 91
at CompAnalytics.Execution.Isolation.SandBox.Create(ExecutionContext context) in C:\agent_work\1\s\Product\CompAnalytics.Execution\Isolation\SandBox.cs:line 52
at CompAnalytics.Execution.Isolation.SandBoxExtension.Create() in C:\agent_work\1\s\Product\CompAnalytics.Execution\SandBoxExtension.cs:line 15
at CompAnalytics.Execution.CodeModuleExecutor.Execute(IExecutionContext context) in C:\agent_work\1\s\Product\CompAnalytics.ExecutionCodeModuleExector.cs:line 76
at CompAnalytics.Execution.ExecutionContext.ExecuteModuleWithRetry(ModuleExecutor executor, Module module) in C:\agent_work\1\s\Product\CompAnalytics.Execution\ExecutionContext.cs:line 1066
at CompAnalytics.Execution.ExecutionContext.ExecuteModule(ModuleExecutor executor, Module module, ExecutionCallback postExecutionCallback) in C:\agent_work\1\s\Product\CompAnalytics.Execution\ExecutionContext.cs:line 900
System.MissingMethodException
Method not found: 'Void Microsoft.VisualStudio.CodeCoverage.Shim.2218451032__coverage_runtime_method__56282_18307__(UInt64)'.
at CompAnalytics.Execution.Isolation.RemoteSandBox..ctor(IsolatedTraceRedirector traceRedirector)
at CompAnalytics.IServices.ClientMessageInspector.AfterReceiveReply(Message& reply, Object correlationState) in C:\agent_work\1\s\Product\CompAnalytics.IServices\ErrorHandler.cs:line 161
at System.ServiceModel.Dispatcher.ImmutableClientRuntime.AfterReceiveReply(ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

Environment

Windows Server 2012 R2
Test Execution Command Line Tool Version 14.0.25463.0

[johnwhoffman]

We are seeing this out of the blue starting today. Nothing has changed in the code or the build environments.

[ShahViral6438]

We are also getting the same error for our gated builds and it started out of nowhere.

VSTS test task fails with below error for 50% of unit test cases:
Error: System.MissingMethodException: Method not found: 'Void Microsoft.VisualStudio.CodeCoverage.Shim.4251430275__coverage_runtime_method__28318_17131__(UInt64)'

Any update is highly appreciated.

[jons-aura]

Also having nearly half our builds on Azure Devops fail with this issue.

[nohwnd]

We did a release of 16.7.1 yesterday, are your builds picking up that version of TestPlatform installer task?

[jakubch1]

This is same issue as https://github.com/microsoft/azure-pipelines-tasks/issues/12808
You can find workaround there.
@ShreyasRmsft what is the issue here?

[ovirta]

We are still experiencing build failures during test phase running code coverage in Azure devops as part of the PR pipeline. The workaround suggests providing path to empty directory for adapters and we have tried adding -a ./path/to/empty/dir but that didn't help.

Roughly 50% of our builds are now failing.

This is excerpt from one of the failures.

2020-10-29T07:55:46.0883405Z [command]"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\Tests\IntegrationTests\IntegrationTests.csproj --logger trx --results-directory D:\a\_temp --no-restore --configuration Release "--collect:Code Coverage" -s CodeCoverage.runsettings
2020-10-29T07:55:54.1023258Z Test run for D:\a\1\s\Tests\IntegrationTests\bin\Release\netcoreapp3.1\IntegrationTests.dll(.NETCoreApp,Version=v3.1)
2020-10-29T07:55:54.2460466Z Microsoft (R) Test Execution Command Line Tool Version 16.7.0
2020-10-29T07:55:54.2461279Z Copyright (c) Microsoft Corporation.  All rights reserved.
2020-10-29T07:55:54.2461620Z 
2020-10-29T07:55:54.4303001Z Starting test execution, please wait...
2020-10-29T07:55:55.3598112Z 
2020-10-29T07:55:55.3599439Z A total of 1 test files matched the specified pattern.
2020-10-29T07:55:57.8844196Z An exception occurred while invoking executor 'executor://xunit/VsTestRunner2/netcoreapp': Method not found: 'Void Microsoft.VisualStudio.CodeCoverage.Shim.2481202972__coverage_runtime_method__41931_20370__(UInt64)'.
2020-10-29T07:55:57.9940605Z M�i�c�r�o�s�o�f�t� �(�R�)� �C�o�v�e�r�a�g�e� �C�o�l�l�e�c�t�i�o�n� �T�o�o�l� �V�e�r�s�i�o�n� �1�6�.�0�.�3�0�3�1�9�.�2�0�0�
2020-10-29T07:55:57.9941354Z �
2020-10-29T07:55:57.9941560Z �
2020-10-29T07:55:57.9942237Z �C�o�p�y�r�i�g�h�t� �(�c�)� �M�i�c�r�o�s�o�f�t� �C�o�r�p�o�r�a�t�i�o�n�.� � �A�l�l� �r�i�g�h�t�s� �r�e�s�e�r�v�e�d�.�
2020-10-29T07:55:57.9942739Z �
2020-10-29T07:55:57.9942932Z �
2020-10-29T07:55:57.9943371Z �
2020-10-29T07:55:57.9943647Z �
2020-10-29T07:55:58.1713110Z �Results File: D:\a\_temp\VssAdministrator_WIN-I03V8NT17AF_2020-10-29_07_55_58.trx
2020-10-29T07:55:58.1713746Z 
2020-10-29T07:55:58.1714225Z Attachments:
2020-10-29T07:55:58.1714878Z   D:\a\_temp\89168bd3-ad1a-4562-bbdd-1da12060b5ef\VssAdministrator_WIN-I03V8NT17AF_2020-10-29.07_55_55.coverage
2020-10-29T07:55:58.1731074Z Test Run Failed.
2020-10-29T07:55:58.1994938Z C:\Program Files\dotnet\sdk\3.1.403\Microsoft.TestPlatform.targets(32,5): error MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error. [D:\a\1\s\Tests\IntegrationTests\IntegrationTests.csproj]
2020-10-29T07:55:58.3311826Z ##[error]Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

Start of the log file with version:

2020-10-29T07:54:24.3728850Z ##[section]Starting: dotnet test Release 2020-10-29T07:54:24.3911353Z ============================================================================== 2020-10-29T07:54:24.3911686Z Task : .NET Core 2020-10-29T07:54:24.3912011Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command 2020-10-29T07:54:24.3912343Z Version : 2.175.0 2020-10-29T07:54:24.3912567Z Author : Microsoft Corporation 2020-10-29T07:54:24.3912901Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli 2020-10-29T07:54:24.3913274Z ============================================================================== 2020-10-29T07:54:25.5543381Z [command]C:\windows\system32\chcp.com 65001 2020-10-29T07:54:25.5717495Z Active code page: 65001 2020-10-29T07:54:25.6071009Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.

Any updates or help is appreciated.

[ovirta]

FYI

After downgrading packages in our test projects issue has not reproduced:

    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="3.1.9" />   ->  "3.1.4"
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />            ->  "16.6.1"
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">            ->  "2.4.1"

[jons-aura]

This is still an active issue for my company something like half of our builds are failing with this error and having to go in and manually re-run them is a lot of overhead.

[spaasis]

I also worked around this by downgrading Microsoft.NET.Test.Sdk from 16.8.3 to 16.6.1

[jakubch1]

There is 1 more way of fixing the issue in case of .NET Core, .NET Standard and .NET 5: you can start using unverifiable probes. Unverifiable probes don't require Microsoft.VisualStudio.CodeCoverage.Shim dependency.

To enable unverifiable probes please set in your runsettings:

<?xml version="1.0" encoding="utf-8"?>
<!-- File name extension must be .runsettings -->
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
......
            <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
......
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

If you don't use runsettings you can add -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.UseVerifiableInstrumentation="False" suffix to your command. For example:

dotnet test --collect "Code Coverage" --diag "log.txt" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.UseVerifiableInstrumentation="False"

We will make unverifiable probes default for all .NET Core and .NET 5 projects in VS 16.10.

[spaasis]

@jakubch1 Seemingly after your proposed workaround, our pipelines no longer collect any coverage information:
"Coverage for changed lines cannot be determined. Coverage data not found."

The tests are run as before, and the output of the test process implies that the results are published

Our CodeCoverage.runsettings:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <!-- Match the path of the source files in which each method is defined: -->
            <Functions>
              <Exclude>
                <!-- Removes Migrations namespace from code coverage-->
                <Function>^Persistence.Migrations\..*</Function> 
              </Exclude>  
            </Functions>
            <Sources>
              <Exclude>
                <!-- Removes translation files from code coverage -->
                <Source>.*\\Translations\\.*</Source>
                <Source>.*Designer.cs</Source>
              </Exclude>
            </Sources>
            <!-- We recommend you do not change the following values: -->

            <!-- Set this to True to collect coverage information for functions marked with the "SecuritySafeCritical" attribute. Instead of writing directly into a memory location from such functions, code coverage inserts a probe that redirects to another function, which in turns writes into memory. -->
            <!-- Disabled due to https://github.com/microsoft/vstest/issues/2459 -->
            <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
            <!-- When set to True, collects coverage information from child processes that are launched with low-level ACLs, for example, UWP apps. -->
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <!-- When set to True, collects coverage information from child processes that are launched by test or production code. -->
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <!-- When set to True, restarts the IIS process and collects coverage information from it. -->
            <CollectAspDotNet>False</CollectAspDotNet>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

And the pipeline:

# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4

pool:
  vmImage: 'windows-2019'

variables:
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

...

- task: DotNetCoreCLI@2
  displayName: 'dotnet test $(buildConfiguration)'
  inputs:
    command: test
    arguments: '--no-restore --configuration $(BuildConfiguration) --collect:"Code Coverage" -s "CodeCoverage.runsettings"'
    projects: 'Tests/**/*.csproj'

[jakubch1]

@spaasis could you please post also output from dotnet test command? Could you please run it locally and see if .coverage file is generated?

[spaasis]

@jakubch1 sure, here's the raw output from the pipeline (it's a bit verbose..): https://pastebin.com/bijKZ5d8

Running dotnet test --no-restore --configuration Release --collect:"Code Coverage" -s "CodeCoverage.runsettings" does generate a code coverage result in e.g. /Tests/IntegrationTests/TestResults/

[jakubch1]

@spaasis in both places code coverage report was generated:

2021-02-03T10:26:09.7518355Z Attachments:
2021-02-03T10:26:09.7518874Z   D:\a\_temp\8e991836-327b-44b2-82ec-991573fa95ac\VssAdministrator_fv-az350-317_2021-02-03.10_26_04.coverage

could you please send me those .coverage files for investigation? [You can add it in pipeline to artifacts if it's not accessible.]

[spaasis]

@jakubch1 Here you go. This is from the Azure Pipeline. Seems to open correctly in VS and have proper data.
20210204.1.Release.Any CPU.9239.zip

Could the issue be then in the Pipeline reporter?

Our azurepipelines-coverage.yml:

coverage:
  status:           #Code coverage status will be posted to pull requests based on targets defined below.
    comments: on    #Off by default. When on, details about coverage for each file changed will be posted as a pull request comment. 

EDIT:
here's the comment from the pipeline:

[jakubch1]

@ganesp @ShreyasRmsft could you please help on this? I've checked that coverage file is generated and it contains data for example:

        <source_file id="5" path="D:\a\1\s\Core\Application\Cases\Commands\CreateOrUpdateCaseCommand.cs" />

[spaasis]

Huh.. All of a sudden one pipeline reported the coverage successfully with no changes from our side.
UPDATE: that seems to have been a fluke. Majority of the coverage checks still does not work

[ShreyasRmsft]

@ganesp to help out here.

[spaasis]

For a couple days our code coverage reports have worked as expected

[acn-sbuad]

We are still experiencing this sporadically for our DevOps pipeline running .NET 5.
Log from the pipeline

[jakubch1]

@acn-sbuad what version of TestPlatform do you use? Did you put

<UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>

to your runsettings?

[lhfiedler]

Why would the verifiable instrumentation work sometimes and not work other times? In all the scenarios that folks have listed, it's not like it never works or always works. It's an occasional failure of the missing method exception.

[hhoangnl]

FYI

After downgrading packages in our test projects issue has not reproduced:

    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="3.1.9" />   ->  "3.1.4"
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.7.1" />            ->  "16.6.1"
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">            ->  "2.4.1"

In my case I only needed to downgrade Microsoft.NET.Test.Sdk (from 16.8.3) to 16.6.1

[qualitas-software]

I had this error pop up for 1 of my 5 solutions being built, but only some of the time.

(_Using a DevOps self hosted docker container agents; VS 2019 with Microsoft.VisualStudio.Component.TestTools.BuildTools included_)

Upgrading nugets related to testing helped me.

Eg:

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.8.3" />  --was 16.5.0 in this solution
<PackageReference Include="xunit" Version="2.4.1" /> --was 2.4.0
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3"> ... </PackageReference>  --was 2.4.0

I also added missing .csproj properties (to match our other solution's projects):

<Platforms>AnyCPU</Platforms>
<Prefer32Bit>false</Prefer32Bit>

[jakubch1]

Closing this issue. In latest VS and stable Microsoft.CodeCoverage packages we use non verifiable instrumentation.