Vstest: Enabling code coverage leads to errors in child processes

[overeemm]

Description

We have a net47 project with tests that we run with vstest. The test itself starts a child process (via System.Diagnostics.Process), running a netcorapp2.0 dll with dotnet.exe

However, the child process crashes with the following exception:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.CodeCoverage.Shim, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

It seems that the Environment variables that enable profiling are passed to the child process. And the dotnet.exe runtime expects Microsoft.VisualStudio.CodeCoverage.Shim.dll to be found, which is not the case because the child process is not referencing the dll.

Steps to reproduce

Create a net47 test project. Start a netcoreapp2.0 dll with dotnet.exe, using System.Diagnostics.Process.
Then run vstest.console.exe with enabling code coverage.

Expected behavior

I expected the child process not to crash. I might even expected the child process to be profiled (why should I reference a CodeCoverage.Shim, and why can't the runtime load this dll itself).

Actual behavior

The child process crashes, and the test fails.

Diagnostic logs

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.CodeCoverage.Shim, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

Environment

I'm using VS2017 Enterprise 15.4.2 and dotnet.exe is version 2.0.2

[codito]

@overeemm you're right, coverage profiler tries to attach to child processes by default. If that is not desired, may be try explicitly disabling profiler, just setting COR_ENABLE_PROFILING=0 should suffice (in the ProcessStartInfo.EnvironmentVariables member for child process). Environment variables are documented here.

Let's say if coverage for the child process is desired. I think current scenario (profiler for .NET Desktop parent process which forks a .NET Core child process) will not work out of box. The environment variables to enable profiler are different. Also the .net core project needs to provide a reference to CoverageShim since GAC or a common assembly location is not used for coreclr. Coverage for .net core is documented here.

[overeemm]

True, the workaround works. What I find strange, is that if you want the coverage for the child process, you need to somehow inject the Shim dll. I do not see a way in doing that.

[mayankbansal018]

@overeemm, I agree that there doesn't seem any way to inject Shim dll to child process, unless you explicitly take reference to it.

Apart from the solution that codito mentioned you can also use runsettings to enablde codecoverage, & while doing so please set "UseVerifiableInstrumentation" to false, this would eliminate the need for shim dll, & simultaneously giving you coverage from child process.

Please see this on how to use runsettings to enable coverage https://msdn.microsoft.com/en-us/library/jj635153.aspx

[dasMulli]

Is it still expected to be able to set this environment variable using ProcessStartInfo?
I can add and override any other environment variable, but setting COR_ENABLE_PROFILING and CORECLR_ENABLE_PROFILING to "0" does not work. (verified in process explorer)
Is there any known hook that tries to re-set these values for child processes?

[dasMulli]

I'm trying to start a MongoDB server using --storageEngine ephemeralForTest and the process inadvertently always loads covrun64.dll and msdia140.dll from the microsoft.codecoverage NuGet package and then fails to start up fully and accept connections.

[mayankbansal018]

@dasMulli can you check the environment variables set on top of the process where this code is executing, check for COR_PROFILER* variables.

[dasMulli]

I digested it into the following repro which runs with dotnet test but not with dotnet test --collect "Code Coverage":
repro-subproc-profiling.zip

The problem is that i can modify or add any other variables when starting the subprocess, but not the profiling-related ones.

[briandunnington]

Like @dasMulli , it seems that setting the env vars does not result in them being overridden, and thus when --collect is on, our tests do not work due to child process not starting properly.

[mayankbansal018]

@briandunnington can you please file a new bug describing the problem, & a possible repro.

[Jihane-Bettahi]

Are there any updates regarding this issue? The workaround to set the the profiling environment variables does not work, are there any other workarounds we can try?

[briandunnington]

We found a work around that worked for us, so I will share it here in case anyone else finds it useful. The behavior of disabling the collection for child processes can also be set using the RunSettings file and that did work for us. But since we were using this in a share DevOps pipeline, we didn't want to have to have the RunSettings file copied in every project, so I was able to use the 'RunSettings command line arguments' to do the same thing. The syntax is kind of like a modified xpath query and looks like this:

dotnet test --collect "Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.UseVerifiableInstrumentation=False DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.AllowLowIntegrityProcesses=False DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.CollectFromChildProcesses=False'

More info on passing RunSettings via the command line can be found here: https://github.com/Microsoft/vstest-docs/blob/master/docs/RunSettingsArguments.md

[Jihane-Bettahi]

This last workaround worked for me. Thanks @briandunnington.

[yangyadi1993]

When I run the workaround with run settings in Azure pipeline, it gave me not implemented exception.

Unhandled exception. System.NotImplementedException: The method or operation is not implemented.
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.CLIRunSettingsArgumentExecutor.Initialize(String argument)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
   at System.Lazy`1.PublicationOnlyViaFactory(LazyHelper initializer)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List`1& processors)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
   at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)

Anyone knows why? I am running netcore 3.1. And have you encountered this problem? @briandunnington

[jakubch1]

I'm working on changes related to this:
1) When you specify DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.CollectFromChildProcesses=False we make sure to not instrument child processes and this should make code coverage never fails. Currently this doesn't work correctly.
2) I will try also to use DOTNET_ADDITIONAL_DEPS and pass reference to Shim.dll which should lead to having code coverage collected also for dotnet child processes. Will let you know when this will be available

[jakubch1]

@yangyadi1993 could you please provide info what exact parameters you are passing? This looks like something wrong with your command.

[jakubch1]

I've added logic described above for dotnet test case inside this nuget package: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=16.9.0-preview-20201113-07

We can't correctly support all the cases because of this: https://github.com/dotnet/runtime/issues/3368

Settings DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.CodeCoverage.CollectFromChildProcesses=False should work correctly in above nuget package and in VS 16.9