Vstest: The active test run was aborted. Reason: Test host process crashed

[xuzhg]

Description

>

For a defect reproducable from the vstest command line, describe the issue you've observed.

Steps to reproduce

1. git clone https://github.com/xuzhg/WebApi.git
git checkout -b RoutingBaseForNEtCore30 origin/RoutingBaseForNEtCore30

2. dotnet --version
   

3. run the following command (please replace the following Folder):
   dotnet test [YOURLocalClonedFolder]\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\Build.AspNetCore3x\Microsoft.Test.E2E.AspNetCore3x.OData.csproj --logger trx --results-directory C:\BuildAgent\_work\_temp --configuration release --no-build -v diag

4. result:

Expected behavior

Shall pass all test cases running

Actual behavior

Crash with the error message:

    <RunInfos>
      <RunInfo computerName="ODataBuild50" outcome="Error" timestamp="2019-12-02T20:25:22.4166882+00:00">
        <Text>The active test run was aborted. Reason: Test host process crashed : Unhandled exception. Unhandled exception. System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
System.Threading.Tasks.TaskCanceledException: The operation was canceled.&#xD;
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)&#xD;
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.TestDosAttack(String filter) in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 85&#xD;
   at Microsoft.Test.E2E.AspNet.OData.QueryComposition.SecurityTests.&lt;&gt;c__DisplayClass9_0.&lt;&lt;TestDosAttackWithMultipleThreads&gt;b__0&gt;d.MoveNext() in C:\BuildAgent\_work\1\s\test\E2ETest\Microsoft.Test.E2E.AspNet.OData\QueryComposition\SecurityTests.cs:line 95&#xD;
--- End of stack trace from previous location where exception was thrown ---&#xD;
   at System.Threading.Tasks.Task.&lt;&gt;c.&lt;ThrowAsync&gt;b__139_1(Object state)&#xD;
   at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()&#xD;
   at System.Threading.ThreadPoolWorkQueue.Dispatch()&#xD;
</Text>
      </RunInfo>
    </RunInfos>

Environment

Please share additional details about the test environment.

only installed VS2019 Enterprise.

Additional Info

If i run the failure test case individually, it can pass:

as:

[singhsarab]

@xuzhg I am not able to reproduce the issue. I followed the steps you have shared.

[MarioAtTrimble]

We are getting this issue from time to time on our build server, too. It's not consistent and sometimes it works just fine. We didn't see before when we were running core 2.2, but now see when we are running 3.1

[nohwnd]

We were seeing similar issues in VS, there was a problem with cancellation of run that produced a cancellation error and was not caught in the correct place. It resulted in the same error, but probably not the same stack. That error is kind of a catch all for all errors when the other process disconnects in the middle of an operation so it is hard to tell. I would suggest upgrading to 16.5.0 to see if that was fixes.

[adaughterson]

We are using vstest indirectly (via Microsoft.NET.Test.Sdk), and I have upgraded it to 16.5.0, and yet the error persists.

> A total of 1 test files matched the specified pattern.
> The active test run was aborted. Reason: Test host process crashed : Fatal error. Internal CLR error. (0x80131506)
>    at System.Reflection.RuntimeModule.GetTypes(System.Reflection.RuntimeModule)
>    at System.Reflection.RuntimeAssembly.get_DefinedTypes()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetAvailableRunnerReporters(System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass23_0.<GetRunnerReporter>b__0()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ViaFactory(System.Threading.LazyThreadSafetyMode)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].ExecutionAndPublication(System.LazyHelper, Boolean)
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CreateValue()
>    at System.Lazy`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].get_Value()
>    at Xunit.Runner.VisualStudio.VsTestRunner.GetRunnerReporter(LoggerHelper, Xunit.Runner.VisualStudio.RunSettings, System.Collections.Generic.IEnumerable`1<System.String>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.RunTests(Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle, LoggerHelper, Xunit.Runner.VisualStudio.TestPlatformContext, Xunit.Runner.VisualStudio.RunSettings, System.Func`1<System.Collections.Generic.List`1<Xunit.Runner.VisualStudio.AssemblyRunInfo>>)
>    at Xunit.Runner.VisualStudio.VsTestRunner.Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor.RunTests(System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.Utilities.LazyExtension`2<Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.ITestExecutor,Microsoft.VisualStudio.TestPlatform.Common.Interfaces.ITestExecutorCapabilities>, System.Tuple`2<System.Uri,System.String>, Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Adapter.RunContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(System.Collections.Generic.IEnumerable`1<System.Tuple`2<System.Uri,System.String>>, Int64)
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestsInternal()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTests()
>    at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.ExecutionManager.StartTestRun(System.Collections.Generic.Dictionary`2<System.String,System.Collections.Generic.IEnumerable`1<System.String>>, System.String, System.String, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ClientProtocol.TestExecutionContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Engine.ITestCaseEventsHandler, Microsoft.VisualStudio.TestPlatform.ObjectModel.Client.ITestRunEventsHandler)
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c__DisplayClass30_3.<OnMessageReceived>b__3()
>    at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TestRequestHandler+<>c.<.ctor>b__18_1(System.Action)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SafeProcessJob(System.__Canon)
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].BackgroundJobProcessor()
>    at Microsoft.VisualStudio.TestPlatform.Utilities.JobQueue`1[[System.__Canon, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__12_0()
>    at System.Threading.Tasks.Task.InnerInvoke()
>    at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
>    at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
>    at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
>    at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
>    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
>    at System.Threading.ThreadHelper.ThreadStart(System.Object)

Are there any other recommendations?

[gregpakes]

Our builds have started failing with this error. Is there a way to get some decent logs out? We are using azure hosted pipelines.

We have been getting this error for months but it's been very intermittent, so hasn't bothered us. But over the past 2 days it has spiked hugely.

[amoerie]

I'm also seeing persistent test host crashes in a project that I contribute to.

Links:

• https://github.com/fo-dicom/fo-dicom/runs/861061964
• https://github.com/fo-dicom/fo-dicom/runs/857136803
• https://github.com/fo-dicom/fo-dicom/runs/857086214 (this was an internal CLR error)

[ChaosEngine]

In follow up to this I've created https://github.com/microsoft/vstest/issues/2501

[jesperoastrom]

We had the exact same issue with our build. It turned out to be misconfigured settings for code coverage. In our CodeCoverage.runsettings-file we were not properly excluding test projects from coverage.

[mcallaghan-bsm]

We have (possibly) the same intermittent issue here.

rough important list of packages in our UT test project:

        <PackageReference Include="Microsoft.NETCore.Platforms" Version="3.1.0" />
        <PackageReference Include="coverlet.collector" Version="1.3.0">
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1" />

Our default gitlab runner job is on a kubernetes runner, and defaults to docker image: "mcr.microsoft.com/dotnet/core/sdk:3.1-alpine"

Adding diag to a parallel job to see if we can repro with more diagnostics.

[plachta11b]

I have got here due to use of wrong function.
Assert.ThrowsException<MyException>(async () => await myFunction())
Fixed by replacing to async version:
await Assert.ThrowsExceptionAsync<MyException>(async () => await myFunction())

[chaoticsoftware]

Anyone has any update for this issue or any idea how to solve it?

[dbtfsbre]

I am seeing the same error message both with vstest.console.exe on Visual Studio 2019 16.8.4 and with dotnet test.
My test projects are using MSTest V2 v2.1.0. On the Windows 10 machine (OS build 17763.1697) I have installed .NET framework 4.7.2.

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag
Microsoft (R) Test Execution Command Line Tool Version 16.8.3
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed : Process is terminated due to StackOverflowException.


Test Run Aborted.

With dotnet test I get this error in the Application event log:

_Faulting application name: testhost.net472.x86.exe, version: 15.0.0.0, time stamp: 0xd89b8ef9
Faulting module name: KERNELBASE.dll, version: 10.0.17763.1697, time stamp: 0x672c12bb
Exception code: 0xc00000fd
Fault offset: 0x00108f0d
Faulting process id: 0x1b38
Faulting application start time: 0x01d6ea7f68419dc1
Faulting application path: C:\Program Files\dotnet\sdk\5.0.102\TestHost\testhost.net472.x86.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 25d2099b-1d8a-4847-a90d-680e65d12f6e
Faulting package full name:
Faulting package-relative application ID:_

With vstest.console.exe this error is logged:

_Faulting application name: testhost.net472.exe, version: 15.0.0.0, time stamp: 0x89f6cab7
Faulting module name: clr.dll, version: 4.7.3740.0, time stamp: 0x5f7e5a1b
Exception code: 0xc00000fd
Fault offset: 0x000000000005c71b
Faulting process id: 0x1a70
Faulting application start time: 0x01d6ea818c7cd9ec
Faulting application path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\testhost.net472.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Report Id: 62f466eb-cece-41ac-a934-cf21ea4080c8
Faulting package full name:
Faulting package-relative application ID:_

A _TestResults_ directory was created next to the test assembly, but it is completely empty.

Any idea how I can find the cause of the error?

[nohwnd]

@dbtfsbre you can install 5.0.101 dotnet SDK, and also install ProcDump so it is available in your PATH (if you open command line and type procdump, you should see output of the procdump tool). Then you do just:

dotnet test MyMsTestProject.dll --filter OneSpecificTest -v diag --blame-crash

And it should create a dump for you, open that in VS and you should see where that exception is coming from. Most likely it is your code calling some method recursively and forgetting to terminate.

[dbtfsbre]

@nohwnd Thanks. Something went wrong when copying files to the test machine for an investigation. The test is running fine in Visual Studio using Test Explorer and during our regular Release test runs. After compiling the test again directly on the test machine, I was able to run it with dotnet test as well as vstest.console.exe.

[mcallaghan-geotab]

FWIW: the solution we found was actually due to OOM kills from kubernetes gitlab-runner setup. The nodes were stretching their RAM too tightly, and eventually digging showed events where k8s was killing the dotnet process.

Looked like this:

Memory cgroup out of memory: Killed process 3010546 (dotnet) total-vm:5186748kB, anon-rss:378116kB, file-rss:67156kB, shmem-rss:0kB, UID:0 pgtables:1336kB oom_score_adj:936

This might not be the problem or solution for everyone, but wanted to share back as it is definitely something to consider!