Kestrelhttpserver: An existing connection was forcibly closed by the remote host

[halter73]

_From @sepehr1014 on May 8, 2018 10:25_

Using ASP.NET Core 2.1 RC1 the process is killed after an hour or so and we see these logs:

Application: dotnet.exe
CoreCLR Version: 4.6.26426.2
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AggregateException: One or more errors occurred. (An existing connection was forcibly closed by the remote host) ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
   at System.Net.Sockets.Socket.Shutdown(SocketShutdown how)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Shutdown()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Abort()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.Abort(Exception error)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Abort(Exception error)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.Abort(Exception ex)
   at System.IO.Pipelines.PipeCompletionCallbacks.Execute()
   --- End of inner exception stack trace ---
   at System.IO.Pipelines.PipeCompletionCallbacks.Execute()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Faulting application name: dotnet.exe, version: 2.1.26426.3, time stamp: 0x5ae1770d
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2189, time stamp: 0x5abdad60
Exception code: 0xe0434352
Fault offset: 0x0000000000033c58
Faulting process id: 0x2e90
Faulting application start time: 0x01d3e6aa48e5f5c6
Faulting application path: C:\Program Files\dotnet\dotnet.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: f10816d3-b0de-49bc-9745-60f9b7dea199
Faulting package full name: 
Faulting package-relative application ID: 

BTW we're using web sockets without SignalR.

_Copied from original issue: aspnet/Home#3125_

[halter73]

_From @davidfowl on May 8, 2018 10:34_

Looks like this: https://github.com/aspnet/KestrelHttpServer/issues/2536

[halter73]

_From @davidfowl on May 8, 2018 10:34_

/cc @halter73

[halter73]

_From @sepehr1014 on May 8, 2018 12:38_

We're having this issue on production and relying on memory cache, it's quite frustrating to have the process killed. Is there any workaround we could use until this is fixed?

[halter73]

_From @Ikcelaks on May 8, 2018 14:1_

I am getting the EXACT same Unhandled Exception trace on my development machine using Core 2.1 rc1 and a freshly created MVC project. The Exception can be triggered reliably within a minute by rapidly clicking between "Home", "About", and "Contacts" from multiple browsers (Multiple browsers aren't required, but I find that it makes the problem happen quicker).

This development machine formerly ran 2.1 preview2 (without Kestrel crashes).

[Ikcelaks]

I'm moving this comment, which I submitted just after the issue was moved:

Troubleshooting Update:
I attempted to recreate this bug with a fresh install of DotNet-2.1-RC1 on a fresh Windows 1709 VM, and I was unable to do so. All of my projects ran just fine without crashing.

I returned to my original dev machine (also Windows 1709) and attempted to recreate a fresh environment by uninstalling all versions of DotNet Core and manually removing the NuGetFallbackFolder before reinstalling RC1. This did not resolve the issue.

Then I realized that the fresh VM was on a network that bypasses Web-Filtering. I put my original dev machine on that same network, and suddenly I was no longer able to trigger the Uncaught Exception error. Fortunately/Unfortunately, returning the device to the original filtered network did not cause the bug to return. I'm also unable to correlate the Uncaught Exceptions with any events logged by the Web Filter, but that doesn't mean much, because the kinds of rules that telemetry reporting would trigger don't show up in the log.

Question:
Is there any chance that the development WebHost is attempting to communicate out to the internet and crashing when the communication is cut off (possibly mid-stream) by a web-filter?

[neilcawse]

We are having the same issue after upgrading to 2.1 RC1.
The issue can be reliably recreated (if manually at this stage) by aborting the client socket in chrome when making a request to kestrel. With the chrome dev tools I turned the network speed down to slow 3g and while chrome was loading pages, I hit the button to stop requests. As soon as this happens, Kestrel has an unhandled exception with call stack below. What is bad is there looks to be no way to handle the exception.

This looks to be a critical showstopper bug!

System.AggregateException: One or more errors occurred. (An existing connection was forcibly closed by the remote host) ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName) at System.Net.Sockets.Socket.Shutdown(SocketShutdown how) at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Shutdown() at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Abort() at Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.TransportConnection.Microsoft.AspNetCore.Connections.Features.IConnectionLifetimeFeature.Abort() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.Abort(Exception error) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Abort(Exception error) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.Abort(Exception ex) at System.IO.Pipelines.PipeCompletionCallbacks.Execute() --- End of inner exception stack trace --- at System.IO.Pipelines.PipeCompletionCallbacks.Execute() at System.IO.Pipelines.Pipe.<>c.<.cctor>b__79_2(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location where exception was thrown --- at System.Threading.ThreadPoolWorkQueue.Dispatch() ---> (Inner Exception #0) System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName) at System.Net.Sockets.Socket.Shutdown(SocketShutdown how) at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Shutdown() at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.Abort() at Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.TransportConnection.Microsoft.AspNetCore.Connections.Features.IConnectionLifetimeFeature.Abort() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.Abort(Exception error) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.Abort(Exception error) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.Abort(Exception ex) at System.IO.Pipelines.PipeCompletionCallbacks.Execute()<---

[halter73]

@neilcawse We're looking into it. In the meantime you can still use the old libuv transport in 2.1. This doc-in-progress shows which package to add. From there, you can just call IWebHostBuilder.UseLibuv() extension method.

[sonicmouse]

Just a heads up that I am still seeing this in 2.1.0-rc1-final

[halter73]

@sonicmouse Unfortunately, 2.1.0-rc1-final is the first (and last) release with this issue. It will be fixed in rtm. You can work around the issue by selecting the libuv transport in the meantime.

[John0King]

I facing this problem today.

I'm using asp.net core 2.1.0 on .net461.
here's the error log

2018-06-27 15:22:59.1865|13|ERROR|Microsoft.AspNetCore.Server.Kestrel|Connection id "0HLERVO6PKH3B", Request id "0HLERVO6PKH3B:00000001": An unhandled exception was thrown by the application. System.IO.IOException: 远程主机强迫关闭了一个现有的连接。 ---> System.Net.Sockets.SocketException: 远程主机强迫关闭了一个现有的连接。
   在 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitable.GetResult()
   在 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.<ProcessSends>d__25.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.<DoSend>d__24.MoveNext()
   --- 内部异常堆栈跟踪的结尾 ---
   在 System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   在 System.IO.Pipelines.Pipe.GetFlushResult(FlushResult& result)
   在 System.IO.Pipelines.Pipe.FlushAsync(CancellationToken cancellationToken)
   在 System.IO.Pipelines.Pipe.DefaultPipeWriter.FlushAsync(CancellationToken cancellationToken)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.WriteAsync(ReadOnlySpan`1 buffer, CancellationToken cancellationToken)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.FlushAsync(CancellationToken cancellationToken)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.WriteAsync[T](Func`3 callback, T state)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteChunkedAsync(ReadOnlyMemory`1 data, CancellationToken cancellationToken)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.WriteAsync(ReadOnlyMemory`1 data, CancellationToken cancellationToken)
   在 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   在 System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count)
   在 Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter.<FlushInternalAsync>d__27.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.<ExecuteAsync>d__22.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.<ExecuteAsync>d__21.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.ViewFeatures.ViewResultExecutor.<ExecuteAsync>d__6.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.ViewResult.<ExecuteResultAsync>d__26.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultAsync>d__20.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResultFilterAsync>d__28`2.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultFilters>d__26.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__23.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__18.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__16.MoveNext()
--- 引发异常的上一位置中堆栈跟踪的末尾 ---
   在 System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   在 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   在 Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()

Does asp.net core 2.1 require the new ANCM to avoid this ?

[Tratcher]

Comments on closed issues are not tracked, please open a new issue with the details for your scenario.