Aspnetcore: Make model binding robust against client disconnects

Created on 22 Aug 2019 · 27Comments · Source: dotnet/aspnetcore

Describe the bug

I'm receiving what appears to be random exceptions on my web app. It's running ASP.NET Core 2.2.108 on Azure App Service (Windows). I've only seen it happening for a single endpoint which is a POST request with a file upload. The files are restricted to 5MB.

I'm struggling to reproduce the issue but was hoping someone might be able to make more sense of the stack trace than I can and point me in the right direction.

The outer type and message reported by App Insights is Microsoft.AspNetCore.Connections.ConnectionResetException and The client has disconnected. Does this actually mean that the client device has lost it's internet connection (could be a mobile device)?

The inner type and message are System.Runtime.InteropServices.COMException and An operation was attempted on a nonexistent network connection. (Exception from HRESULT: 0x800704CD)

My searching has led me to this issue but not sure if it's related in any way, https://github.com/aspnet/AspNetCore/issues/6415 the only place I can find the message The client has disconnected in the repo is here : https://github.com/aspnet/AspNetCore/blob/c60d1e1a14691a624f46b350f6561dc752e3964c/src/Servers/IIS/IIS/src/Core/IO/AsyncIOOperation.cs

Happy to provide more information if it is a genuine bug rather than something I'm doing wrong, just let me know.

Stack trace from App Insights is below:

Microsoft.AspNetCore.Connections.ConnectionResetException:
   at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult (Microsoft.AspNetCore.Server.IIS, Version=2.2.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<ReadBody>d__312.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe+DefaultPipeReader.GetResult (System.IO.Pipelines, Version=4.0.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<ReadAsync>d__309.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream+<ReadAsyncInternal>d__10.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=2.2.6.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.BufferedReadStream+<EnsureBufferedAsync>d__37.MoveNext (Microsoft.AspNetCore.WebUtilities, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream+<ReadAsync>d__36.MoveNext (Microsoft.AspNetCore.WebUtilities, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream+<ReadAsync>d__35.MoveNext (Microsoft.AspNetCore.WebUtilities, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions+<DrainAsync>d__3.MoveNext (Microsoft.AspNetCore.WebUtilities, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Http.Features.FormFeature+<InnerReadFormAsync>d__18.MoveNext (Microsoft.AspNetCore.Http, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenStore+<GetRequestTokensAsync>d__3.MoveNext (Microsoft.AspNetCore.Antiforgery, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery+<ValidateRequestAsync>d__9.MoveNext (Microsoft.AspNetCore.Antiforgery, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ValidateAntiforgeryTokenAuthorizationFilter+<OnAuthorizationAsync>d__3.MoveNext (Microsoft.AspNetCore.Mvc.ViewFeatures, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeFilterPipelineAsync>d__18.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker+<InvokeAsync>d__16.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=2.2.5.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Builder.RouterMiddleware+<Invoke>d__4.MoveNext (Microsoft.AspNetCore.Routing, Version=2.2.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Phinite.AIM.Common.Middleware.CacheAutomationMiddleware+<Invoke>d__2.MoveNext (Phinite.AIM.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullPhinite.AIM.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: d:\a\1\s\Phinite.AIM\Phinite.AIM.Common\Middleware\CacheAutomationMiddleware.csPhinite.AIM.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 21)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Phinite.AIM.Core.Multitenancy.TenantResolverMiddleware+<Invoke>d__3.MoveNext (Phinite.AIM.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullPhinite.AIM.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: d:\a\1\s\Phinite.AIM\Phinite.AIM.Core\Multitenancy\TenantResolverMiddleware.csPhinite.AIM.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware+<Invoke>d__7.MoveNext (Microsoft.AspNetCore.StaticFiles, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
Inner exception System.Runtime.InteropServices.COMException handled at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult:

affected-most area-mvc enhancement severity-major

Source

stap123

Most helpful comment

There are proposals for some components that read the request body like MVC model binding to catch these exceptions and short circuit with a 400 response. That should help limit some if the log noise.

Tratcher on 11 Jun 2020

👍3

All 27 comments

Happy to provide more information if it is a genuine bug rather than something I'm doing wrong, just let me know.

This is expected behavior when a client disconnects in the middle of uploading a request body.

We might want to have MVC swallow these exceptions so we don't get scary logs from the server when this happens. @Tratcher

halter73 on 28 Aug 2019

@rynowak thinks we should change model binding such that request body read errors are converted to model state errors. This includes client disconnects and form parsing errors.

Note this one is a little different, it happens in AntiForgery but we should still make sure it gets handled at that layer.

Tratcher on 30 Aug 2019

Agreed, this kind of issue should not result in a 500.

@halter73 @Tratcher - is there a clear way to detect an IO failure due to connection loss? There's going to be a try/catch somewhere in our code and I'm wondering what we're going to be catching.

rynowak on 3 Sep 2019

They should be IOExceptions 😁, or at least derived from IOExceptions.

With the form reader in the middle it might be a bit trickier, it may wrap the IOExceptions.

Tratcher on 3 Sep 2019

@Tratcher is right. Technically it's a Microsoft.AspNetCore.Connections.ConnectionResetException, but I'd catch any IOException.

halter73 on 3 Sep 2019

I also run into this problem , .netcore 3.1 running on IIS, is there a way I can catch this ConnectionReset exceptions in current version, or I have to wait for the 5.0.0

liuliang-wt on 17 Jan 2020

@luisquintanilla If you want to catch this, you can catch IOException because ConnectionResetException is an IOException. That said, ConnectionResetException has always been public. What are you waiting for in 5.0?

halter73 on 18 Jan 2020

@halter73 OK, I saw you add it to milestones 5.0.0, I thought I have to wait until then ....

liuliang-wt on 18 Jan 2020

@liuliang-wt the plan for 5.0 is to have MVC catch these errors from the form reader so they don't show up in the logs anymore.

Tratcher on 18 Jan 2020

I think this isn't just a co-incident but I'm seeing these errors when someone tries to log in to my website (using Edge). There will be about 20 of these exceptions all within the same second, before the login is successful (the login does take a while, but is finally successful). It's not easy to reproduce, the site must be running for quite a while before this happens.

Here's my stacktrace (from the EventLog):

Microsoft.AspNetCore.Connections.ConnectionResetException: The client has disconnected
 ---> System.Runtime.InteropServices.COMException (0x80070040): The specified network name is no longer available. (0x80070040)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Pipelines.StreamPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenStore.GetRequestTokensAsync(HttpContext httpContext)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.ValidateRequestAsync(HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.ValidateAntiforgeryTokenAuthorizationFilter.OnAuthorizationAsync(AuthorizationFilterContext context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

It may be a issue with Edge though, so I'll try to get them to use another browser and see if these still occur.

mlsomers on 19 Apr 2020

👍1

What kind of auth? For something like windows auth that would almost make sense, its connection based so connections with the wrong credentials would need to be closed by the client.

Tratcher on 20 Apr 2020

Not sure but I guess this is called Forms Authentication? My Configure just has app.UseAuthentication();. It's supposed to send username and password in a Post request to Identity/Account/Login which sets a cookie.

Maybe I have some setting that would have the client attempt windows authentication? What should I be looking for? And why would it not be happening consistently (it never happens in development, only after some time in production, and not always).

mlsomers on 20 Apr 2020

Nevermind, Forms/Identity are unrelated to windows auth.

Tratcher on 20 Apr 2020

I am getting these errors too in my logs. Is there a workaround for this?

rocklan on 18 May 2020

Just saw this around midnight PST. ASP.NET Core 3.1 on Azure App Service (Windows instances, Linux containers haven't exhibited this - yet?). Procedurally, this is happening during a custom ASP.NET machine-machine authentication and our last record shows this happening during a sudden 40x burst of requests. Everything settled down within 6-7 minutes.

Here is one of the exception:

Microsoft.AspNetCore.Connections.ConnectionResetException:
   at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult (Microsoft.AspNetCore.Server.IIS, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<ReadBody>d__337.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException (System.IO.Pipelines, Version=4.0.2.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadResult (System.IO.Pipelines, Version=4.0.2.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult (System.IO.Pipelines, Version=4.0.2.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51)
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext+<ReadAsync>d__334.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream+<ReadAsyncInternal>d__11.MoveNext (Microsoft.AspNetCore.Server.IIS, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream+<ReadAsync>d__36.MoveNext (Microsoft.AspNetCore.WebUtilities, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.IO.Stream+<CopyToAsyncInternal>d__30.MoveNext (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   <snip>
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   <snip>
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeFilterPipelineAsync>g__Awaited|19_0>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker+<<InvokeAsync>g__Logged|17_1>d.MoveNext (Microsoft.AspNetCore.Mvc.Core, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware+<<Invoke>g__AwaitRequestTask|6_0>d.MoveNext (Microsoft.AspNetCore.Routing, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware+<Invoke>d__5.MoveNext (Microsoft.AspNetCore.Authorization.Policy, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=3.1.3.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   <snip>
Inner exception System.Runtime.InteropServices.COMException handled at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult:

SidShetye on 19 May 2020

In my case these errors were related to deadlocks.
During login (amongst other places) I create or update a "last seen" record. After disallowing page locks on that table these issues went away. We would only notice this during login since the other failing ajax calls did not notify the user of any failures.

Unfortunately this means that the deadlock-victim-exception was eaten up somewhere in the process and I ended up only getting this generic COMException (0x80070040): The specified network name is no longer available. That should be fixed!
If there is an InnerException on the COMException, it should be logged as well. (I don't know for sure because I did not reproduce this while debugging)

mlsomers on 19 May 2020

I'm getting this error quite frequently when a client disconnects. There are any plans on catching this exception on the next releases or should I catch this exception?
Thanks!

joch0a on 10 Jun 2020

@JoseOchoaITH There are no plans to internally catch ConnectionResetExceptions thrown while reading the request body. To the contrary, we go out of our way to throw ConnectionResetExceptions so truncated requested bodies caused by early client disconnects are not mistaken for complete request bodies.

If the goal is to clean up the error logs that result when a client disconnects mid-request-body, then you should catch the exception yourself. ConnectionResetException is an IOException, so I recommend catching IOExceptions. Handling all IOExceptions in the same manner works well for most apps.

halter73 on 10 Jun 2020

👀1 🎉1 👍1

Tratcher on 11 Jun 2020

👍3

Any recommendation on how to best error handle this, ends up from my controller which has try/catch to the last middleware which is not the most obvious place for custom error handling IMO.

cgountanis on 18 Aug 2020

I don't know if my issue is related to ModelBinding or not, but I've been seeing these errors since January and still haven't found a resolution. We were running Asp.Net Core 2.2 and everything was fine, not a single one of these error. We rolled out and update that included an upgrade to Asp.Net Core 3, and we are no on 3.1. Literally the following day, 8 of these errors. On a monthly bases, we see 150 - 200 of these errors. That's with thousands of unique users where at any given time there are 100+ active unique sessions. These errors only happen on POST, but they happen fairly equally across our user-base. No pattern there.

Microsoft.AspNetCore.Connections.ConnectionResetException: The client has disconnected
---> System.Runtime.InteropServices.COMException (0x80070040): The specified network name is no longer available. (0x80070040)
--- End of inner exception stack trace ---
at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult(Int16 token)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory1 memory, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.EnsureBufferedAsync(Int32 minCount, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool1 bytePool, Nullable1 limit, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList1 factories) at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.TryCreateAsync(ActionContext actionContext, IList1 factories)
at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

ItWorksOnMyMachine on 1 Oct 2020

Were you using IIS in-proc in 2.2? This might just be a change in how the error surfaces due to changing defaults like in-proc vs out-of-proc. But yes, this is in the category of errors that model binding should catch (it derives from IOException).

Tratcher on 1 Oct 2020

I wish I had a better answer than "I'm not quite sure" but that's it. We never specified, and were too ignorant at the time to really understand that we should know, but I believe the default was out-of-proc for 2.2, so that's why I say I'm not quite sure. But I believe we were out of proc. I'm actually moving it back to out-of-proc now just to see if that resolves the issue. I've seen other anecdotal evidence that suggests it is an issue with not enough stack space. We do have logs of request headers when this happens, and the ones that aren't legitimate client disconnects, all have large content-length headers.

ItWorksOnMyMachine on 1 Oct 2020

Stack space is unrelated to request body sizes, and you'd get a StackOverflowException if that were the issue.

Can you share more about the requests you saw that weren't legitimate disconnects? How could you tell?

Tratcher on 1 Oct 2020

I suppose I should be more clear. Some of the disconnects we can prove users navigated away from the page. However, the ones we consider not "legitimate" disconnects may still be disconnects at the client, but not due to an intentional action by the user. In these cases, the user clicks a button to perform an action like upload a file, save the state of the page, etc. We have a standard object that handles the low-level Http Fetch operation and upon completion of the fetch promise, we parse the result and if it's not a successful result, we present the user with an error dialog. These specific results return a "Network request failed" error. We get a screenshot of the error box from our users clearly showing they didn't navigate away from the page. Now, that doesn't mean that a firewall or something didn't terminate the request. So, I don't know for sure that it's not a "legitimate" client disconnect, just that it's not a user-initiated disconnect.

Prior to the upgrade to 3+, we never got reports like this from users. So while it might just be that an exception is presenting itself differently due to being in-proc instead of out-of-proc, we never received reports of errors happening randomly like this previously. And, in most cases, if the user simply dismissed the error dialog and clicks the "Save" button again, it frequently works just fine.

ItWorksOnMyMachine on 1 Oct 2020

How large is the content-length?

Tratcher on 1 Oct 2020

We have 1,351 of these exceptions where we think they happened without an intentional navigation event. They range anywhere from 2 bytes to 443 MB! We have some file upload endpoints that allow for large files. We explicitly set the size limits larger on those, so they should work, and usually do. The average size of the failed request is about 10 MB though. This has happened across 51 unique api endpoints and only a couple of those accept files. 1,351 errors is a lot, but in context, this website has served up nearly 35 million api requests so far this year.

ItWorksOnMyMachine on 1 Oct 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Test concurrent connection limit on different Azure VM sizes

BrennanConroy · 3Comments

Support multiple [FromRoute], [FromHeader], [FromBody], etc in a single object

snebjorn · 3Comments

How to use ADO.NET in asp.net core 1.0?

ermithun · 3Comments

Component compilation support needs to add diagnostics for markup errors

rynowak · 3Comments

ADOMD.NET in ASP.NET Core 2.0.0

Kevenvz · 3Comments