Kestrelhttpserver: System.IO.IOException: Error -4077 ECONNRESET connection reset by peer

Created on 6 May 2017 · 16Comments · Source: aspnet/KestrelHttpServer

I have a high traffic website behind iis load balancer and cloudflare.

i'm using "Microsoft.AspNetCore.Server.Kestrel": "1.1.1",

and "Microsoft.AspNetCore.Server.IISIntegration.Tools": "1.1.0-preview4-final",

but i'm constantly getting this error and all request are gets dropped

what should i do?

Source

amirjalali1

👍1

Most helpful comment

The error suggest that the client sent an RST. The server exception is expected when that happens. It can happen for example if the application was closed during the POST.

davidfowl on 5 Jul 2017

👍2

All 16 comments

but i'm constantly getting this error and all request are gets dropped

What do you mean "all requests get dropped"? That stack trace looks like a single error on a single request. The client disconnected while the antiforgery store was reading the form. It looks harmless. The logs might just be scary but that behavior is expected.

davidfowl on 6 May 2017

@davidfowl when this kind of error happens which happens a lot,
on monitoring section of iis load balancer , all request are get failed.
also im using redis for dapi

number of failed request at the following image is faked but after the mentioned error every request is failed

errors

load balancer

amirjalali1 on 6 May 2017

That's interesting, what load balancer are you using?

davidfowl on 7 May 2017

@davidfowl I'm using IIS arr 3.0 as load balancer.
iis load balancer is behind cloudflare.
i have frequent down time after checking everything i reach to this error.
I have logged and inspected everything from iis logs to db,redis instances, networks and so on but i can't find the problem.
I'm going to change the code which causes this problem but it will take 2 or 3 days.

amirjalali1 on 7 May 2017

@1amirjalai any updates?

muratg on 23 Jun 2017

@muratg no i still have this issue. even after migrating to 1.1.2 and vs2017

System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer --- End of inner exception stack trace --- at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInput.CheckConnectionError() at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInputExtensions.<PeekAsyncAwaited>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ValueTaskAwaiter1.GetResult() at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ForContentLength.d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ValueTaskAwaiter1.GetResult() at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.<ReadAsyncAwaited>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.IO.StreamReader.<ReadBufferAsync>d__102.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.IO.StreamReader.<ReadAsyncInternal>d__69.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.WebUtilities.FormReader.<BufferAsync>d__41.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.WebUtilities.FormReader.<ReadNextPairAsyncImpl>d__34.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.WebUtilities.FormReader.<ReadFormAsync>d__43.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Features.FormFeature.<InnerReadFormAsync>d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenStore.<GetRequestTokensAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery.<ValidateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Mvc.ViewFeatures.Internal.ValidateAntiforgeryTokenAuthorizationFilter.<OnAuthorizationAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NWebsec.AspNetCore.Middleware.Middleware.MiddlewareBase.<Invoke>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.<Invoke>d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.<Invoke>d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.<Invoke>d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware1.<Invoke>d__18.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Session.SessionMiddleware.<Invoke>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Session.SessionMiddleware.<Invoke>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()

amirjalali1 on 27 Jun 2017

I'm also getting this error while on the latest 1.1.2 release. The error is occurring about once per hour when posting a fairly large number of 10KB JSON messages. I'm using IIS as a reverse proxy.

System.AggregateException: One or more errors occurred. ---> 
System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInput.CheckConnectionError()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInputExtensions.<PeekAsyncAwaited>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ForContentLength.<PeekAsyncAwaited>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.<ReadAsyncAwaited>d__12.MoveNext()
   --- End of inner exception stack trace ---
   at System.IO.Compression.DeflateStream.EndRead(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.<ReadAsync>d__38.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamReader.<ReadBufferAsync>d__97.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamReader.<ReadToEndAsyncInternal>d__62.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Bridge.Logging.AspNetCore.RequestResponseLoggingAttribute.<GetContentValue>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Bridge.Logging.AspNetCore.RequestResponseLoggingAttribute.<HandleResult>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Bridge.Logging.AspNetCore.RequestResponseLoggingAttribute.<OnResultExecutionAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.RequestDecompression.RequestDecompressionMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware`1.<Invoke>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.<Invoke>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame`1.<RequestProcessingAsync>d__2.MoveNext()
---> (Inner Exception #0) System.IO.IOException: Error -4077 ECONNRESET connection reset by peer ---> 
Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInput.CheckConnectionError()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.SocketInputExtensions.<PeekAsyncAwaited>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ForContentLength.<PeekAsyncAwaited>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.<ReadAsyncAwaited>d__12.MoveNext()<---

RehanSaeed on 4 Jul 2017

@RehanSaeed did the client disconnect during the post?

davidfowl on 5 Jul 2017

We use an X-Request-ID HTTP header to correlate requests made from an Android app to our API. After looking in the logs this morning, I didn't see any client errors correlated to the above error. I need to get our Android dev to confirm though.

RehanSaeed on 5 Jul 2017

The error suggest that the client sent an RST. The server exception is expected when that happens. It can happen for example if the application was closed during the POST.

davidfowl on 5 Jul 2017

👍2

@1amirjalai / @RehanSaeed: We are facing the same problem; 100+ requests per second per machine with no problems then suddenly, all requests fail for a machine with an IOException "Error -4077 ECONNRESET connection reset by peer". Did you resolve the issue?

dotarj on 27 Sep 2017

Still not resolved, I still see lots of these errors in my Kibana dashboard. I'm hoping an upgrade to ASP.NET Core 2.0 will resolve this issue, just need to find the time to do that.

RehanSaeed on 28 Sep 2017

We still have no idea why the client is being disconnected but the error itself is expected when the client disconnects. There's no evidence to suggest there's a bug here.

davidfowl on 27 Nov 2017

Hi @davidfowl, for us the problem was related to thread starvation as described in this post: https://forums.couchbase.com/t/frozen-application-after-application-pool-starts-or-recycles-under-load/14573. After applying the described workaround the problem disappeared. I Guess the IOExceptions were caused by clients that disconnected because requests took too long.

dotarj on 27 Nov 2017

👍1

It happens a lot lately, for some reason. Behind Azure Traffic Manager (DNS load balancing really).
They all drop with response times of about 1000ms. Possibly related, it seems like the client is so far always a React Native app, indeed suggesting the client being the source of error.

janpieterz on 15 Jun 2018

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.