Kestrelhttpserver: BadHttpRequestException Request timed out.

[themetis]

We keep having this error frequently and randomly on any request from our clients since we upgraded from .Net Core 1.1.1 to .Net Core 2.

BadHttpRequestException Request timed out.
at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion.ThrowFailed()
at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetResult(ReadResult& result)
at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()

Any idea what's the cause of this exception and how to avoid it?

[davidfowl]

I believe it's related to this https://github.com/aspnet/KestrelHttpServer/issues/2151 but I'm not 100% sure. Do you have any other logs?

[themetis]

No it's all I've got

[halter73]

Given that this is a timeout, it might not be related to #2151. Do you know by chance if any of the requests are coming from a .NET linux HttpClient?

[themetis]

@halter73 No, I don't think so.

I manage to have a more complete log of my exception:
BadHttpRequestException: Request timed out.
Module "Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion", line 0, col 0, in ThrowFailed
Void ThrowFailed()
Module "Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe", line 0, col 0, in GetResult
Void GetResult(Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadResult ByRef)
Module "Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe", line 45, col 0, in Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult
Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadResult Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()
Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody", line 130, col 0, in ReadAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameRequestStream", line 135, col 0, in ReadAsyncInternal
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "System.IO.StreamReader", line 250, col 0, in ReadBufferAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "System.IO.StreamReader", line 152, col 0, in ReadAsyncInternal
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.WebUtilities.FormReader", line 155, col 0, in BufferAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.WebUtilities.FormReader", line 120, col 0, in ReadNextPairAsyncImpl
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.WebUtilities.FormReader", line 118, col 0, in ReadFormAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 8, col 0, in ValidateEnd
Void ValidateEnd(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Http.Features.FormFeature", line 442, col 0, in InnerReadFormAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory", line 121, col 0, in AddValueProviderAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider", line 145, col 0, in CreateAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider", line 128, col 0, in CreateAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider+<>c__DisplayClass0_0+<g__Bind0>d", line 121, col 0, in MoveNext
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker", line 152, col 0, in InvokeInnerFilterAsync
Void MoveNext()
Module "System.Runtime.CompilerServices.TaskAwaiter", line 11, col 0, in ThrowForNonSuccess
Void ThrowForNonSuccess(System.Threading.Tasks.Task)
Module "System.Runtime.CompilerServices.TaskAwaiter", line 40, col 0, in HandleNonSuccessAndDebuggerNotification
Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Module "Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker", line 159, col 0, in InvokeNextExceptionFilterAsync

[davidfowl]

I'm like 87% sure it's #2151, we'll try to get a patched build so you can try out the fix.

[greatkeke]

I had met that exception. I post StreamConent via HttpClient .

public static async Task GoAsync()
{
    var client = new HttpClient() { BaseAddress = new Uri("http://upload.domain.com") };
    using (var memory = new MemoryStream())
    {
        var file = File.OpenRead("img/download.jpeg");
        file.CopyTo(memory);
        file.Dispose();

        var tasks = new List<Task>();
        for (int i = 0; i < 10; i++)
        {
            var request = new HttpRequestMessage(HttpMethod.Post, "image");
            memory.Seek(0, 0);
            request.Content = new StreamContent(memory);
            tasks.Add(client.SendAsync(request));
        }
        await Task.WhenAll(tasks);
    }
}

The end of Server

[HttpPost("image")]
public IActionResult Post()
{
   using(var stream = new MemoryStream())
   {
       Request.Body.CopyTo(stream);
        //do noting....
    }
    return Ok();
}

The error is log.txt

2017-12-05 22:29:25.461 +08:00 [Error] An unhandled exception has occurred while executing the request
 /image
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request timed out.
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion.ThrowFailed()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetResult(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadableBufferAwaitable.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.<ReadAsync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameRequestStream.<ReadAsyncInternal>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize)
   at XXX.UploadController.<Post>d__11.MoveNext() in C:\Users\I\Documents\src\Web\Controllers\UploadController.cs:line 21
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.<Invoke>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[Drawaes]

you are blocking on the server

Request.Body.CopyTo(stream);

Try

[HttpPost("image")] public async Task<IActionResult> Post() { using(var stream = new MemoryStream()) { await Request.Body.CopyToAsync(stream); //do noting.... } return Ok(); }

[greatkeke]

@Drawaes I had tried your code, but the exception occurred again.

[halter73]

@kexxxfeng what image are you using?

[benaadams]

You are sending the same memory concurrently to 10 connections? Reading position in MemoryStream will get mixed up?

[benaadams]

You don't need to dispose MemoryStream but do need to dispose HttpClient as used in the sample you've given.

Try this to create a MemoryStream for each task:

public static async Task GoAsync()
{
    using (var client = new HttpClient() { BaseAddress = new Uri("http://upload.domain.com") })
    {
        var memory = new MemoryStream();
        using (var file = File.OpenRead("img/download.jpeg");)
        {
            await file.CopyToAsync(memory);
        }
        memory.Seek(0, 0); // Not sure needed
        var data = memory.ToArray();

        var tasks = new List<Task>();
        for (int i = 0; i < 10; i++)
        {    
            var taskMemory = new MemoryStream(data);
            var request = new HttpRequestMessage(HttpMethod.Post, "image");
            request.Content = new StreamContent(taskMemory);
            tasks.Add(client.SendAsync(request));
        }
        await Task.WhenAll(tasks);
    }
}

[halter73]

You can even use the array directly. So instead of creating taskMemory, you can just do request.Content = new ByteArrayContent(data);.

[greatkeke]

@benaadams @halter73 That's great. Thx 👍

[greatkeke]

:( Today, I found a lot of exception like below in the log of server end.

2017-12-07 14:39:58.709 +08:00 [Error] An unhandled exception has occurred while executing the request
/Upload/Image
Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions.Internal.ConnectionResetException: Error -4077 ECONNRESET connection reset by peer ---> Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4077 ECONNRESET connection reset by peer
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompletion.ThrowFailed()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetResult(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwaiter.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadableBufferAwaitable.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.LibuvOutputConsumer.<WriteOutputAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

I've just sent 100 post request to the server. Every image is only 4MB.
```c#
public static async Task GoAsync()
{
using (var client = new HttpClient() { BaseAddress = new Uri("http://upload.domain.com") })
{
var memory = new MemoryStream();
using (var file = File.OpenRead("img/download.jpeg");)
{
await file.CopyToAsync(memory);
}
memory.Seek(0, 0); // Not sure needed
var data = memory.ToArray();

    var tasks = new List<Task>();
    for (int i = 0; i < 100; i++) 
    {    
        var request = new HttpRequestMessage(HttpMethod.Post, "image");
        request.Content = new ByteArrayContent(data);
        tasks.Add(client.SendAsync(request));
    }
    await Task.WhenAll(tasks);
}

}
```
Or, the exception means that the resource of the server is not enough to handle the 100 requests?

[halter73]

Error -4077 ECONNRESET connection reset by peer

This generally means the client aborted the connection mid-request.

[greatkeke]

@halter73 Yeah, I resolved it by setting httpClient.TimeOut = new TimeSpan(1,0,0);// 1 hour

[zaozaoniao]

wow， set the httpClinet.TimeOut not enough grace. any other soultion?

[halter73]

@Julian89757 Timeouts are generally caused by the client not hitting the default rate limit. Did you capture any logs?

[aspnet-hello]

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.