Aspnetcore: Problem uploading large files with Kestrel + Docker

[david-preston-triad]

Description

I have two issues with file uploads in ASP.Net Core running on Kestrel and Docker for Windows/Docker Desktop (Linux container):

1. When I try to upload a file that is larger than the default limit (30MB), my website hangs and I cannot make any further requests. I need to restart Docker Desktop and redeploy my website to get it running again.
2. I want to display a custom error page when a user attempts to upload a file that exceeds the default limit. I have implemented a middleware component to catch the BadHttpRequestException that is thrown and redirect to a custom error page. I have found that this doesn't work unless I either debug the code in Visual Studio and step-through the code or I insert a call to System.Threading.Thread.Sleep().

To Reproduce

Steps to reproduce the behavior:

1. I am using Visual Studio 2017 15.9.4, ASP.NET Core SDK 2.2.101 and Docker Desktop 2.0.0.2 (30215)
2. I attach a file containing a sample project to reproduce the problem (see FileUpload.zip below). Run this code and navigate to the /Update controller. Upload a file larger than 30MB (mine was 64MB). In my environment, the website hangs.
3. Edit Startup.cs to uncomment the call to app.UseBadHttpRequestExceptionHandlerMiddleware().
   Restart Docker Desktop and run the website again. Upload the file larger than 30MB again. This time, the custom error page should be displayed after a delay.

FileUpload.zip

Expected behavior

I was expecting the middleware to redirect to the custom error page without having to call Thread.Sleep() before Response.Redirect().

Additional context

Here is the output of dotnet --info:

.NET Core SDK (reflecting any global.json):
Version: 2.2.101
Commit: 236713b0b7

Runtime Environment:
OS Name: Windows
OS Version: 10.0.15063
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.2.101\

Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed

.NET Core SDKs installed:
1.0.0 [C:\Program Files\dotnet\sdk]
2.1.4 [C:\Program Files\dotnet\sdk]
2.1.103 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.502 [C:\Program Files\dotnet\sdk]
2.2.101 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

[david-preston-triad]

I have tried running my sample project in Windows using Kestrel (i.e. by running dotnet FileUpload.dll at the command line). I found that I always get an error - ERR_CONNECTION_RESET in Chrome and INET_E_DOWNLOAD_FAILURE in Edge.

[osnoser1]

I have tried running my sample project in Windows using Kestrel (i.e. by running dotnet FileUpload.dll at the command line). I found that I always get an error - ERR_CONNECTION_RESET in Chrome and INET_E_DOWNLOAD_FAILURE in Edge.

This problem also occurs without a docker. I have tried to return an object of type ProblemDetails in a web api without success. 😢

[AdamDotNet]

Short version: I'm not using Docker, but catch (BadHttpRequestException ex) when (ex.StatusCode == 413) does not work as expected. The connection dies instead of letting me return a custom response.

Your BadHttpRequestExceptionHandlerMiddleware looks similar to mine, but I'm trying to write a JSON response not redirect. In any case, I've seen the response actually work and other times just die. Usually just die. It also looks like when the connection dies, that possibly Angular's HttpClient retries 2-4 times, which would upload the file more than once which makes Kestrel feel like it's opening itself up to DOS attack by not responding correctly. (Not entirely sure why the request is being repeated from our Angular client; we don't explicitly have retry logic in our code. I just know that when I test with fiddler instead, there are no repeated calls. Maybe web browsers are doing it automatically? I tried Chrome.).

The smallest reproduction case I can come up with is this.

Use Asp.Net Core 2.2.3 or 3.0 preview 3, and debug using kestrel, not IIS Express.

Upload a relatively small file such as 500KB. Observe that Fiddler gets the expected status code and text.

Upload a relatively large file, such as 367MB (not sure what the size threshold is, but that's the size of my "real" file test). Observe that always, or almost always, you'll get the connection to die and fiddler reports HTTP 504, with either the connection was closed forcibly or the server did not return a complete response.

The entire app is just Program.cs doing its default thing, and this Startup file:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;

namespace TestHandleLargeUploadException
{
    public class Startup
    {
        public void Configure(IApplicationBuilder app)
        {
            app.Use(async (httpContext, next) =>
            {
                // Set limit to really small - imagine [RequestSizeLimit()] in MVC, but with a normal value.
                var limitFeature = httpContext.Features.Get<IHttpMaxRequestBodySizeFeature>();
                limitFeature.MaxRequestBodySize = 1024;

                try
                {
                    await next();
                }
                catch (BadHttpRequestException ex) when (ex.StatusCode == 413)
                {
                    // This catch block always fires, but the client doesn't always get this message.
                    // Small files seem to always work, while larger files almost always just kill the connection.
                    httpContext.Response.StatusCode = 400;
                    // Imagine JSON payload instead of this example.
                    await httpContext.Response.WriteAsync($"Data uploads are limited to {limitFeature.MaxRequestBodySize} bytes.");
                }
            });

            app.Use(async (httpContext, next) =>
            {
                // Use Fiddler to post a file over the limit specified above.
                if (httpContext.Request.Method == "POST")
                {
                    // Trigger BadHttpRequestException
                    int count = httpContext.Request.Form.Count;
                }
                await httpContext.Response.WriteAsync("Hello World!");
            });
        }
    }
}

[anurse]

1. I need to restart Docker Desktop and redeploy my website to get it running again.

As far as I can tell, this is a docker issue. I can reproduce it and still access the site just fine from within the container.

The connection dies instead of letting me return a custom response.

I can also reproduce this, we'll have to look in to this a little further. What appears to happen is that the recovery code (redirecting to an error page) in the middleware does run, and the response is written to the socket. However, Kestrel goes to drain the request body and the "request body too large" exception is thrown again during that process, which causes the connection to be closed. What is likely happening is that the client (browser) sees the connection close while it's still uploading content and simply terminates the connection without ever actually processing the incoming response.

[drauch]

We also ran into the same problem, is there a way to return a proper BadRequest response?