Aspnetcore: Uploading huge files

[Jcomper]

Unable to upload big file when using IIS hosting

A clear and concise description of what the bug is.
When and file size is greater then 2GB file upload is stucking and 502 Bad Gateway is returned

To Reproduce

1. Download Sample app which is based on samples Upload files in ASP.NET Core with a slight modification to StreamingController to use FileStream instead of MemoryStream
2. In the appsettings.json file set the path for stored files (StoredFilesPath).
3. Publish it to IIS
4. Modify web.config to have hostingModel="outofprocess" and add requestLimits maxAllowedContentLength="4294967295" to requestFiltering section

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <location path="." inheritInChildApplications="false">
    <system.webServer>
      <handlers>
        <remove name="aspNetCore" />
        <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
      </handlers>
      <aspNetCore processPath="dotnet" arguments=".\SampleApp.dll" stdoutLogEnabled="true" stdoutLogFile=".\logs\stdout" hostingModel="outofprocess" requestTimeout="00:02:00" />
      <security> 
        <requestFiltering>
         <requestLimits maxAllowedContentLength="4294967295" />
        </requestFiltering>
        </security>
    </system.webServer>
  </location>
</configuration>

1. Navigate to http://localhost/StreamedSingleFileUploadPhysical choose file greater then 2GB and click Upload button

• In a scenario when hosting using Kestrel Web Server everything is working as expected and I'm able to upload huge file (6GB)

• With modification to run with hostingModel="inprocess" (Program.cs)
  C# public static IHostBuilder CreateHostBuilder(string[] args) => Host.CreateDefaultBuilder(args) .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>() //.UseKestrel(options => options.Limits.MaxRequestBodySize = null) .UseIISIntegration(); ; });
  and adding [DisableRequestSizeLimit] attribute to the UploadPhysical() endpoint I'm able to upload files up to 4 GB. However, when uploading files greater than 4GB I'm getting 413 Request Entity Too Large response. Is it possible to upload really huge files with ASP.NET Core when hosting it in IIS?

Further technical details

❯ dotnet --info

.NET Core SDK (reflecting any global.json):
Version: 3.0.100
Commit: 04339c3a26

Runtime Environment:
OS Name: Windows
OS Version: 10.0.18362
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk3.0.100\

Host (useful for support):
Version: 3.0.0
Commit: 7d57652f33

.NET Core SDKs installed:
2.1.100 [C:\Program Files\dotnet\sdk]
2.1.101 [C:\Program Files\dotnet\sdk]
2.1.201 [C:\Program Files\dotnet\sdk]
2.1.202 [C:\Program Files\dotnet\sdk]
2.1.300 [C:\Program Files\dotnet\sdk]
2.1.403 [C:\Program Files\dotnet\sdk]
2.1.500 [C:\Program Files\dotnet\sdk]
2.1.502 [C:\Program Files\dotnet\sdk]
2.1.505 [C:\Program Files\dotnet\sdk]
2.1.507 [C:\Program Files\dotnet\sdk]
2.2.401 [C:\Program Files\dotnet\sdk]
3.0.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

After timeout exceeded following are from stdout_log:

dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[11]
List of registered output formatters, in the following order: Microsoft.AspNetCore.Mvc.Formatters.HttpNoContentOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StringOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StreamOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[4]
No information found on request to perform content negotiation.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[8]
Attempting to select an output formatter without using a content type as no explicit content types were specified for the response.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[10]
Attempting to select the first formatter in the output formatters list which can write the result.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[2]
Selected output formatter 'Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter' and content type 'application/json' to write the response.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value of type 'Microsoft.AspNetCore.Mvc.SerializableError'.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Executed action SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp) in 135935.2648ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp)'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 135989.30250000002ms 400 application/json; charset=utf-8
dbug: Microsoft.AspNetCore.Server.Kestrel[26]
Connection id "0HLRD6A3O7L0O", Request id "0HLRD6A3O7L0O:00000001": done reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel[2]
Connection id "0HLRD6A3O7L0O" stopped.
dbug: Microsoft.AspNetCore.Server.Kestrel[39]
Connection id "0HLRD6A3O7L0P" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel[1]
Connection id "0HLRD6A3O7L0P" started.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/1.1 POST http://localhost/Streaming/UploadPhysical multipart/form-data; boundary=--------------------------532290777779290238672096 2149435735
dbug: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[1]
POST requests are not supported
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
1 candidate(s) found for the request path '/Streaming/UploadPhysical'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
Endpoint 'SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp)' with route pattern '{controller=Home}/{action=Index}/{id?}' is valid for the request path '/Streaming/UploadPhysical'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
Request matched endpoint 'SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp)'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp)'
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
Route matched with {action = "UploadPhysical", controller = "Streaming", page = ""}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] UploadPhysical() on controller SampleApp.Controllers.StreamingController (SampleApp).
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Execution plan of authorization filters (in the following order): None
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Execution plan of resource filters (in the following order): Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.SaveTempDataFilter, SampleApp.Filters.DisableFormValueModelBindingAttribute
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Execution plan of action filters (in the following order): Microsoft.AspNetCore.Mvc.Filters.ControllerActionFilter (Order: -2147483648), Microsoft.AspNetCore.Mvc.ModelBinding.UnsupportedContentTypeFilter (Order: -3000)
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Execution plan of exception filters (in the following order): None
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Execution plan of result filters (in the following order): Microsoft.AspNetCore.Mvc.ViewFeatures.Filters.SaveTempDataFilter
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[1]
Executing controller factory for controller SampleApp.Controllers.StreamingController (SampleApp)
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Executed controller factory for controller SampleApp.Controllers.StreamingController (SampleApp)
dbug: Microsoft.AspNetCore.Server.Kestrel[25]
Connection id "0HLRD6A3O7L0P", Request id "0HLRD6A3O7L0P:00000001": started reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[4]
Connection id "0HLRD6A3O7L0P" paused.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[5]
Connection id "0HLRD6A3O7L0P" resumed.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[19]
Connection id "0HLRD6A3O7L0P" reset.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
Connection id "0HLRD6A3O7L0P" sending FIN because: "The client closed the connection."
dbug: Microsoft.AspNetCore.Server.Kestrel[10]
Connection id "0HLRD6A3O7L0P" disconnecting.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[11]
List of registered output formatters, in the following order: Microsoft.AspNetCore.Mvc.Formatters.HttpNoContentOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StringOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.StreamOutputFormatter, Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[4]
No information found on request to perform content negotiation.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[8]
Attempting to select an output formatter without using a content type as no explicit content types were specified for the response.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[10]
Attempting to select the first formatter in the output formatters list which can write the result.
dbug: Microsoft.AspNetCore.Mvc.Infrastructure.DefaultOutputFormatterSelector[2]
Selected output formatter 'Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonOutputFormatter' and content type 'application/json' to write the response.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value of type 'Microsoft.AspNetCore.Mvc.SerializableError'.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Executed action SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp) in 137223.6736ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'SampleApp.Controllers.StreamingController.UploadPhysical (SampleApp)'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 137227.7231ms 400 application/json; charset=utf-8
dbug: Microsoft.AspNetCore.Server.Kestrel[26]
Connection id "0HLRD6A3O7L0P", Request id "0HLRD6A3O7L0P:00000001": done reading request body.
dbug: Microsoft.AspNetCore.Server.Kestrel[2]
Connection id "0HLRD6A3O7L0P" stopped.

[Tratcher]

Related to https://github.com/aspnet/AspNetCore/issues/2711.

[jkotalik]

From what I can tell, the max you can have in IIS is 4 GB as that is the max value of the uint field that can be read.

The only thing I can think of is removing the request filtering module from IIS, but I wouldn't recommend doing that. Removing default modules can be risky.

Can you control how the client is uploading the file? Potentially doing a chunked upload instead?

[Jcomper]

@jkotalik

I agree that chunked upload looks like a reasonable way for production. And clients of API will have to implement it their side. But I'm still interested in removing the request filtering module from IIS for test purposes. Do you know how that can be achieved?

[jkotalik]

You can remove the RequestFiltering module from the appliationhost.config file, however I'd be cautious doing that.

[Cloudmersive]

How is this still not fixed? It's 2020 and ASP.NET still can't take input files > 2GB.

[davidfowl]

@Cloudmersive I believe the options while running in IIS were explained here. What exactly are you looking for? Changes to IIS a by default to allow the large content length based uploads to work?

I’d like to get an idea of what “fixed” means to you

[Cloudmersive]

@davidfowl Thanks for reaching out. Fixed ideally would be across Microsoft technologies large files "just works" end-to-end. So IIS has a story for large files, ASP.NET Core/5 has a story for large files, and everything just works - and documentation accurately describes how to achieve this outcome. Indeed, fixed would mean there is nothing special about 2 GB vs. 10 GB vs. 1 TB. IIS has an archaic 32-bit integer for setting the size limit and so maxes out at 2 GB. Ideally, this parameter should be deprecated and replaced with a 64-bit long for setting the size limit. Ideally, also any limits in ASP.NET Core use 64-bit longs as well. In addition, ideally all of the machinery in ASP.NET core can smoothly handle large files.

Large files are commonplace in 2020 and so this is an important scenario. Moreover, the fact that this limit is driven by a 32-bit integer overflow feels really broken, especially in a 64-bit world.

We can try this workaround and hopefully this will work. But ideally Microsoft can fix the scenario of large files. Anything relating to large file transfer hits this and causes all sorts of issues.

Any help with this would be greatly appreciated by us and the ASP.NET community!

[davidfowl]

ASP.NET Core works, the issue is IIS specific. So you can disable request filtering to allow bigger content length based uploads if you’re stuck on IIS.

As much as we try to give developers a unified experience we sometimes can’t hide the layers things are built on top of so you directly have to deal with that pain. This has changed because they are well known workarounds and changing defaults is a big deal for backwards compatibility (IIS ships with windows).

This issue should be closed and this limitation should be documented as there’s nothing here that’s going to be fixed.

[Cloudmersive]

But @davidfowl why not just get IIS to fix this? They can fix it without breaking backward comparability by just adding a new/different parameter that overrides the current legacy one and deprecating the old one.

[davidfowl]

I can think of a few reasons:

• this behavior has been around forever. Changing the default could render severs insecure. A new setting would need to be introduced.
• there’s a way to work around it already.
• the fix would be available in the next version of Windows. I’m not sure this meets the “backport to an older version of Windows bar”.

[Cloudmersive]

@davidfowl Putting in a new version of Windows seems OK if that is what is required. We wouldn't need to change the default, just introduce a new setting and deprecate the current one, or make the new setting override the old one when present and not deprecate - either way no need to break backward compatability.

The workaround is not documented and seems like then other aspects of request filtering will be lost, so is not a great workaround. You would, for example, presumably lose the ability to have a limit. The goal is not to remove the limit, just allow it to be larger than 2^32

[davidfowl]

It should also important to understand that fixing it in IIS in a new version of Windows sever doesn’t mean it’ll be available everywhere on release day (whenever that would be). It means you still wouldn’t be able to use it everywhere and would have to work around it otherwise.

Using a long to store the limit would suffice as you said but again but features like this aren’t great sever wide. You have to bump it for all requests rather than the one that is receiving the upload. Better to remove the IIS request filtering limit and apply more granular limits in ASP.NET Core IMO.

As for fixing the issue is IIS for some future windows cc @shirhatti for his thoughts

[Cloudmersive]

@davidfowl I think that is fair - but it is still highly worthwhile actually fixing IIS's 2 GB limit. In a new major version of the OS, presumably there would be a new major version of IIS and so something could be done, but without necessarily breaking backward compatability. This is super broken in IIS and then at least this issue would be finally fixed after literally decades of not being fixed.

Is there any other functionality in request filtering that would be lost that cannot be replaced by an equivalent in ASP.NET Core? If so, hopefully at least the documentation could be holistic in describing how to turn off request filtering, what is lost, what settings to put into ASP.NET core to regain this functionality, etc. It does seem a little risky that the request filtering would be completely off and now things are only filtered on the .NET core side.

[msftbot[bot]]

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!