Aspnetcore: Unable to upload file greater than 2GB in size with ANCM Out-Of-Process and Kestrel

[aspnet-hello]

_From @kagoyal on Wednesday, February 1, 2017 3:11:34 PM_

I am trying to upload file greater than 2GB using multipart formdata.

The issue I am observing is after 2GB upload, read of input stream gets stuck and process hangs.
Microsoft.AspNetCore.Server.Kestrel version we are using is "1.1.0".

HTTP request’s response never come back and timesout after one hour (execution time we have set).
I have modified IIS post limits for upto 4GB, also in ConfigureServices –

            services.Configure<FormOptions>(x =>
            {
                x.MultipartBodyLengthLimit = 4294967296; 
            });

I am following the sample here - https://github.com/aspnet/Docs/tree/master/aspnetcore/mvc/models/file-uploads/sample/FileUploadSample to read multipart data.

Please let me know if any more information is required to debug this, or if this is expected behavior.

_Copied from original issue: aspnet/HttpAbstractions#766_

[aspnet-hello]

_From @benaadams on Wednesday, February 1, 2017 4:11:23 PM_

What's the upload speed/transfer rate?

[aspnet-hello]

_From @benaadams on Wednesday, February 1, 2017 4:13:31 PM_

/cc @Eilon

[aspnet-hello]

_From @kagoyal on Wednesday, February 1, 2017 4:15:33 PM_

I am able to upload approx. 1.8 GB of data in about 30-35 seconds.

[aspnet-hello]

_From @blowdart on Wednesday, February 1, 2017 4:18:46 PM_

Well the timeout is working, so not a security concern, just a weird bug. @muratg

[aspnet-hello]

_From @JunTaoLuo on Wednesday, March 1, 2017 2:46:26 PM_

I am able to upload 4GB files when using Kestrel directly but I ran into problems when running it on IIS. I was not able to produce a hang but I do see a 502 failure response. It seems like the connection is being reset and it could be due to ANCM. I filed https://github.com/aspnet/AspNetCoreModule/issues/78.

[aspnet-hello]

_From @JunTaoLuo on Wednesday, March 1, 2017 7:26:39 PM_

I confirmed that ANCM is the module that is producing the 502 error code from the FREB logs. The limit seems to be 2GB for upload file size so there's no way of uploading bigger files when running behind IIS. I'm closing this issue as this is not a restriction in AspNet Core @kagoyal please re-open if you have any additional concerns or comments.

[aspnet-hello]

_From @JunTaoLuo on Thursday, March 2, 2017 11:37:32 AM_

Re-opening investigation

[aspnet-hello]

_From @cesarbs on Thursday, March 2, 2017 2:30:06 PM_

We've had a similar issue in Kestrel because we were using and int instead of a long to keep track of the request body size. Could that be the case here?

[aspnet-hello]

_From @JunTaoLuo on Thursday, March 2, 2017 3:06:27 PM_

There are a few issues at play here. There is a limit with the Request Filter in IIS which has a limit of UInt32.Max (~4GB) but this filter can be removed. By testing files of different sizes I also found that the request size is limited by ANCM to ~2GB likely limited by int. However, I'm reopening the issue for investigation since I believe the way I'm reproducing the issue is flawed. I was trying streaming uploads from browsers (chrome, firefox, ie, edge) but these browsers seem to automatically add a Content-Length header to the request which is not what we want for multipart forms. I'm going to try reproducing this issue with a custom client.

[muratg]

Moving this one out. We'll need to do more investigation.

[sonicmouse]

So is this fixed in 2.1? We are trying to upload a file that's ~3GB and it it stops as soon as the upload byte count exceeds 2,147,483,647. I am currently using ASP.NET Core 2.0. To move to 2.1 will be some work, so I'd like to make sure the work would be worth it. If not, i'll just wait until this is fixed.

thanks

[ryanski44]

This is a major blocking bug for our application. We can't upload anything over 2GB when running through IIS. Is there any workaround?

[JunTaoLuo]

@shirhatti do you know about the IIS limit at play here?

[ryanski44]

I found an area of code that would limit uploads to 4GB when Content-Length is passed. Here are some code changes that may support greater than 4GB (untested). However, since I'm finding the limit at 2GB, this is probably not where the 2GB limit is.
https://github.com/ryanski44/AspNetCore/commit/007379627c2d05ddc822f8d898c2828739cf0823

[ryanski44]

I think I found the actual issue. atol was limiting the max value of content length to 2 GB (2^31). Instead, if atoll is used, the problem is fixed.
https://github.com/ryanski44/AspNetCore/commit/fdec0d797fcefeb7af8278e1e98701442a2bd6e8
I tested it and it worked. Granted I tested with my other changes as well, but I suspect it will work for up to 4 GB by just changing the one line to use atoll instead of atol and casting back down to DWORD.

[sonicmouse]

It sure would be nice to allow unlimited file size uploads. There are cases where we need to upload 8 GB+ archives. Right now we have to use a middle-man solution (upload to blob storage using the blob storage API, then the service downloads the blob via HTTP URI). This adds a lot of overhead/time. It would be nice to be able to upload directly to our service.

Seems most other server software solutions don't have silly upload size limitations. Just IIS.

[shirhatti]

@ryanski44 I'm assuming you tried with in-process hosting only? We almost certainly will have another issue in out-of-process hosting where we don't call WinHttpSendRequest correctly for large uploads.

[ryanski44]

@shirhatti I'm fairly confident that we are using the out of process hosting. The code I changed was in the out of process hosting. I did change the code that was calling WinHttpSendRequest, if you look a few comments up you should see it, but I'll put it here as well.
https://github.com/ryanski44/AspNetCore/commit/007379627c2d05ddc822f8d898c2828739cf0823

Also, I got around to removing the request filtering module from IIS and tried an upload of a 5 GB file, and it worked! I'm testing using a file upload (via multipart/form-data) and I think the browser is passing the content-length header.

[Jcomper]

@shirhatti I'm fairly confident that we are using the out of process hosting. The code I changed was in the out of process hosting. I did change the code that was calling WinHttpSendRequest, if you look a few comments up you should see it, but I'll put it here as well.
ryanski44@0073796

Also, I got around to removing the request filtering module from IIS and tried an upload of a 5 GB file, and it worked! I'm testing using a file upload (via multipart/form-data) and I think the browser is passing the content-length header.

@ryanski44
Could you please share the way you get rid of the request filtering module from IIS? I'm struggling with the same problem.

[Cloudmersive]

@Jcomper Was there any resolution on this?

Hard to believe that it is 2020 and ASP.NET still cannot handle file uploads > 2 GB, there is no documentation of this fact, and this ticket has been sitting open for years.

Even Node.JS which is based on Javascript can handle files uploaded larger than 2 GB.

[alialtansaka]

Another weird problem with 2gb limit w/Kestrel;

when we upload a file under 2GB, the file saved to /tmp directory while uploading
if the file size bigger than 2GB, there is no file in /tmp and server memory increases while uploading then we get "connection reset" error on client side, also NO EXCEPTION or server side.. uploaded file goes to data heaven..