Aspnetcore: Buffering needs to be configurable

[javier-alvarez]

I got this exception because my request was 300MB and I was running in an Azure Cloud Service. Azure Cloud services cannot write to temp storage they need to configure a disk space and write to a specific directory. It was very difficult to find that I had to set a environment variable (ASPNETCORE_TEMP) to configure this. https://github.com/aspnet/HttpAbstractions/blob/07d115400e4f8c7a66ba239f230805f03a14ee3d/src/Microsoft.AspNetCore.Http/Internal/BufferingHelper.cs#L25-L26

Expected: This should be a first class config and documented.

System.IO.IOException: There is not enough space on the disk.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.BeginWriteCore(Byte[] bytes, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
   at System.IO.FileStream.FlushWrite(Boolean calledFromFinalizer)
   at System.IO.FileStream.BeginWriteAsync(Byte[] array, Int32 offset, Int32 numBytes, AsyncCallback userCallback, Object stateObject)
   at System.IO.FileStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.<ReadAsync>d__35.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.<DrainAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Formatters.JsonInputFormatter.<ReadRequestBodyAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.<BindModelAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.<BindModelAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[Tratcher]

Yes this needs to be better integrated with the config system.

I do have to ask why you're trying to buffer a 300MB upload? At a certain point buffering becomes impractical and you need to find an alternative algorithm for processing your data.

[javier-alvarez]

We need this to be able to upload machine learning models that are a big binary blob. This is not a frequent call. Only done by admins of the API. We are not worried about scalability of this call.

[Tratcher]

What happens to it once it's buffered? In memory processing, or copied to permanent storage? Any reason those operations can't be streamed?

[javier-alvarez]

In memory processing, we need to do a few lightweight checks in the API before we save it.

[javier-alvarez]

• Is it possible to disable the buffering completely?
• Is it possible to set the maximum body size per API route?

[Tratcher]

Buffering is off by default. How are you reading the request body? Some of the MVC components may opt to buffer.

Yes, there's a RequestSizeLimitAttribute. But watch out for IIS's conflicting limit.

[javier-alvarez]

If you check in the stack trace JsonInputFormatter is in the stack. We are using the usual model binding.
We are using self hosted HttpSys.

[Tratcher]

HttpSys has the same request body size feature.

You're trying to model bind 300MB of JSON? Turning off model binding seems like a more pressing question than turning off buffering. @rynowak

[javier-alvarez]

Yes we are using NSwag to autogenerate the client. It would be awesome if we could auto-generate the same client but with protobuf instead of json. 300MB and it is compressed.

[RehanSaeed]

I'm trying to enable a read-only file system in Docker (See https://github.com/RehanSaeed/ReadOnlyDockerTest). Buffering to a temp file system is not going to work in that scenario.

Some of the MVC components may opt to buffer.

Which ones exactly? Is Buffering only turned on when EnableRewind on the response body is used?

[Tratcher]

The JSON and XML model binders buffer the request body before reading it as they don't have async parsing APIs.

[RehanSaeed]

Will this be the case with the new JSON API's in .NET Core 3?

[Tratcher]

No, that's one of the main things we're planning to fix.

[RehanSaeed]

Microsoft happened to have blogged about this today:
https://devblogs.microsoft.com/aspnet/re-reading-asp-net-core-request-bodies-with-enablebuffering/