Aspnetcore: Response headers say content is chunked, when it's not

[jamers99]

Describe the bug

We have an OData service that is returning a response and it says that it is chunked ("Transfer-Encoding": "chunked") when the content is not actually chunked. We worked around this by adding hacky Middleware to IApplicationBuilder:

    /// <summary>
    /// Gets around a .netcore bug that seems to mark everything as chunked when it really isn't
    /// </summary>
    /// <remarks>
    /// This class is taken from https://stackoverflow.com/questions/37966039/disable-chunking-in-asp-net-core/54355016#54355016 - click the link and upvote the answer if you see this, this person deserves it :)
    /// </remarks>
    public class ResponseDeChunker
    {
        readonly RequestDelegate next;

        public ResponseDeChunker(RequestDelegate next)
        {
            this.next = next;
        }

        public async Task InvokeAsync(HttpContext context)
        {
            var originalBodyStream = context.Response.Body;
            using (var responseBody = new MemoryStream())
            {
                context.Response.Body = responseBody;
                long length = 0;
                context.Response.OnStarting(() =>
                {
                    context.Response.Headers.ContentLength = length;
                    return Task.CompletedTask;
                });
                await next(context);
                length = context.Response.Body.Length;
                context.Response.Body.Seek(0, SeekOrigin.Begin);
                await responseBody.CopyToAsync(originalBodyStream);
            }
        }
    }

To Reproduce

We have a large project that this is happening in, we have not attempted to reproduce it in a smaller project. If required, we may be able to try that.

Further technical details

• IDE: Visual Studio 2019 (16.6.3)
• ASP.NET Core version: 2.2.5
• dotnet --info:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.301
 Commit:    7feb845744

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19041
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.301\

Host (useful for support):
  Version: 3.1.5
  Commit:  65cd789777

.NET Core SDKs installed:
  3.1.301 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[jamers99]

This seems very similar to this very old (2013) .NET Framework issue on StackOverflow:
https://stackoverflow.com/questions/16736233/web-api-as-a-proxy-and-chunked-transfer-encoding

[Tratcher]

Can you show how you're generating the original response?

All HTTP/1.1 responses must set Content-Length or Transfer-Encoding: chunked. ASP.NET Core defaults to chunked if you do not set the content-length. This allows responses to be sent immediately without buffering and helps minimize latency, memory overhead, etc..

Why is chunking a problem for you?

[jamers99]

We have a relay service that forwards requests and responses to and from a server. We copy headers, which is causing some problems. This was working in .NET FW, but when we moved to .NET Core, it broke. @brentesh knows this better, perhaps he can elaborate.

Is there a way to easily turn chunking off from the server sending the responses?

[Tratcher]

When copying headers in a proxy/relay you should filter out the Transfer-Encoding: chunked header, most servers will re-add it if needed.

Is there a way to easily turn chunking off from the server sending the responses?

No. Chunking can only be disabled by setting the Content-Length header, which often requires buffering to calculate.

ASP.NET 4 (System.Web) buffered responses by default which caused performance issues.

[Tratcher]

FYI the above code sample should not be using OnStarting to set the Content-Length, it should be set here instead:

                await next(context);
                context.Response.Headers.ContentLength = context.Response.Body.Length;

[jamers99]

Ah, okay. So we would need to add the Content-Length in our proxy/relay by reading the response? Obviously would read the entire response which could introduce a performance issue, but it was working okay before.

[Tratcher]

I didn't say to add the Content-Length header, I said to filter out the Transfer-Encoding header. ASP.NET servers have a curious feature where if the app sets the Transfer-Encoding then the server assumes the app applied the chunked framing itself. This tends to break relays because the HttpClient removes the chunked framing but not the chunked header.

YARP is a proxy I work on and this is how it deals with the response Transfer-Encoding header, it removes the header and lets the server re-apply it:
https://github.com/microsoft/reverse-proxy/blob/43929dc9894ad3b986b61fe9966610297f0232e2/src/ReverseProxy/Service/Proxy/HttpProxy.cs#L586-L594

[brentesh]

Hey, Brent Esh here, I'm working with James on this project. I implemented this and it seems to have taken care of the issue. Thanks!

[brentesh]

Do the relays not worked with chunked responses at all?

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.