Aspnetcore: ANCM/IIS forces chunking for responses containing a Content-Length header
For obscure reasons, ANCM or IIS seems to force chunking even if the Content-Length header is returned (with the appropriate value):
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/html;charset=UTF-8
Vary: Accept-Encoding
Server: Kestrel
X-Powered-By: ASP.NET
Date: Thu, 05 Jan 2017 20:01:23 GMT
public void Configure(IApplicationBuilder app) {
app.UseDeveloperExceptionPage();
app.Run(context => {
var encoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
var document = @"<!doctype html>
<html>
<body>
<strong>Hello world</strong>
</body>
</html>";
context.Response.ContentLength = encoding.GetByteCount(document);
context.Response.ContentType = "text/html;charset=UTF-8";
return context.Response.WriteAsync(document, encoding, context.RequestAborted);
});
}
When using Kestrel without IIS, chunking is not used (as expected):
HTTP/1.1 200 OK
Date: Thu, 05 Jan 2017 20:06:01 GMT
Content-Length: 79
Content-Type: text/html;charset=UTF-8
Server: Kestrel
@Tratcher is that the intended behavior?
kevinchalet
All 27 comments
No it's not supposed to do that. We'll check it out.
Tratcher
on 6 Jan 2017
I can only reproduce this with IIS dynamic response compression. I see you have the Vary: Accept-Encoding header, but I don't see Content-Encoding: gzip in your example.
It stops happening if I change the content type to something outside the default compression list or disable the compression outright.
<system.webServer>
<urlCompression doStaticCompression="false" doDynamicCompression="false"/>
</system.webServer>
Indeed, this oddity disappears when disabling static/dynamic compression.
Is IIS supposed to switch to chunking even if it doesn't compress the response?
I see you have the Vary: Accept-Encoding header, but I don't see Content-Encoding: gzip in your example.
Yup, both Chrome and Fiddler confirm that the output is not compressed at all.
kevinchalet
on 7 Jan 2017
That's bizarre, no idea why it's forcing chunking for you even without compression. At least it demonstrates the issue is unrelated to ANCM.
The other test you can do is to use an unknown/unknown content-type.
Tratcher
on 9 Jan 2017
The other test you can do is to use an
unknown/unknowncontent-type.
That works too:
HTTP/1.1 200 OK
Content-Length: 79
Content-Type: unknown/unknown
Server: Kestrel
X-Powered-By: ASP.NET
Date: Mon, 09 Jan 2017 21:29:49 GMT
I don't have much time to investigate ATM, but I'll try to find an hour or two this week to see if I can reproduce this bug with older IIS/IIS Express versions (on my main Windows 7 dev' machine, I use IIS Express 10, maybe it's a recent change?).
kevinchalet
on 9 Jan 2017
could you please share the request headers and a repro?
pan-wang
on 10 Jan 2017
Here's the repro:
{
"dependencies": {
"Microsoft.AspNetCore.Diagnostics": "1.1.0",
"Microsoft.AspNetCore.Server.IISIntegration": "1.1.0",
"Microsoft.AspNetCore.Server.Kestrel": "1.1.0",
"Microsoft.Extensions.Logging.Debug": "1.1.0",
"Microsoft.NETCore.App": { "version": "1.1.0", "type": "platform" }
},
"frameworks": {
"netcoreapp1.1": { }
},
"buildOptions": {
"emitEntryPoint": true,
"preserveCompilationContext": true
},
"publishOptions": {
"include": [
"wwwroot",
"web.config"
]
}
}
using System.IO;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Logging;
namespace ChunkingBug {
public class Program {
public static void Main(string[] args) {
var host = new WebHostBuilder()
.ConfigureLogging(options => options.AddDebug())
.UseKestrel()
.UseContentRoot(Directory.GetCurrentDirectory())
.UseIISIntegration()
.UseStartup<Startup>()
.Build();
host.Run();
}
}
}
using System.Text;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
namespace ChunkingBug {
public class Startup {
public void Configure(IApplicationBuilder app) {
app.Run(context => {
var encoding = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false);
var document = @"<!doctype html>
<html>
<body>
<strong>Hello world</strong>
</body>
</html>";
context.Response.ContentLength = encoding.GetByteCount(document);
context.Response.ContentType = "text/html;charset=UTF-8";
return context.Response.WriteAsync(document, encoding, context.RequestAborted);
});
}
}
}
... and the headers:
GET http://localhost:21204/ HTTP/1.1
Host: localhost:21204
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
definitely something related to the dynamic compression module. the same behavior occurs for static files not handled by ANCM at all.
albigi
on 13 Dec 2017
@bangbingsyb Will the new IIS compression module mitigate this?
@Tratcher Do you recall any other mitigation we may have?
cc @shirhatti
muratg
on 20 Dec 2017
@muratg It looks like a bug in the compression modules or IIS core. Unfortunately, the new compression schemes we plan to ship won't fix this problem, because they are only extensions to the existing compression modules, and simply provide the actual implementation of IIS HTTP Compression APIs: https://msdn.microsoft.com/en-us/library/dd692872(v=vs.90).aspx
However, I will look into this issue and will try to fix it. The current theory is that IIS core strips any content-length set by the app - simply because HTTP.sys will perform the final calculation. However, stripping this header might create the illusion that the response data has been chunked to the compression modules.
bangbingsyb
on 21 Dec 2017
@bangbingsyb, see the below FREBs attached.
it looks like the Content-Length header is properly set by the request handler and then when the SEND_RESPONSE notification starts it gets lost in the way.
Oddly, I don't see it being set by the dynamic compression module, but it is still appended to the general response headers prior flushing (I assume by HTTP.sys)
hope this helps
albigi
on 21 Dec 2017
@albigi Thanks for the FREBs.
Dynamic Compression module indeed removes the Content-Length header on RQ_SEND_RESPONSE. The reason you don't see such event in FREB is because it directly resets the known header array of the HTTP.sys native request (HTTP_REQUEST structure) rather than calling IHttpResponse::DeleteHeader. I guess the motivation is performance optimization. Check HTTP_COMPRESSION::SetupDynamicCompression from the source.
This is an expected behavior because the compression will obviously change the content size. After compression, the module will not add back the updated Content-Length header simply because HTTP.sys will finally calculate it for us.
What is unclear to me is somehow IIS determines that the final compressed contents are sent in chunked mode. Dynamic Compression module is not supposed to alter Transfer-Encoding. In other words, if the raw response data is chunked, dynamic compression module will keep the compressed data as chunked, and verse visa. But I suspect some operation triggers IIS to alter the mode. Will need to setup a live repro to understand the root cause.
bangbingsyb
on 22 Dec 2017
Why would Http.Sys add a content-length header for you? That's not one of its features.
Tratcher
on 22 Dec 2017
@Tratcher HTTP.sys does set CL on-the-fly in certain scenario, and the behavior is correlated with how IIS calls HttpSendHttpResponse (HTTP_SEND_RESPONSE_FLAG_MORE_DATA) and whether IIS supplies CL. In fact, we've seen bugs that two different CLs are both added for range request.
So to be more accurate, my understanding is that it is OK for the compression module removing CL and not adding back the new CL. It will be the responsibility of either IIS core or HTTP.sys to add it. The problem is IIS gets the compressed response body and makes the decision to chunk the data, and the key question is why?
bangbingsyb
on 22 Dec 2017
Nothing can re-add it unless the response is fully buffered. Is buffering enabled but with a limit?
I've seen that range bug when using kernel mode cache.
Tratcher
on 22 Dec 2017
@albigi you mentioned the issue even happens with static file when compression is enabled. Could you provide the repro steps?
bangbingsyb
on 22 Dec 2017
@bangbingsyb actually I was wrong. The test I did was browsing a plain .html page with dynamicCompression enabled, but the request was handled by PHP instead of the SFHandler. So what I found out now is the chunked transfer encoding header seems to be added when either CGI or Fast-CGI is used to read the response buffer. I can repro with static files as long as a CGI handler is called (could be PHP or ANCM).
albigi
on 27 Dec 2017
@albigi Is it possible to share with me the repro app and configuration? I did quite massive code review during the holiday week, and figured out the root cause. But I need some simply repro app so that once I purpose the fix, I can validate it.
bangbingsyb
on 1 Jan 2018
@bangbingsyb, please see attached. You should be able to repro on Blue and RS2/3 with the dynamic compression module installed. I can share the full source code but it is essentially just a plain asp.net core app with the static file support
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
app.UseBrowserLink();
}
else
{
app.UseExceptionHandler("/Error");
}
app.UseStaticFiles();
}
then I just browsed to /app/Index.html
albigi
on 3 Jan 2018
I too have problems with this.
I have narrowed it down to model data containing the quote character " (and possible others). E.g. I have a field called Title and when it contains the quote character, ASP.NET Core dynamic compression will return chunked encoding.
Using ASP.NET Core 2.2.3 - in-proc.
If I replace the quote character " with " then there is no problem.
My razor code has something like this:
Works - correct encoding returned
Model.Title = Model.Title.Header.Normalize(System.Text.NormalizationForm.FormC).Replace("\"",""");
Fails - chunked encoding
Model.Title = Model.Title.Header.Normalize(System.Text.NormalizationForm.FormC);
The Normalization does not matter in my case - it seems to be return chunked no matter with above.
shapeh
on 5 Apr 2019
@anurse let's move this to the ASP.NET Core repo.
jkotalik
on 5 Apr 2019
Leaving this open in the backlog to solicit some feedback. The out-of-proc handler is fairly limited here in what it can do to improve upon this. If people have scenarios with the in-proc handler where chunked encoding is used unexpectedly, it would be good to hear about those.
anurse
on 10 Apr 2019
@anurse - I am running in-proc with my above comment.
shapeh
on 14 Apr 2019
Bump. Any progress on this one?
shapeh
on 17 Jul 2019
@shapeh The earlier conversations indicated IIS Dynamic Compression was in play. Is that the situation in your case? If not, perhaps https://github.com/aspnet/AspNetCore/issues/10773 is more appropriate.
@jkotalik @Tratcher it seems like this thread is actually dealing with an external issue (encoding changes in IIS Dynamic Compression). Is that a correct assessment from your understanding? Perhaps we should close this and move discussion of other issues (such as possibly @shapeh 's issue) to #10773 ?
anurse
on 17 Jul 2019
@anurse - I have just checked / problem happens no matter if Dynamic compression is on or off in IIS.
shapeh
on 17 Jul 2019
Ok, let's move that discussion to #10773. Can you isolate a runnable sample that reproduces the problem and post it on that thread?
anurse
on 17 Jul 2019
Related issues
githubgitgit
路
3Comments
Pixel-Lord
路
3Comments
UweKeim
路
3Comments
rbanks54
路
3Comments
mj1856
路
3Comments