Runtime: [release/2.1] Expect100Continue with Windows auth and request body > 1024 bytes fails

[mconnew]

[davidsh]

cc: @stephentoub @geoffkizer

[karelz]

@mconnew mentioned to me offline that it is a regression 2.0->2.1
What is the impact on WCF scenarios/customers? How common are the scenarios?
Are there any workarounds on WCF side or on customer side?
(just trying to understand the scope & impact)

[davidsh]

@mconnew mentioned to me offline that it is a regression 2.0->2.1

It could only be a regression on Linux from using CurlHandler on .NET Core 2.0 (which supported Expect: 100-continue) to using SocketsHttpHandler on .NET Core 2.1.

On Windows, with .NET Core 2.0 (WinHttpHandler), Expect: 100-continue is not supported.

[mconnew]

@davidsh, while Expect: 100-Continue is not supported by WinHTTP in actually implementing the functionality to hold off sending the request body, if you add the Expect header, it will still functionally be able to send the request and get the response when using authentication. Now on Windows, that is no longer the case. So while the purpose of the header couldn't be used, things were still functional if you did use it. So this is a regression on Windows in that you could get a response before, you can't now.

[geoffkizer]

@mconnew, I'm curious: Why are you setting Expect: 100-continue in this scenario?

[mconnew]

Without it, the entire request body will be sent for the initial request and for the challenge response. If the request message is for example 10MB, that's double the data you are then sending and double the amount of time it takes to get a response. Additionally, according to the customer who saw this issue, in their testing setting Connection: Close header without Expect: 100-Continue also has the same problem.

[karelz]

@mconnew can you please answer the questions above for us to prioritize it properly?

• What is the impact on WCF scenarios/customers? How common are the scenarios?
• Are there any workarounds on WCF side or on customer side?

[mconnew]

Impact
Anybody using Windows authentication with requests > 1024 bytes. I don't know how common this is as we don't have any telemetry around which authentication method is used.

Work around
Revert to earlier release of WCF which didn't use the Expect header. In an earlier release we sent a HEAD request before every real request as an attempted mitigation to the problems caused by lack of universal support for the Expect header. So basically roll back. The prior approach with the HEAD request was only best effort and would still sometimes have problems as well as introduced new problems.

[karelz]

I don't know how common this is as we don't have any telemetry around which authentication method is used.

If you have customers complaining, let's use that as indicator. How many impacted so far? Is there public issue where it happens or is it internal via support?
How badly is the specific customer impacted? (RSP/SLA/other metric) Or was it just something that was noticed during testing?

Regarding workaround: Does it mean "use older WCF package"? Is that even supported scenario?

[mconnew]

The issue where this is reported is dotnet/wcf#2923 and there's 3 customers there who have reported it. I would expect any new instances of customers having the problem would see that issue and use the workaround.
Using the older WCF package is completely supported. We aren't a part of .Net Core, we're more like a third party library which also happens to have an implementation in the full framework when your runtime is netfx. That means all earlier WCF packages can run on later versions of .NET Core. In addition to that, as we target .NET standard and not .NET Core and .NET Standard is still at 2.0, as I made sure we didn't add any usage of classes added to .NET Core 2.1 someone can use our latest package on .NET Core 2.0 and they wouldn't get any runtime type errors. They might get functional problems if there's missing capabilities in e.g. HttpClientHandler, but it should generally run fine both ways.

[mconnew]

Btw, 3 reported issues in a short period of time is quite high for WCF.

[danmosemsft]

@karelz if this is to go into 2.1.x it will need a mail to corefx shiproom. Do you plan to do that?

[danmosemsft]

Removing label until this is ready for big shiproom.

[geoffkizer]

Additionally, according to the customer who saw this issue, in their testing setting Connection: Close header without Expect: 100-Continue also has the same problem.

Do you expect this (i.e. setting Connection: close and doing NT auth) to work?

[geoffkizer]

@mconnew, I'm not entirely sure what you want us to do in this scenario. When we get an early response with Expect: 100-continue, we either need to (a) finish sending the request body, so we can keep the connection alive, or (b) close the connection and stop sending the request body.

[mconnew]

@geoffkizer, the current solution means that you can't use use NTLM authentication with the Expect header. It simply can't work as it abandons the request and throws an HttpRequestException at the caller. I ran some experiments and recorded what HttpWebRequest does on the .Net Framework so I'll give a quick overview here.

1. --> Client sends headers including an Expect: 100-continue header
2. <-- Server responds with 401 status and WWW-Authenticate: NTLM header (no blob)
3. --> Client sends the request body (to keep socket open)
4. --> Client sends request headers including Authorization header with initial NTLM authorization blob, no expect header and Content-Length: 0 header (no expect header because no request body).
5. <-- Server responds with a 401 status and a WWW-Authenticate header with NTLM response blob
6. --> Client sends request headers including Authorization header with final NTLM authorization blob, the correct Content-Length header and an Expect: 100-continue header
7. <-- Server responds with a 100 status
8. --> Client sends request body
9. <-- Server responds with final response status, headers and body

This is the minimum that should be done. I believe that this can be optimized further. I suspect in step 3 the client could abandon the socket as there's no state information about the authentication that has been transmitted yet. This is the general idea that the current code attempts to do. But the current implementation then throws an HttpRequestException at the caller and there's no way to complete the request. Instead it should close the connection, not throw and create a new connection instead. Although the current limit of 1024 is too small as that is less than a single ethernet frame payload. Ideally the RTT would be queried from the OS and a heuristic used to work out which approach would be quicker but the System.Net.Socket abstraction is too far removed from the OS to query this information.

Additionally, when PreAuthenticate has been enabled and a credentials cache is used which only contains NTLM credentials, the first request can probably be skipped and can jump straight to step 4 as the server doesn't provide any challenge data in the initial authentication challenge. That would avoid sending the request body completely until after the authentication is completed. There is a risk that some servers could initialize some state when replying with the empty NTLM challenge, but that's easily tested and could be turned off by not using setting PreAuthenticate to true if some implementation failed with this optimization.

[danmosemsft]

@karelz @caesar1995 should this still be 2.1.x? Is it being actively worked on?

[karelz]

Sorry for delayed updates. It is still on our backlog, we hope to get back to it soon.

[mconnew]

WCF has another customer hitting this issue dotnet/wcf#3185

[mconnew]

@karelz, do you have any idea of timeline to fix this issue? The cause of this bug is well understood, the behavior on .NET Framework is understood as I investigated and documented what that behavior is. This bug is now over 6 months old and is blocking multiple customers and there's been no updates for 3 months.

[caesar-chen]

[mconnew]

[caesar-chen]

[mconnew]

[TPucke]

[JWH89]

[stofte]

[davidsh]

[davidsh]

[karelz]