Runtime: HttpClientHandler needs Expect: 100-Continue support to use client certificates fully

[mconnew]

Without support for this header, some scenarios are completely broken with no workaround. If you are sending a request to a server over HTTPS which doesn't immediately require a client certificate on the initial handshake and the request entity body is non-trivial in size then the request will always fail. This is because when the server sends the SSL renegotiation request the client has already sent too much data for the server to buffer before it can receive the SSL renegotiation response.
If you support HTTP and HTTPS on the same site in IIS, then you can't set certificates to be required, only accepted. When communicating with IIS/HTTP.SYS, 16KB is the amount of data that the client needs to send so that the connection can't be established.
The Expect: 100-Continue header works around this issue by causing the client to hold off sending the entity body until the server sends a "100 Continue" status response. This allows the server to do the SSL renegotiation before it sends the "100 Continue" response and the connection/request succeeds.
Without full support for this header, client certificates with an entity body > 16KB are going to be a fragile use case depending on server config and client/server latency.

[davidsh]

cc: @CIPop @himadrisarkar @stephentoub

[davidsh]

Currently, this is impossible to implement in the .NET layer for Windows (either non-UWP or UWP) since HttpClient builds on the Windows native HTTP stacks (WinHTTP, WinINET) and those stacks do not support Expect: 100-Continue semantics.

I think Curl on *Nix supports it at that layer. However, the architecture of HttpClient as written would need to be revised to take advantage of such support.

[mconnew]

HttpClient using WebRequestHandler on the full framework supports Expect: 100-Continue just fine with the current architecture.

[vadimline]

Would be nice to have this. Having problem with Azure API.

On Windows you will have next exception "The connection with the server was terminated abnormally".
On Linux(Windows Ubuntu subsystem) works fine.

Workaround(At least it works with azure api): Send empty request, and next request with data will work fine.

[CIPop]

Without support for this header, some scenarios are completely broken with no workaround. If you are sending a request to a server over HTTPS which doesn't immediately require a client certificate on the initial handshake and the request entity body is non-trivial in size then the request will always fail. This is because when the server sends the SSL renegotiation request the client has already sent too much data for the server to buffer before it can receive the SSL renegotiation response.

There are instances (HTTP Load Balancers, Proxies) where 100-continue is not going to be properly handled and so the usage is discouraged.
I am contacting WinHTTP regarding this scenario.

@vadimline please add more specific details about the remote service, API involved and, if possible, sample code.

[vadimline]

@CIPop Sure
Azure Cloud Service API (Service Management), the problem occur when I'm trying to publish cloud service, so I'm doint POST request that contains all information for cloud service.
Since my last post I found out that you can send not just empty request to same API but any request, for instance I'm doing GET request to get cloud info and then I'm doing POST request to publish service.

In attachment you can find sample project. You will need to have azure account, azure management certificate, and cloud service with some test project. Right now if you will provide that info you should face that problem I had "The connection with the server was terminated abnormally", it might take few minutes before you will receive this exception.

Then you can go to ComputeClient.cs and remove comment from line 45 and run again, this time everything should be fine.

I'm not quite remember right now, but I think there is a matter of how big your request, so if you will not see original problem try to use bigger cloud service configuration file (just add some dummy configs with long values) so it in total will be bigger then 60Kb for instance.

AzureCoreHttpClient.zip

[CIPop]

Thanks @vadimline!

[CIPop]

I've contacted both WinHTTP and Azure Service Management API.
WinHTTP doesn't have 100-Continue support and they do not recommend the usage. I got no answer yet from Azure Service Management.

[CIPop]

We have discussed this with the WinHTTP team and they are considering supporting 100-Continue in a future version of Windows.

We are still discussing the problem with the Azure Cloud Service/Service Management team.

[CIPop]

Here is the WinHTTP team's answer:

the 100 Continue mechanism was a way to avoid wasting bytes with large bodies that weren’t going to be processed. It wasn’t intended to function as a critical part of the authentication mechanism. There’s no client-server negotiation for the client to know whether the server supports 100 Continue. That means that all clients have implemented this feature with a delay rather than waiting indefinitely.

If you have a hard dependency on 100 Continue, then you’re vulnerable to deadlocks when the client exceeds that timeout as well. Getting 100 Continue support helps guard against that when latency is low, but it’s not a full solution. The right solution would be to use a small request (GET, OPTIONS, etc.) to trigger the certificate authentication first, or simply configuring a separate binding that requests the certificate up-front.

This sounds exactly like the workaround proposed by @vadimline.

I'm closing this for now:

1. We have no way of implementing this using WinHTTP at this point (although they are considering supporting it in a future version of Windows).
2. We have already raised the issue with ACS/Service Management and they are investigating on the server side.

[mconnew]

If you have two sites listening on https://contoso.com/foo and https://contoso.com/bar and /bar requires a client certificate, the server can't ask for the certificate until it has received the request headers. If the request to /bar is a POST and the server is hosted on Windows (HTTP.SYS based so includes IIS) and the entity body is > 48K, then the request will fail. The suggestion to make a small request first might not work for 2 reasons.
First, methods other than POST might not be routed to the site code which asks for the certificate in the /bar site. Modifying the server routing to accept other verbs isn't the right solution to a client deficiency. Someone writing the client might not have control of the server, and that's kind of the point of standards such as the HTTP protocol, the client and server don't need to be controlled by the same entity.
Second, making a small request first doesn't work when using connection pooling.
Thread1 sends a GET to /bar. WinHTTP creates a new socket A and sends the request.
The server asks for the client cert for the request to /bar
WinHTTP sends the client cert
The server sends the response to WinHTTP which delivers it to the application code.
WinHTTP puts socket A in it's connection pool.
Thread 2 sends a GET to /bar. WinHTTP takes socket A from the connection pool and sends the request.
Thread 1 sends a 1MB POST to /bar. WinHTTP creates a new socket B and sends the request including the POST body.
The server reads the request on socket B to /bar and sends the request to renegotiate the SSL connection and asks for the client certificate.
WinHTTP receives the request for the client certificate on socket B and sends it.
The server reads 48KB from socket B looking for the client certificate but there's 1MB of data ahead of the client certificate response.

The POST on Thread 1 can never be completed. The technique of sending a small request first only works if you will only ever have one request in flight at a time. If you are in a scenario where you will send more than one request at a time, you will have failures. This means WinHTTP isn't suitable for mid-tier scenarios when communicating with a server which doesn't request the certificate up front.
The only way to configure a separate binding that requests the certificate up front would be to run on a different port or ip address. Is the recommended solution to a feature missing in the client side stack to make significant infrastructure changes on the server? I don't believe that is a suitable answer as I mentioned earlier, you might not control the server side.

I also opened the bug as an issue with HttpClientHandler. How this is implemented internally is irrelevant to a consumer of HttpClientHandler. If WinHTTP isn't capable of fully supporting the HTTP protocol, then don't use WinHTTP as the basis for HttpClientHandler. If you stick with WinHTTP, is the solution for others to switch to something different than HttpClientHandler to be able to get a functioning HTTP stack?

[CIPop]

/cc @DavidGoll @karelz

@mconnew My current understanding (based on WinHTTP's response) is that using 100 Continue can lead to deadlocks and is not the recommended solution regardless of what HTTP stack is being used.
My suggestion is to follow-up directly with the WinHTTP team regarding your concerns.

If you stick with WinHTTP, is the solution for others to switch to something different than HttpClientHandler to be able to get a functioning HTTP stack?

Creating your own handler based on a different HTTP stack is very simple when working with HttpClient. On the other hand, implementing a HTTP stack is very complex.
At the moment the WinHTTP based handler is the only supported HTTP stack for Windows in CoreFX (for non-UWP applications).

[mconnew]

Using 100 Continue won't lead to deadlocks if rfc7231 is implemented correctly. It says:

...since 100 (Continue) responses cannot be sent through an HTTP/1.0 intermediary, such a client SHOULD NOT wait for an indefinite period before sending the message body.

The only way a deadlock can happen is if the client is implementing 100-continue incorrectly. In the case where the timeout occurs, the server will have had opportunity to parse the headers and request a client cert if necessary. If the server doesn't support 100-continue, then you probably wouldn't be requesting it. We're talking about programmatic requests using HttpClientHandler which is used to make requests to a specific server and usually an api of some sort and not something like a web browser which will be connecting to arbitrary http servers. If I am using a service which requires a client certificate and POSTs data, I am specifically writing code to do so and will use 100-Continue very purposefully. I'm not suggesting 100-continue should be used by default, but if I need it, I should be able to enable it.

It comes down to this, if a server request would fail when using 100-Continue, it can simply not be used. If a server request would fail without using 100-Continue (and I've outlined scenarios where this is the case) and there is no implementation, there's nothing that can be done and you simply cannot communicate with that server.

You have focused on Windows, but libcurl supports 100-Continue. Having support on unix platforms would be better than where we are today. On Windows, people can use the full .Net framework to get 100-Continue support. Could this bug be fixed on unix so that we at least have a solution for other platforms?

[davidsh]

I'm not suggesting 100-continue should be used by default, but if I need it, I should be able to enable it.

As we know from .NET Framework experience, many servers did not support 100-continue and had problems when the 'Expect: 100-continue' was sent as a request header. Unfortunately, .NET Framework ServicePointManager class has the Expect100Continue property default value as true. This led to many problems for developers and many of them added ServicePointManager.Expect100Continue=false to their code. In hindsight, .NET Framework should have had the default for that property to be 'false'.

[stephentoub]

@mconnew, what is the bug on Unix?

[mconnew]

@stephentoub, if I remember a previous discussion it's on by default on unix. The way to not use it is to not provide the header, but you can't add a negative header to HttpHeaders 😄 while means it can't be turned off on unix. So while the header is always emitted on unix, I wouldn't say the feature is "supported" in that it can be controlled.

[mconnew]

@davidsh, the fact that the default is incorrect on the full framework doesn't change the fact that it should be supported when needed.

[davidsh]

@davidsh, the fact that the default is incorrect on the full framework doesn't change the fact that it should be supported when needed.

I don't think I ever said that the default is "incorrect on the full framework". I simply commented that on .NET Framework, the default is TRUE. And for many years, developers had problems with that because servers at the time didn't support it correctly. And there were many articles written about how to set the default to FALSE.

[stephentoub]

means it can't be turned off on unix

Sure it can, just set requestMessage.Headers.ExpectContinue = false;:
https://github.com/dotnet/corefx/blob/fdb309f1421d4006c609097fd4b48443d3b1ad18/src/System.Net.Http/src/System/Net/Http/Unix/CurlHandler.EasyRequest.cs#L734-L740

[stephentoub]

SocketsHttpHandler is the default in .NET Core 2.1 and supports Expect: 100-continue. While there are known issues in a few cases, those are tracked by separate open issues, so I'm closing this one. Thanks.