Wcf: Unnecessary HTTP HEAD method request

[zipswich]

Services generated from WSDL files send out an HTTP HEAD request before sending a SOAP message every time. This is not the case for WinRT apps, but it causes problems with UWP apps. Though the HEAD method is legitimate, I do not see any use of it. Most servers can handle it without any problems, but some not. Unfortunately, devices from one of the most popular brands that we deal with cannot handle it. It would be nice to remove it or allow the app to decide whether to use it.

[shmao]

/cc: @mconnew

[shmao]

@mconnew , can you please link the issue to the NCL issue?

[zipswich]

Does anyone know the ETA of the fix?

[mconnew]

The problem is that HttpClient doesn't support the Expect: 100-Continue header on non-linux platforms. This is necessary for multiple reasons. For example, to avoid sending a large request to a server to be greeted with an authentication challenge. The entire request would need to be sent again when responding to the challenge. Some authentication protocols require multiple requests back and forth (such as NTLM) and this could result in a very large amount of data being sent. Another scenario is on a high latency high bandwidth connection such as can exist when your data network is cellular, you can have problems with client certificates being used and can result in the server side being completely unable to serve requests. A partial mitigation is to send a priming HEAD request first which sorts out the authentication without sending the POST body first. This isn't a full mitigation as there is a potential for two clients to race for a connection in the underlying connection pool, it's a best effort only. The issue in corefx tracking this is dotnet/corefx#8673 which I opened 20 months ago.
You didn't see this with WinRT as that actually used the full desktop .NET Framework for applications which uses HttpWebRequest which does fully support the Expect header. On UAP, HttpClientHandler wraps the native UAP HttpClient which is built on top of WinINet which doesn't support the Expect header.
I'm going to try and implement issue #2400 for the next release which would enable you to put in an intercepting handler which no-ops the HEAD requests. We can't generally remove the extra HEAD request as that would break more people than it would enable.

[zipswich]

Thanks a lot for the elaboration, @mconnew
Intercepting handler would work. I am already using a message inspector.
Would it be the quickest fix to have an option to disable HEAD?
I have a rudimentary question about this WCF project. Is it possible for me to build a version for my own app to make many app users happy for now while waiting for the final fix? I have red the following:
https://github.com/dotnet/wcf/blob/master/Documentation/developer-guide.md
It seems possible.

[mconnew]

You can certainly build your own version, but I'm not sure how that will work when submitting to the store. When you submit an app to the app store, your app is compiled to native on your behalf. It's possible that your own built binary might be ignored in preference of the official binary. We also have some injection logic in the tool chain to make sure the pre-generated serializers are available to WCF. I can't tell you for certain that your own built binary would get everything wired up correctly. So you can certainly try it, but I would recommend trying it with a test app with restricted visibility.
I was just double checking the code to tell you where to change things and I had a thought and there might be a work around you can use. The only time we send the HEAD request is when authentication is required. What client credential type are you using? It might be possible to add the authentication header yourself and let WCF think no authentication is required so skipping the HEAD request. The relevant method is AuthenticationSchemeMayRequireResend and you modify it to always return false.

[zipswich]

@mconnew Your elucidation is very enlightening, and I am a bit excited now.
First, building my own version should be out of the question. I had exactly the problem you described. Actually, I was using a preview version of VS, not my own build of library, but the store did not like what was working perfectly with the VS. Even after the official VS was released, it took some time for the store to catch up (long story).

Let me tell you some background about the app. I am abbreviating it from a book-long to a few sentences. The first version was released in December 2011 as a WP7 app, then WP7.1, WP8.0, WP8.1, WinRT for Windows 8.1, and finally UWP for Windows 10 last year. The list of issues with the proxy classes generated by the VS WSDL tool is very long.

Here is a highly relevant part: I had to create a custom IEndpointBehavior with a custom IClientMessageInspector to deal with some issues. The custom IClientMessageInspector handles authentication which is digest authentication mandated by an international standard. They have persisted till today.

You hinted workaround, if I understand it correctly, seems to be already 90% done by me. I may just need some further hints to finish it, then hallelujah!!!

[mconnew]

So I just want to verify, it sounds like you have implemented Digest authentication yourself via a message inspector? If that's the case, then you just need to tell WCF that you are using no client credentials. I presume you are using BasicHttpsBinding, in which case you would do something similar to the following:

var binding = new BasicHttpsBinding();
binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.None;

I've read through the code and it looks correct that setting ClientCredentialType to None should set HttpsTransportBindingElement.AuthenticationScheme to System.Net.AuthenticationSchemes.Anonymous. If for some reason it doesn't, please let me know (I don't think we have an explicit test for this but it should work) but you can still do it by creating a custom binding and modifying the HttpsTransportBindingElement directly. This will mean the AuthenticationSchemeMayRequireResend will return false and the HEAD request won't be sent.
If you haven't implemented your own digest authentication implementation, I can point you to a managed implementation in corefx.

[zipswich]

Thank you, @mconnew I think we are really close.
Yes, I have implemented the digest authentication. I did it more than 5 years ago, and it is still working.
I use CustomBinding, not BasicHttpsBinding. I have been using the following code:

    var binding = new CustomBinding()
            {
                CloseTimeout = TimeSpan.FromSeconds(5),
                OpenTimeout = TimeSpan.FromSeconds(5),
                SendTimeout = TimeSpan.FromSeconds(5),
            };
...
   binding.Elements.Add(new HttpTransportBindingElement() { AuthenticationScheme = System.Net.AuthenticationSchemes.Digest, MaxReceivedMessageSize = 65536 * 2 });  //65536 is the default size

Now, I am reading the code that I wrote more than 5 years ago.

[zipswich]

@mconnew The authentication header is injected at method BeforeSendRequest() of custom IClientMessageInspector. OnWriteHeaderContents() of a custom MessageHeader is where my authentication header is written.

[mconnew]

So your digest authentication is actually sent via a SOAP message header and you aren't using HTTP authentication? If that's so, then I would expect changing the final line of your code snippet to this would work:

   binding.Elements.Add(new HttpTransportBindingElement() { AuthenticationScheme = System.Net.AuthenticationSchemes.Anonymous, MaxReceivedMessageSize = 65536 * 2 });  //65536 is the default size

[zipswich]

Hallelujah!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your one property fix - Digest to Anonymous - works instantly, @mconnew. Thank you so much. Many app users will be thrilled to have their devices work again after upgrading the app from WinRT to UWP.

BTW, I have to mention something else. You mentioned Expect: 100-Continue earlier. It gave us headache with WinRT version because a major brand's devices (Samsung) do not support it, the app could not work with their devices until the UWP version which got rid of Expect: 100-Continue. Hope this problem will not recur. If the future version has to have this, please have a way to get rid of it.

The HTTP/HTTPS implementation in the real world is ugly. One can only assume an HTTP/HTTPS server accepts only a few basic methods (e.g. GET, POST, ...), everything else must be assumed optional even the mature international standards mandate them. We are dealing with real lives, not doing academic exercises. I have not had any luck in asking vendors to fix their firmware to comply with protocols.

[mconnew]

@zipswich, I'll make sure there's a way to suppress the Expect header if we ever get the ability to add it on UWP. At the very least, with the ability to wrap the HttpClientHandler that we're going to add, there will be an extensibility point which will allow you to suppress it but I would hope to be able to find an easier/cleaner way to do so. At the very least I'll make it so that it's not used when the AuthenticationScheme is Anonymous..

[zipswich]

Thank you so much, @mconnew It is greatly appreciated.

You can imagine the uproar of Samsung device users would have if the app suddenly stopped working with their devices.

[daver77]

@mconnew did this ever get resolved? We're seeing the same thing, an extra request containing HEAD only that returns a 500 from the service when called using .NET Core. Same/similar code in .NET Framework doesn't do it

[mconnew]

@daver77, if you update to the latest version of WCF you won't get the extra HEAD request. .NET Core 2.1 introduced a new managed implementation of HttpClientHandler (internally wraps SocketsHttpHandler) which support the Expect header. There is still an outstanding bug in the implementation when using NTLM authentication which causes requests over 1024 bytes to fail.

[daver77]

@mconnew thank you! We were still on .NET Standard 1.6 with the various ServiceModel packages on 4.3. Updated the packages to 4.5.3 and code to .NET Core 2.1 and all good. I couldn't get it to compile on .NET Standard 2.0, is that correct?

[daver77]

@mconnew strange thing with the above. It's a Web API (.NET Core 2.1.5) which calls a legacy Java SOAP web service using kerberos.

We deployed to a few web servers and 1 did not work, a 500 when calling the SOAP WS (_The server returned an invalid or unrecognized response_). We compared the servers and the only thing we could find different was the one's that worked had .NET Core 2.1.7 installed. We installed 2.1.7 on the other server and all worked.

I've checked the release notes and cannot see any fixes in that area, are you aware of any issue fixed recently?

[mconnew]

There were a few bugs around usage of the Expect header in SocketsHttpHandler which took a while to fix correctly. The bugs specifically affected non-WCF servers which did unusual (but still valid) things around the 100 response code. I suspect you were hitting one of those issues.