Runtime: .NET Core 2.0 and 2.1 HTTP Client returns http 407 (Proxy Authentication Required) when attempting to authenticate with a proxy server

[stewartdclark]

Our development team is working on a .NET Core 2.0 service that needs to access external services through a proxy server that requires NTLM authentication. First, we build a set of network credentials specifying a user, password, and domain that is valid for the proxy server, and assign the credentials to a credentials cache specifying the auth type as NTLM. When we create the http client handler, we create a new WebProxy and initialize it with the ProxyURL and credentials from the credentials cache.

When we run the test app, the auth headers are not being sent/presented to the proxy server. The issue is not a Kerberos/NTLM negotiation handshake issue (as we've seen other teams post about) our issue seems to be that .NET Core 2.0 is not sending any auth header at all, regardless of authentication type. The proxy server consistently returns 407 Proxy Authentication Required, and when we capture the request in Fiddler, the auth headers are not present. Our target environment is a Linux container (where we initially encountered the issue during testing), but to simplify the investigation and help debug this issue, we built a very simple console app (running from within .NET Code in Windows). The issue seems to be the same whether running the service in a Linux container or the console app in Windows.

Below is the sample code from the console app using an HTTP Client handler specifying a WebProxy.
Note: We've tried this in both .NET Core 2.0 and 2.1 and neither sends the auth headers. We also tried a sample app using a web request instead of an HTTP Client, but still no auth header.

```c#
using System;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Security.Authentication;
namespace TestConsoleApp
{
class Program
{
static void Main(string[] args)
{
string proxyUrl = @"http://127.0.0.1:8888";
string proxyUsername = @"username";
string proxyPassword = @"password";
string proxyDomain = @"domain";
string authType = "NTLM";
string targetUrl = @"http://www.google.com";
HttpClient httpClient;

        var credCache = new CredentialCache();
        var credentials = new NetworkCredential(proxyUsername, proxyPassword, proxyDomain);
        credCache.Add(new Uri(proxyUrl), authType, credentials);
        var handler = new HttpClientHandler
        {
            UseProxy = true,
            Proxy = new WebProxy
            {
                Address = new Uri(proxyUrl),
                Credentials = credCache
            },
        };

        httpClient = new HttpClient(handler);
        var httpRequest = new HttpRequestMessage(HttpMethod.Get, targetUrl);
        using (var httpResponse = httpClient.SendAsync(httpRequest).GetAwaiter().GetResult())
        {
            Console.WriteLine(httpResponse.Content);
        }
    }
}

}
```

[davidsh]

We have several proxy related bugs we have already fixed. Will be available in servicing releases of .NET Core 2.1.x. Perhaps dotnet/runtime#26461 or dotnet/runtime#26462 etc. applies to your issue?

[stewartdclark]

David, thank you for your quick response, much appreciated! Yes, we have searched extensively and read every post we could find related to proxy authentication. Some of them describe scenarios where the authentication handshake begins, but the connection is closed before the handshake completes or drops down to the next auth method. But in our case, we never see any auth header is presented to the proxy server in the original/first request. Other posts describe newer issues in 2.1 with the SocketsHttpHandler (using the new TCP stack) not working the same as in 2.0 using the WinHTTP stack. We've tried the context switch to revert back to the WinHTTP stack with no luck. We've tried both .NET 2.0 (which is our current production environment) and we've tried .NET Core 2.1 in our test environment, all with the same result, no auth header is present.

[davidsh]

Can you try out our .NET Core 3.0 pre-release daily builds?

https://github.com/dotnet/core/blob/master/daily-builds.md

You would need to install the SDK above and then re-target your CSPROJ for 'netcoreapp3.0'. Testing with this would determine whether the bug is already one of the known bugs and fixed etc.

[stewartdclark]

Sure we'd be happy to give 3.0 a try. Also note, one of the odd things about this case is, I've read in some cases/issues where other teams have reported they were able to authenticate with a proxy server in .NET Core 2.0 (using the WinHTTP Stack), but then started having issues on .NET Core 2.1 when the underlying TCP stack moved to SocketsHTTP, but our code does not send proxy authentication headers even when using the .NET Core 2.0 TCP stack, which is what other teams are reverting back to (using the context switch that was recommended) to revert back to the WinHTTP stack.

[davidsh]

Our target environment is a Linux container (where we initially encountered the issue during testing)

In terms of .NET Core 2.0 on Linux, Windows auth schemes such as NTLM or Negotiate are NOT supported for authentication to a proxy server. This is a limitation of the HTTP stack (which uses Curl) on .NET Core 2.0 on Linux.

See: dotnet/runtime#25366

[stewartdclark]

Yes, agreed. Sorry, I didn't provide the context that this is precisely why we started testing with the simple console app (posted above) on Windows. We were trying to remove the Linux container as a variable and figured we'd surely be able to see the auth headers in Fiddler from a simple console app on Windows, but we were surprised (and stumped) that no auth headers are presented, and even more surprised to read that some teams had initially been able to authenticate to a proxy in a Windows environment on .NET Core 2.0 (although it later broke in .NET Core 2.1).

I'd be happy to change our target environment to Windows if we could get the service to present the proxy auth headers in .NET Core 2.0 (as other teams seemingly had working), but nothing we have tried will present any auth headers in either .NET Core 2.0 or 2.1 on Windows. I'll go back and re-read those other cases and post the issue #'s here, but I'm sure I read several cases where teams had this working on .NET Core 2.0 (on Windows), and their issue was that it broke in .NET Core 2.1 with the TCP stack switch. I'd love to know how they had it working in .NET Core 2.0 as I'd live with that solution until the next service release.

Assuming we confirm the fixes in 3.0 solve the issue, will these be eventually committed down to 2.1 and 2.0 in a future service release?

[davidsh]

Assuming we confirm the fixes in 3.0 solve the issue, will these be eventually committed down to 2.1 and 2.0 in a future service release?

Many of the proxy-related bugs fixed in master branch (3.0) have already been ported to the 2.1.x servicing branches. Those bugs were ones were the bug appeared in .NET Core 2.1 but were not present in .NET Core 2.0.

It's possible that your bug is something different (although related to proxy auth scenario). But testing it against the very latest 3.0 daily build will help identify root cause.

[davidsh]

cc: @karelz

[karelz]

@stewartdclark to set expectations: 2.0 is near end of support (see here) in 1 month, so we won't likely patch it, unless it is a security problem.
We port fixes into 2.1 servicing for impactful bugs. They need to be fixed first in master (3.0). Let's first root-cause your problem (ideally on master=3.0) as it seems to be different from what we have seen so far in the proxy space. Then we can fix it in master and after that it will be good time to start discussion about porting to 2.1 servicing release.

Hopefully it clarifies things.

[Genbox]

@davidsh @karelz

This is very much still an issue and needs to be fixed ASAP. When using the old HTTP socket handler in .NET Core 2.1:

AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

We get the issue described here: https://github.com/dotnet/corefx/issues/27971

When using the new socket handler:
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", true);

It does not forward credentials when using NTLM, Kerberos etc. - it only work with Basic auth. Proxy support is currently limited to either not working at all, or only working for Basic.

[karelz]

@Genbox are you talking about the original small test case in top post here?

[Genbox]

We observed this issue with our own software. The case described by the top post is exactly the same we have observed. I have a feeling the fix done in https://github.com/dotnet/corefx/issues/30327 will fix it for the SocketsHttpHandler, but I'm not sure there is a fix for when not using the SocketsHttpHandler.

We did extensive tests today between .NET Core 2.0 and 2.1 with the new handler and without. Our conclusion is that .NET Core 2.0 fails to send the HTTP content when using proxies with auth, and .NET Core 2.1 does not send the auth response header when the proxy requires NTLM authentication

.NET Core 2.1 with basic auth works as intended, but our customers use NTLM auth for their proxies, and this issue hit us really hard in production. Any estimated timeframe for the 2.1.5 service release?

[karelz]

You can try daily build of 2.1.x (which is what 2.1.5 will be about). I don't think we have official ETA, but I expect it will be soon.

We do not plan to fix non-SocketsHttpHandler bugs unless they are super-horrible and are blocking significant scenarios for many. The whole purpose of SocketsHttpHandler is to get rid of native handlers and phase them out (while providing consistency and great perf everywhere).

[Genbox]

@karelz I'm fine with the move to SocketsHttpHandler. I love that we try and improve on areas that we have been stuck with since the inception of .NET Framework 1.1.

Larger companies tend to use NTLM authentication and customers using .NET Core 2.0 are stuck with a broken implementation of proxy support. Upgrading to 2.1 (which is what we tried) does not help since it has other bugs that breaks proxy support.

Note that one of the reasons for us to not upgrade to .NET Core 2.1 was the removal of support for older Macs. If the old WinHttpHandler is not fixed, everyone is forced to upgrade to at least 2.1.5

[Genbox]

I'm trying to reproduce the issues we saw in .NET Core 2.0 from yesterday, but it seems there are some discrepancies. If I manage to reproduce them, I will report a separate issue.

So far, the .NET Core 2.1.5 runtime (from daily feed) seems to have fixed our issue.

[karelz]

.NET Core 2.0 is not supported anymore (since yesterday 2018/10/1).

Happy to hear that 2.1.5 fixes the problem. In such case, we can close this issue as there is nothing left to track.

[karelz]

BTW: 2.0 and 2.1 seem to have the same Mac OS X support 10.12+:

• 2.0 Mac OS X support
• 2.1 Mac OS X support

Therefore, upgrade from 2.0 to 2.1 (2.1.5) should be ok. Additional changes to WinHttpHandler should not be needed.

[Genbox]

I have no idea where we saw the missing mac version then. Must have been somewhere else than the links you posted as not even their history mentions any change. Thanks for your help karelz.

[davidsh]

.NET Core 2.1.5 has shipped as of today:

https://blogs.msdn.microsoft.com/dotnet/2018/10/02/net-core-october-2018-update/