Runtime: .NET Core >= 2.1 HttpClient fails with automated proxy configuration (Multiple proxies/Windows)

[NikMeyer]

I have created a .NET Core 2.2 application that uses the HttpClient to invoke an HTTP call over a proxy. The application works while:

• Running without a proxy.
• Using a manually configured proxy.
• Using .NET Core 2.0
• Explicitly setting a proxy in the HttpClientHandler. (See Code sample)

The proxy is not recognized once our specific proxy auto-config (PAC) is used. The PAC script returns multiple proxies separated by “;”.

Sample Application:
```c#
using System;
using System.Net;
using System.Net.Http;

namespace ProxyTest
{
class Program
{
static void Main(string[] args)
{
var baseUrl = "http://www.microsoft.com";

        var handler = new HttpClientHandler();
        // handler.Proxy = new WebProxy("proxy1.company.com:8080, false); - Works correctly when active.
        var client = new HttpClient(handler);

        try
        {
            HttpResponseMessage res = client.GetAsync(baseUrl).Result;
            if (res.IsSuccessStatusCode)
            {
                Console.WriteLine("Success");
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }
    }
}

}

Debugging into the .NET Core framework shows an interesting behavior inside [SocketsHttpHandler/HttpWindowsProxy.cs](https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpWindowsProxy.cs) - GetUriFromString:

- GetUriFromString is invoked with a value ```proxy1.company.com:8080;proxy2.company.com:8080;proxy3.company:8080```
- Uri.TryCreate (Line 206) will fail to parse the URI as it is invalid.
- GetUriFromString returns null and further calls use no proxy.

Call stack to GetUriFromString:

System.Net.Http.dll!System.Net.Http.HttpSystemProxy.GetUriFromString(string value) Line 196 C#
System.Net.Http.dll!System.Net.Http.HttpSystemProxy.GetProxy(System.Uri uri) Line 344 C#
System.Net.Http.dll!System.Net.Http.HttpConnectionPoolManager.SendAsync(System.Net.Http.HttpRequestMessage request, bool doRequestAuth, System.Threading.CancellationToken cancellationToken) Line 279 C#
System.Net.Http.dll!System.Net.Http.HttpConnectionHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 21 C#
System.Net.Http.dll!System.Net.Http.RedirectHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 32 C#
System.Net.Http.dll!System.Net.Http.SocketsHttpHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 345 C#
System.Net.Http.dll!System.Net.Http.HttpClientHandler.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 474 C#
System.Net.Http.dll!System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) Line 51 C#
System.Net.Http.dll!System.Net.Http.HttpClient.SendAsync(System.Net.Http.HttpRequestMessage request, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationToken cancellationToken) Line 459 C#
System.Net.Http.dll!System.Net.Http.HttpClient.GetAsync(System.Uri requestUri, System.Net.Http.HttpCompletionOption completionOption, System.Threading.CancellationToken cancellationToken) Line 303 C#
System.Net.Http.dll!System.Net.Http.HttpClient.GetAsync(string requestUri) Line 266 C#
ConsoleApp3.dll!ProxyTest.Program.Main(string[] args) Line 19 C#

```

Having a PAC script that returns multiple proxies seems to be valid and previous frameworks would correctly recognized and use one of the proxies.

[stephentoub]

cc: @davidsh

[davidsh]

Having a PAC script that returns multiple proxies seems to be valid and previous frameworks would correctly recognized and use one of the proxies.

What previous frameworks are you talking about? .NET Framework? Or .NET Core 2.0?

What happens if you disable SocketsHttpHandler by adding this to your code?

c# AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

[NikMeyer]

What previous frameworks are you talking about? .NET Framework? Or .NET Core 2.0?

.NET Core 2.0 - Running the sample above with TargetFramework - netcoreapp2.0 results in a successful HTTP call trough the proxy. (I did not test any other .NET versions yet.)

What happens if you disable SocketsHttpHandler by adding this to your code?

AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);

The HTTP call is successful and using the proxy with this flag (and netcoreapp2.2).

[davidsh]

It isn't clear exactly what the behavior is expected to be with multiple proxies being specified. We'll need to test what .NET Framework and .NET Core 2.0 (i.e. WinHttpHandler) do regarding this. It's possible they select the first one and if it doesn't "work", then try the next one. We don't have any such retry logic in SocketsHttpHandler currently for this.

[scalablecory]

Confirmed we have a bug. WinHttpHandler and .NET Framework allow WinHTTP to do the work, and so do not have this issue.

Per WINHTTP docs, WinHttpGetProxyForUrl can return more than one proxy.

Per Mozilla and (ancient) Netscape, expected behavior is that if multiple are supplied, it will try the left-most first, only moving on to subsequent ones if a proxy is unresponsive.

@davidsh @stephentoub I have an idea of how I can implement this but I don't think it meets the bar for 3.0 at this point. Thoughts?

[davidsh]

@davidsh @stephentoub I have an idea of how I can implement this but I don't think it meets the bar for 3.0 at this point. Thoughts?

Supporting enterprise scenarios are a key tenant for .NET Core 3.0. So, I would like to see this fixed. We can even make a partial fix which would unblock the 90% case. For example, separate out potentially multiple Uri's (string.Split(';')) and only try the first proxy in the list.

[davidsh]

@karelz

[scalablecory]

Yea that sounds good. I will make a change to simply choose the first one on the list, and open an new issue to fully support multiple proxies.