Runtime: WebRequest.DefaultWebProxy not compatible with HttpClient

[quoctruong]

If I have the following HttpClient:

```c#
var handler = new HttpClientHandler();
handler.UseDefaultCredentials = true;
handler.Proxy = WebRequest.DefaultWebProxy;
var client = new HttpClient(handler);
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 Bla")

The first time I use this client (with `client.GetAsync("http://www.nuget.org/api/v2")`), I will get back the correct result. However, if I reuse the client, I will get an exception:

```c#
System.PlatformNotSupportedException: Operation is not supported on this platform.
                            at System.Net.SystemWebProxy.IsBypassed(Uri host)
                            at System.Net.Http.WinHttpHandler.SetRequestHandleProxyOptions(RequestState state)
                            at System.Net.Http.WinHttpHandler.SetRequestHandleOptions(RequestState state)
                            at System.Net.Http.WinHttpHandler.<StartRequest>d__1.MoveNext()

Is this a bug on CoreClr? Do I have to create new instance of HttpClient everytime?

[davidsh]

On CoreFx and also for .NET Framework (Desktop), you do not need this line of code:

c# handler.Proxy = WebRequest.DefaultWebProxy;

The default for HttpClientHandler.UseProxy property is true. And the default value of HttpClientHandler.Proxy is NULL which means to use the default proxy.

On .NET Framework, this extra line of code is a no-op. But on CoreFx, it is not supported. In general, you should NOT use the WebRequest class with HttpClient on CoreFx.

[davidsh]

cc: @SidharthNabar @stephentoub

[quoctruong]

Hi David, does it mean that on .NET Framework (Desktop), if the default value of HttpClientHandler.Proxy is null, then HttpClientHandler will use IE proxy settings?

[davidsh]

does it mean that on .NET Framework (Desktop), if the default value of HttpClientHandler.Proxy is null, then HttpClientHandler will use IE proxy settings?

Yes, for both .NET Framework and CoreFx, that is true for HttpClientHandler defaults.

[quoctruong]

Thanks for your help! But still, I feel like if HttpClient behavior is unreliable with WebRequest on CoreCLR, there should at least be some kind of warning/documentation about it.

[davidsh]

feel like if HttpClient behavior is unreliable with WebRequest on CoreCLR, there should at least be some kind of warning/documentation about it.

It is not an issue of being "unreliable". But, System.Net.Requests contract which includes WebRequest is part of our legacy API suite. And on CoreFx, especially, there are some coding patterns that have become deprecated/incompatible especially mixing System.Net.Http.HttpClient with WebRequest methods such as the .Proxy example you gave.

Thank you for your feedback. We are working on revising our documentation to include things like this to assist developers as they use CoreFx.

[StevenLiekens]

Is there any implementation of IWebProxy that does run on CoreFx? How does HttpClient/WinHttpHandler work without it?

[davidsh]

Is there any implementation of IWebProxy that does run on CoreFx? How does HttpClient/WinHttpHandler work without it?

HttpClientHandler/WinHttpHandler can use the 'Internet Explorer/Wininet-style" registry settings for proxy detection. You don't need to have the WebProxy class for that. The default behavior for HttpClientHandler just simply uses these settings to find the proxy. It has the proxy detection logic built-in.

If you have custom needs where you don't want to use those registry settings to discover a proxy, then you simply implement a small class over the IWebProxy interface and implement a couple of methods. Then assign that object/interface to HttpClientHandler.Proxy property.

[StevenLiekens]

Would it be a good idea to wrap that in a IWebProxy that everyone can use? Right now there doesn't seem to be a way to get WinINet proxy URLs directly in CoreFx.

[davidsh]

I don't quite follow your question. Why do you need a "wrapped" IWebProxy? And by that, I think you mean, have a public concrete class that implements IWebProxy interface.

Can you show some example C# code of a coding pattern you use where you used to use WebProxy and/or WebRequest.DefaultWebProxy? Then I could probably explain what new coding pattern to use instead.

As I mentioned, you should be able to just use HttpClientHandler as-is and it will discover proxy information in the registry settings and apply them as needed for outgoing http requests.

[StevenLiekens]

In the full framework, you can get a proxy URL for any given URL, based on system settings.

C# Uri proxyUrl = WebRequest.DefaultWebProxy.GetProxy(new Uri("https://github.com/")); // proxyUrl = https://corporate-proxy:8080/

I can't find a way to do this directly in CoreFx.

If I understand you correctly, there is a bunch of internal code that does exactly this for HttpClient. My question is if we can make that code public.

[davidsh]

We don't support that in CoreFx directly. We don't have a "SystemProxy" wrapper for IWebProxy anymore. Part of the 'WebProxy' class in .NET Framework did that. But we don't support that now.

However, usually you don't need that information directly. Usually, it's only the Http stack itself that needs that information in order to send your request to its final destination.

So, I'm curious as to how you are using the returned 'proxyUrl' information in your app?

[StevenLiekens]

So, I'm curious as to how you are using the returned 'proxyUrl' information in your app?

Writing my own HTTP stack, of course. ;)

[davidsh]

There are a lot of issues with providing the "SystemProxy" wrapper. Proxy discovery and evaluation of the registry settings can involve network operation such as WPAD protocol to find a PAC file. Then that PAC file (a JavaScript file) needs to be evaluated. We used to do that JavaScript evaluation in the .NET Framework but discovered it impacted performance due to hosting the interpreter. Over time, that functionality was delegated (on Windows) to the WinHTTP AutoProxy Discover background service which would do that evaluation for our Http stack and cache the information for other process to use. That improved performance and security. But, therefore, that evaluation is not publicly surfaced, but only used by the internal HTTP stacks.

So, bottom line, we no longer support that functionality which is one reason we don't have the concrete WebProxy class in CoreFx anymore.

[StevenLiekens]

Thanks for explaining. I'm still confused by some lines of code.

In System/Net/SystemWebProxy.cs:

c# internal class SystemWebProxy { // This is a sentinel object and can't support the GetProxy or IsBypassed // methods directly. Our .NET Core and .NET Native code will handle this exception // and call into WinInet/WinHttp as appropriate to use the system proxy. public Uri GetProxy(Uri destination) { throw new PlatformNotSupportedException(); }

https://github.com/dotnet/corefx/blob/d0dc5fc099946adc1035b34a8b1f6042eddb0c75/src/System.Net.Requests/src/System/Net/SystemWebProxy.cs#L64

The comments suggest that all of the HTTP stack was designed to catch and handle PlatformNotSupportedException, but @quoctruong pointed out that HttpClient doesn't handle it.

[davidsh]

Thanks for explaining. I'm still confused by some lines of code.

You should ignore the comments for now since it has drifted out of date.

In summary, the WebProxy class is not present in CoreFx. And you should move away from using 'WebRequest.DefaultWebProxy` coding patterns.

Writing my own HTTP stack, of course. ;)

Not sure why you're doing that. If you have suggestions/requirements for the current HTTP stack support in CoreFx, please open up an issue. Thx.

[StevenLiekens]

You should ignore the comments for now since it has drifted out of date.

Okay, that's what I thought but I wanted to make sure. Thanks for taking the time to explain everything.

Not sure why you're doing that.

For the challenge and for learning more about HTTP and web protocols in general.

[karelz]

WebRequest.DefaultWebProxy uses default IE settings. It's a singleton and behaves differently than HttpClient which has per-instance behavior and settings.
There is workaround by using HttpClientHandler properties to select the proxy you want.

Given that WebRequest is legacy, compat-only API and given there is not super-high demand for this feature from customers and that there is workaround, we do not think it is a good idea to implement it.

If there is additional info about usage of the feature, please let us know, we can change our position.

[StevenLiekens]

@karelz I think it's unfortunate that there is no way to get the address of the system proxy. I don't particularly care that the WebRequest API went away. I care because that was the only way to get that information.

[karelz]

@CIPop @davidsh is there a way to get to the system proxy info via HttpClientHandler?

[davidsh]

is there a way to get to the system proxy info via HttpClientHandler?

No. The HTTP stacks in .NET Core use the OS level HTTP stacks. This is true for both Windows and Linux. While we can specify that a request can USE the system proxy, we cannot query the system to determine what that system proxy is. Usually this isn't needed. It's more important that HTTP requests can use the system proxy and that developers can specify to use the system proxy or a custom proxy. But unfortunately, you cannot read the system proxy settings themselves.

[karelz]

Thanks @davidsh, it makes sense from certain point of view. @StevenLiekens, is it sufficient answer? Is the option to USE system proxy good enough for your scenario?

If not, what are the use cases when you need to know the system proxy yourself?

[StevenLiekens]

The case for wanting to know the system proxy is if you don't want to use the HttpClient stack.

RestSharp is one of the libs that come to mind. I'm sure there are others. I'm working on my own HTTP stack, in fact. I'm doing that for learning purposes but also because HttpClient behaves differently on different operating systems. My goal is to create a HTTP stack that is C# down to the socket level so that only sockets are managed by the OS.

When you use one of the alternative HTTP libs, you used to be able to hook into WebRequest.DefaultWebProxy to let the system resolve any Uri instance into a proxy Uri instance that you can use anywhere. Now only HttpClientHandler has that information and it's a little unfair to the competition.

[karelz]

Did you consider implementing the functionality yourself?
You can probably even grab the code from referencesource - the one which is under MIT license.

Given the very limited API usage, I think it is reasonable workaround.
If there are signs of more widespread usage of the API, we could reconsider and include the API in future.

Note that pure managed implementation of networking stack (for easier portability and better compatibility across platforms) is also an option we are actively considering and discussing. Among other options like using same underlying networking libraries under the hood across platforms, or harmonizing the diverged implementations across various .NET stacks/platforms we have today.

[davidsh]

@StevenLiekens Thank you for clarifying your requirements.

Since you appear to want to access WebProxy class separately, it is possible that the functionality you want can be implemented in .NET Core.

My comments above and this particular issue dealt with an interaction between HttpClient and WebProxy. That interaction doesn't work the same anymore since the HttpClient stack uses OS specific stacks.

But WebProxy itself can be enhanced by implementing the functionality on both Windows and Linux to read the operating system proxy settings. I want to point out that reading the system proxy settings isn't sufficient to return a proxy URL given a destination URL. Simple proxy settings can be static, i.e. a single proxy URL for all destination URLs. But proxy settings on Windows, for example, can involve WPAD protocol to find a PAC file (Javascript file) that then has to be evaluated using a Javascript engine and the functions in that Javascript called to evaluate the correct proxy URL.

Even in pure managed HTTP stacks such as the .NET Framework, we stopped doing Javascript evaluation of the PAC file because it is risky. Instead, we switched over the years to using OS services to run the Javascript evaluation out of process using Win32 APIs with WinHttp AutoProxy Service. I am not sure whether Linux supports this type of proxy evaluation.

Since the .NET Core stacks used OS level HTTP APIs, that mean it evaluated proxy URLs automatically (under the covers) and so we didn't need to implement the WebProxy class fully in .NET Core.

I would recommend you open a new issue and request that the WebProxy class be implemented more fully so that it can resolve and evaluate system level proxy settings and return a proxy URL given all the system settings and a destination URL.

And one final thing....WebRequest.DefaultWebProxy will only return the "system" proxy if you haven't set that property to something else.

So, code like this will cause WebRequest.DefaultWebProxy to return a custom proxy.

```c#
IWebProxy myProxy = new CustomProxy();
WebRequest.DefaultWebProxy = myProxy;

/// future get of the DefaultWebProxy property will return myProxy and not the system proxy.
```

[Gladskih]

does it mean that on .NET Framework (Desktop), if the default value of HttpClientHandler.Proxy is null, then HttpClientHandler will use IE proxy settings?

Yes, for both .NET Framework and CoreFx, that is true for HttpClientHandler defaults.

But how the hell can I pass default credentials to authenticate on auto discovered proxy?!
I just receive "(407) Proxy Authentication Required."

My question on SO

[Gladskih]

for .NET Framework (Desktop), you do not need

@davidsh And how about not a Desktop? How to make HttpClient in service on server to use proxy auto discovery if for account of the service IE was never started. Should I re-implement WPAD mechanism on my own?

[davidsh]

@Gladskih Please do not comment or ask questions about closed issues. They are seldom monitored. If you have a bug or question , please open a new issue.

With respect to your question, you can pass credentials (including default credentials) to the system default proxy using the following code:

c# var handler = new HttpClientHandler(); handler.DefaultProxyCredentials = CredentialCache.DefaultCredentials; var client = new HttpClient(handler);

[Gladskih]

@davidsh Oh, sorry for misuse of issue (it would be better to have no input control on closed one). Thank you for fast reply. Unfortunately it's not cover .Net Framework older than 4.7.