Runtime: HttpClient doesn't detect IE setting for ignoring proxies

[zemien]

@davidsh mentioned halfway through dotnet/runtime#16754 that

HttpClientHandler/WinHttpHandler can use the 'Internet Explorer/Wininet-style" registry settings for proxy detection. You don't need to have the WebProxy class for that. The default behavior for HttpClientHandler just simply uses these settings to find the proxy. It has the proxy detection logic built-in.

How is that behaviour impacted when hosting on IIS through the reverse proxy?

First: The Error

Here is the exception that pops up when I try to do a HttpClient GetAsync(), which Wireshark reveals is caused by the proxy cutting the request off (error 407).

System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection. The connection was closed before all data could be read. Expected 2245 bytes, read 0 bytes.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.StreamToStreamCopy.<CopyAsyncToPreSizedLimitMemoryStream>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext() --- End of inner exception stack trace --- at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()

Second: The Behaviours

I'm experiencing different behaviours when launching the exact same ASP.Net web API on Kestrel command-line versus IIS. The web API is fairly simple, with some calls that GET another external resource.

• Internet Explorer LAN Settings 'Automatically detect settings' ticked
  
  
  
  • Kestrel: Exception
  
  
  • IIS: Exception
  
  
• 'Automatically detect settings' un-ticked
  
  
  
  • Kestrel: No exception - it works
  
  
  • IIS: Exception
  
  

So it appears, at first glance, that HttpClientHandler is ignoring the Automatically detect settings bit, when run under IIS.

What I'm also suspecting is that IIS AppPool account might affect what proxy setting the HttpClient sees, in some way. But I probably need someone from the dotnet team to actually comment.

Thank you.

[davidsh]

So it appears, at first glance, that HttpClientHandler is ignoring the Automatically detect settings bit, when run under IIS.

This is by-design. HttpClientHandler uses the WinInet/WinHTTP AutoProxy Discover Service on Windows. The "Internet Explorer" settings as registry settings in the HKLM_USER (per-user) registry tree. For normal users, these registry settings exists and you can see them via the UI in Internet Explorer.

But for some service identities such as those used in IIS, there are usually no registry settings for that "user". Hence, there is nothing to detect....so specifying a proxy in that manner won't work.

Here are some relevant articles:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa384075(v=vs.85).aspx

https://blogs.msdn.microsoft.com/ieinternals/2013/10/11/understanding-web-proxy-configuration/

[zemien]

Thanks @davidsh that gives some clarity. In the full framework we are able to specify this in the .dll.config file and have it loaded even if it's someone else's binaries:

<configuration>
    <system.net>
      <defaultProxy enabled="true" useDefaultCredentials="true">
        <proxy proxyaddress="http://xxx.yyy.zzz.net:84"/>
      </defaultProxy>
    </system.net>
</configuration>

Does dotnet core read and use this configuration at all? My initial testing would say 'no'. And I even tried putting this config in web.config even though I know Kestrel doesn't see it, just in case.

[davidsh]

Does dotnet core read and use this configuration at all?

No. .NET Core doesn't use config files.

If you want to specify a custom proxy, then you can define a custom class implementing IwebProxy interface. Then set that interface in the HttpClientHandler.Proxy property.

[zemien]

In other words I'll need to re-implement similar configuration properties in my appsettings.json and have it flow through to my own IWebProxy implementation? It's not ideal but I guess that's the only way now. Is there a plan to enhance these proxy-handling situations? It will be get more and more common as the enterprise world picks up Core.

[davidsh]

Thank you for the feedback. There are no specific plans to enhance proxy. In terms of generalized config file support, I'm not sure if there are any plans for that either.

[geoffkizer]

@zemien What sort of enhancement would you like to see here? (Other than supporting the existing config file)

[geoffkizer]

BTW, you don't need to implement IWebProxy yourself. You can use the WebProxy class and configure it based on whatever settings you like. Does this help?

[zemien]

@geoffkizer IIRC WebProxy class is not included in Core. We need to make our own implementation of IWebProxy.

In terms of enhancements, I suppose the WinHttpHandler does a lot of what I need to do. I tried setting the WindowsProxyUsePolicy to DoNotUseProxy, but I'm now getting a System.Net.Http.WinHttpException: A security error occurred with no further specifics.

[geoffkizer]

WebProxy has been added to Core: https://github.com/dotnet/corefx/blob/7cc0590e9b5d12f79d594aed1b35d1c320032ffc/src/System.Net.WebProxy/src/System/Net/WebProxy.cs

[zemien]

@geoffkizer that's great, but when will it be added to a 1.1 release?