Runtime: Default HttpClient not using configured web proxy

[Petermarcu]

@jagalves commented on Sat Apr 28 2018

Issue Title

When using a web proxy setup (in my case Fiddler), HttpClient is not using it by default.

This behavior is different between .net core 2.0 and in 2.1 preview 2.

General

With fiddler running, the following code will try to make a request to "http://localhost.fiddler:65530" (I'm assuming nothing is running on that port) and print the result or the exception message:

```c#
using System;
using System.Net.Http;

namespace HttpC
{
class Program
{
static void Main(string[] args)
{
var http = new HttpClient();

        try
        {
            var response = http.GetStringAsync("http://localhost.fiddler:65530").GetAwaiter().GetResult();

            Console.WriteLine(response);
        }
        catch (Exception ex)
        {
            Console.WriteLine("exception: " + ex.Message);
        }
    }
}

}

```

How to reproduce:

1. Start fiddler making sure it's capturing all requests
2. Setup a dotnet core 2.0 console app and add the code provided
3. Run -> the error is Response status code does not indicate success: 502 (Fiddler - Connection Failed).
4. Change to dotnet core 2.1 preview 2
5. Run -> the error is No such host is known

Note1: in browsers localhost.fiddler works
Note2: I tried the full hostname instead of "localhost.fiddler" and HttpClient still didn't use the proxy.

Link to fiddler: https://www.telerik.com/fiddler

[EDIT] Add C# syntax highlighting by @karelz

[karelz]

@jagalves Just to clarify: Your primary claim is that Fiddler is bypassed by 2.1 Preview2. Is that correct?
We tested the scenario and it works locally for us on Preview2 (it was broken in Preview1). However, this is second report in last few days that it does not fully work on Preview2, so we would love to dig in deeper - it is concerning. Maybe it is influenced by OS version/settings, etc.?
Are you able to reproduce the problem on more than 1 machine? Any chance to try it on a clean VM?

@rmkerr can you please take a look at it?

[jagalves]

@karelz yes that is the case. Fiddler is not used.

This is the output from $ dotnet -info

.NET Core SDK (reflecting any global.json):
 Version:   2.1.300-preview2-008533
 Commit:    996eb6c92c

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17655
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.300-preview2-008533\

Host (useful for support):
  Version: 2.1.0-preview2-26406-04
  Commit:  6833f3026b

.NET Core SDKs installed:
  2.1.100 [C:\Program Files\dotnet\sdk]
  2.1.101 [C:\Program Files\dotnet\sdk]
  2.1.102 [C:\Program Files\dotnet\sdk]
  2.1.103 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.200-preview-007474 [C:\Program Files\dotnet\sdk]
  2.1.200-preview-007517 [C:\Program Files\dotnet\sdk]
  2.1.200-preview-007576 [C:\Program Files\dotnet\sdk]
  2.1.200-preview-007589 [C:\Program Files\dotnet\sdk]
  2.1.300-preview2-008533 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.0-preview2-final [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.0-preview2-final [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-preview2-26406-04 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

and dotnet --version gives 2.1.300-preview2-008533.

You can also see my windows version which is the latest insider build. I'll try to find some time to setup a vm and test it in an official windows version.

[davidsh]

Fiddler usually runs on the loopback port ("localhost", "127.0.01"). I've never seen it named "localhost.fiddler" though. I don't understand how "localhost.fiddler" would work since that name usually can't be resolved with DNS.

[jagalves]

Yes, but when connected through a http proxy, clients normally rely on the proxy itself to do the resolving. localhost.fiddler is one of the ways to bypass the loopback exemption that normally we find on windows.

Try it yourself by typing "http://localhost.fiddler" on any browser while fiddler is running. (make sure the browsers are auto-detecting proxies).

There's also "ipv4.fiddler" and "ipv6.fiddler" to allow you to enforce the protocol you want to test.

[jagalves]

Ran it on a VM from hyper-v's quick create win10 developer environment. Made sure it was patched and Installed the required runtimes. Result of running similar code as before with fiddler running:

PS C:\Users\x\Desktop\New folder> dotnet --info

Host (useful for support):
  Version: 2.1.0-preview2-26406-04
  Commit:  6833f3026b

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.0-preview2-26406-04 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

PS C:\Users\x\Desktop\New folder> dotnet .\netcoreapp2.0\HttpC.dll http://localhost.fiddler
Version: 2.0.7
GET http://localhost.fiddler
exception: Response status code does not indicate success: 502 (Fiddler - Connection Failed).

PS C:\Users\x\Desktop\New folder> dotnet .\netcoreapp2.1\HttpC.dll http://localhost.fiddler
Version: 2.1.0
GET http://localhost.fiddler
exception: No such host is known

I've added some code to detect the runtime and receive the url from the command line but that's it.

[karelz]

Hmm, I am still confused: Does Fiddler work at all in your setups?
Your repro uses http://localhost.fiddler - if that is the only thing that does not work, while Fiddler is able to intercept all HTTP traffic and record it, then it is not probably critical for 2.1 (at least I don't view it as some horrible regression). Can you please confirm?

[jagalves]

Fiddler does not work at all in my setup even if I use some other hostname like http://bing.com.

This stackoverflow answer has a simple explanation how "localhost.fiddler" works: https://stackoverflow.com/a/18454051

In most cases you're likely to encounter the web browser will pass the fully-qualified domain name to the HTTP proxy, which will perform the DNS lookup and forward the request

So, instead of "GET /", the client will do a request to the proxy using "GET http://localhost.fiddler/". This is how those fancy hostnames work.

No one in the team is able to reproduce it? Attached is my project, just use VS to switch between the frameworks in order to have different targets built.

HttpC.zip

[rmkerr]

I'll take a look at your repro now.

[stephentoub]

@jagalves, I just tried your original repro, and with .NET Core 2.1 I get:

exception: Response status code does not indicate success: 502 (Fiddler - Connection Failed).

which is I believe what you expect it to do. I have "2.1.0-preview3-26411-06" installed.

[rmkerr]

I get the same result as Stephen with my currently installed version (2.1.300-preview3-008391). However, when I roll back to preview 2 (2.1.0-preview2-26406-04) I see the error mentioned in the original post:

exception: No such host is known

@karelz is there an easy way to check what changes made it into preview 2? It would be good to verify that the default proxy fix is actually in the preview.
@jagalves could you try installing the latest daily build and see if that fixes things for you?

[karelz]

@rmkerr the best and safest option I know is to check the source code by tag: https://github.com/dotnet/corefx/tree/v2.1.0-preview2-26406-04
(I am not sure if commit details will show the tag as well)

[jagalves]

@rmkerr

Works as expected with 2.2.0-preview1-26430-04. Had to change the runtime config in order to run it but the request went through fiddler as it should.

Maybe the fix you guys mentioned above never made it to preview 2?

[karelz]

Yeah, looks like it didn't make it into Preview2. I want to make sure it is in for 2.1 rc1 at least ... let's verify.

[rmkerr]

It is at least in the release/2.1 branch. How can I check if it is in rc1?

[karelz]

The link above points to Preview2 tag: https://github.com/dotnet/corefx/tree/v2.1.0-preview2-26406-04 - is it there?
release/2.1 branch has rc1 payload now. If it is in that branch, it will be part of rc1.

[rmkerr]

No, the fix isn't present in the preview 2 code.

[rmkerr]

Closing this issue as the fix made it to 2.1 master and only shows up in preview 2. Thanks again for the report @jagalves.

[karelz]

Thanks @rmkerr! Can you please link the PR / main issue which was fixed in 2.1 for future reference? Thanks!

[rmkerr]

Yep. This issue is a duplicate of dotnet/runtime#25674, which was fixed in PR dotnet/corefx#28671.

[jagalves]

Thanks for looking into this. Looking forward for the release. :)