Runtime: SocketsHttpHandler not debuggable

[Tratcher]

As I've been troubleshooting various issues with the new SocketsHttpHandler the largest barrier has been the lack of diagnostic visibility into what's going on while it processes requests.

There is a DiagnosticHandler, but it's a wrapper around SocketsHttpHandler, it doesn't give visibility into the internal operations or final output.
https://github.com/dotnet/corefx/blob/45fb8a627e727df337580291b45317ee5e66b5a5/src/System.Net.Http/src/System/Net/Http/HttpClientHandler.Windows.cs#L481

Nor does SocketsHttpHandler work with Fiddler. Is this an issue with auto-proxy detection? Or is there an issue with bypassing proxies on localhost? This is a significant development blocker. Being able to see the requests as they're emitted to the wire can help troubleshoot formatting issues, redirects, auth, etc.. The hoops I had to jump through to capture the output for https://github.com/dotnet/corefx/issues/28557 were excessive. Similar diagnostic deficiencies are hindering the evaluation of a new issue https://github.com/aspnet/HttpSysServer/issues/439.

With net4x HttpWebRequest by comparison I could enable full logging of every byte output to a Socket, before or after SSL, as well as HTTP level logs about overall protocol flow.

Microsoft Message Analyzer is the next best option for capturing localhost traffic, though their current installer appears to be broken, and it's less than ideal for SSL scenarios.

@karelz @geoffkizer

[karelz]

Fiddler has to work for sure. We also need events for anything interesting in HTTP layer (redirection, server response, proxy use, connection reusing, etc.)
We planned to review diagnostics next week, looks like you are a bit ahead of us :)

[wfurt]

"bypassing proxies on localhost" Localhost should be bypassed if asked for. There is checkbox in network setting for that. Also note there is still pending PR for improving bypass processing on Windows.

[geoffkizer]

Can someone just track down why Fiddler isn't working and make it work?

[karelz]

@geoffkizer yep, that is no.1 for me too. I'll have someone to take a look tomorrow the latest.
If we need any changes in this space (Fiddler or missing HTTP layer events), I consider them 2.1 blocking. We would ideally include them in Preview2 as well (to make even our lives easier when we get feedback, or need to troubleshoot something).

[rmkerr]

I can take a look at why Fiddler isn't working.

[Tratcher]

As far as settings go, no settings have changed since using WinHttpHandler and that was working with fiddler.

[rmkerr]

SocketsHttpHandler appears to be ignoring the default proxy configuration on Windows. I've configured both WinINet and WinHttp to default to the Fiddler proxy and set UseProxy = true on the handler. I don't see any requests I make in Fiddler, and I can examine the handler to see that internally the proxy is still null.

From recent experience (#25640) I know that the equivalent default proxy resolution works in Linux, so I'd expect the problem to be in the platform specific code on that path. In this case that means the implementation of HttpSystemProxy. That is a new implementation is not shared with WinHttpHandler, so I expect that is where the issue is.

I'm continuing to look into this and will update you all when I have more info.

[stephentoub]

The problem is that this logic is wrong:
https://github.com/dotnet/corefx/blob/master/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpSystemProxy.cs#L106-L125
It gets passed a value like "http=127.0.0.1:8888;https=127.0.0.1:8888" that it's then unable to handle and ends up returning null to indicate that there isn't a proxy. That needs to be fixed. It needs logic like this https://referencesource.microsoft.com/#System/net/System/Net/_WebProxyDataBuilder.cs,29.

[rmkerr]

Thanks for taking a look! I'll go ahead and fix that parsing code. The format for those proxy lists is defined here, and looks very straightforward.

[Tratcher]

FYI I found a functional alternative to Microsoft Message Analyzer for loopback traces: Wireshark + NpCap.

[rmkerr]

@stephentoub I was testing a similar parsing implementation to the one used in .NET Framework and ran into this exception when I tested it with an https proxy:

System.NotSupportedException : Only the 'http' scheme is allowed for proxies.
        Stack Trace:
           c:\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionPoolManager.cs (260,0): at System.Net.Http.HttpConnectionPoolManager.SendAsync (HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
           c:\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnectionHandler.cs(21,0): at System.Net.Http.HttpConnectionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

That sure looks intentional to me, but it contradicts this comment in HttpSystemProxy
https://github.com/dotnet/corefx/blob/f6ff5c214bc9ad1d409913c789398be3e5546948/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpSystemProxy.cs#L121-L122

Do you know which of these is correct? For now I'll continue as if we only support http proxies, since I suspect that is the case.

[stephentoub]

Do you know which of these is correct?

We support accessing both http and https urls over proxies, but we only support accessing the proxy itself over http.

[davidsh]

There is a DiagnosticHandler, but it's a wrapper around SocketsHttpHandler, it doesn't give visibility into the internal operations or final output.

The DiagnosticHandler is a wrapper for DiagnosticSource. It is high-level HTTP request/response tracing.

For detailed tracing output, one should use the EventSource tracing. Instructions for turning that on and viewing traces are here: https://github.com/dotnet/corefx/blob/master/Documentation/debugging/windows-instructions.md

Using these EventSource traces, you can get many different System.Net.* level of traces:

*Microsoft-System-Net-Http {bdd9a83e-1929-5482-0d73-2fe5e1c0e16d}: HTTP-related traces.
*Microsoft-System-Net-Mail {42c8027b-f048-58d2-537d-a4a9d5ee7038}: SMTP-related traces.
*Microsoft-System-Net-NameResolution {5f302add-3825-520e-8fa0-627b206e2e7e}: DNS-related traces.
*Microsoft-System-Net-NetworkInformation {b8e42167-0eb2-5e39-97b5-acaca593d3a2}: Network configuration-related traces.
*Microsoft-System-Net-Ping {a771ec4a-7260-59ce-0475-db257437ed8c}: Ping-related traces.
*Microsoft-System-Net-Primitives {a9f9e4e1-0cf5-5005-b530-3d37959d5e84}: Traces related to core networking-related types.
*Microsoft-System-Net-Requests {3763dc7e-7046-5576-9041-5616e21cc2cf}: WebRequest-related traces.
*Microsoft-System-Net-Sockets {e03c0352-f9c9-56ff-0ea7-b94ba8cabc6b}: Sockets-related traces.
*Microsoft-System-Net-Security {066c0e27-a02d-5a98-9a4d-078cc3b1a896}: Security-related traces.
*Microsoft-System-Net-WebHeaderCollection {fd36452f-9f2b-5850-d212-6c436231e3dc}: WebHeaderCollection-related traces.
*Microsoft-System-Net-WebSockets-Client {71cddde3-cf58-52d5-094f-927828a09337}: ClientWebSocket-related traces.

[rmkerr]

Closing this issue since the problem preventing fiddler from working has been fixed, and David has explained how to collect logs in more detail. I think those address the main points here :)