Azure-sdk-for-net: [BUG]System.Net.Sockets.SocketException in netcoreapp TargetFramework

[AndreiDegtiarev]

Describe the bug
Simple console application targeting net48 works without problem, retargeting to netcoreapp3.0 cause SocketException
Exception or Stack Trace
No connection could be made because the target machine actively refused it

To Reproduce
Change TargetFramework from net48 to necoreapp3.0

Code Snippet

    class Program
    {
        static void Main(string[] args)
        {
            var serviceBusConnectionString = "yourConnectionString"
           var queue = "yourqueue";
            var messageReceiver = new MessageReceiver(serviceBusConnectionString, queue);
            messageReceiver.ServiceBusConnection.TransportType = Microsoft.Azure.ServiceBus.TransportType.AmqpWebSockets;
            var message = messageReceiver.ReceiveAsync().Result;

        }
    }

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: [Windows 10]
• IDE : [Visual Studio 2019]
• Version 4.0.0

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x] Bug Description Added
• [x] Repro Steps Added
• [x] Setup information Added

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jfggdl

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jfggdl

[jfggdl]

@AndreiDegtiarev, thank you for letting us know of this issue. Aparna is trying to reproduce this problem. We will get back with our findings.

[nemakam]

@AndreiDegtiarev, I just installed netcore3.0 and tried a sample app and it works pretty well. I would think its a machine problem. Maybe try it on another machine and see if its working.
Also, consider updating to latest visualstudio + .netcore SDK to see if it helps.

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[AndreiDegtiarev]

I don’t think that it is machine problem. I tried it already on different machines and I have the latest versions of Visual Studio and SDK. I think the problem is how .nectcore and .netframework interact with proxy server (otherwise I wouldn’t use TransportType.AmqpWebSockets). Please take more time to investigate this problem. Also I’m not alone with it, see for example discussion here: https://github.com/stewartadam/aspnet-servicebus-hostedservice/issues/1

[nemakam]

We do not do any special handling for proxy in our code. Like you mentioned, this might be a .netcore issue.

[Petermarcu]

@stephentoub @karelz, looks like a possible issue in .NET Core 3.0. Is it best to move this issue over to a .NET Core repo?

[Petermarcu]

Based on the info above I believe @AndreiDegtiarev is using https://www.nuget.org/packages/Microsoft.Azure.ServiceBus/ version 4.0.0. That package only has a .NET Standard 2.0 asset in it.

[AndreiDegtiarev]

Yes, I use Microsoft.Azure.ServiceBus version 4.0.0 and the problem also occurs if i target netcoreapp2.1

[nemakam]

@AndreiDegtiarev , how are you configuring the proxy settings?

[stephentoub]

What's the full stack from the exception?

What proxy is being used / how is it being configured?

Is it best to move this issue over to a .NET Core repo?

This library is a black box... is there a repro that can be written directly against whatever .NET API is being used under the covers here? Obviously there's a Socket at some point, but is this library talking directly to Socket, is it using HttpClient, something else?

[Petermarcu]

There is a callstack in this issue https://github.com/stewartadam/aspnet-servicebus-hostedservice/issues/1#issuecomment-537644345

Source code for the library is here: https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/servicebus/Microsoft.Azure.ServiceBus/src

[Petermarcu]

@stephentoub do you think this could be related to the replacement of HttpClient that happened in 2.1? Is there still a way to opt into the older one?

[AndreiDegtiarev]

I work in quite big enterprise company and I have little info about proxy settings. From intranet following description from Symantec is referenced https://support.symantec.com/us/en/article.tech244697.html
Retarget to netcoreapp2.0 doesn’t help
My callstack looks different:

at System.Net.Http.ConnectHelper.<ConnectAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<ConnectAsync>d__52.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<CreateHttp11ConnectionAsync>d__53.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<GetHttpConnectionAsync>d__45.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<SendWithRetryAsync>d__47.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WebSocketHandle.<ConnectAsyncCore>d__24.MoveNext() 

[karelz]

So to summarize:

• Simple app does not work (on .NET Core 2.0 nor 3.0), but works fine on .NET Framework 4.8

Questions:

1. Is it related to the environment / proxy setup / etc.? In other words, are you able to reproduce the same problem on vanilla Azure VM, not connected to domain, VPN, or anything?
2. Are you able to try to remove the Azure SDK layer out of the picture? (i.e. use HttpClient directly against the same endpoints)

BTW: You can use GitHub code formatting on code and on exceptions for better readability.

[stephentoub]

Source code for the library

@petermarcu, that doesn't appear to be the library that's actually making the connection; it looks like this library sits on top of Microsoft.Azure.Amqp, which does have multiple assets in its nuget package, with different libraries for .NET Framework vs .NET Standard, and more:

In the future for things like this it'd be helpful if the team that owns the library being used could start the investigation and rule out the possibility that the difference is due to an issue with the intermediate library rather than something lower down in the core platform.

Retarget to netcoreapp2.0 doesn’t help

That means that two completely different WebSocket implementations, one based on WinHTTP and one based on SocketsHttpHandler, are unable to connect to the target site. They both, however, follow the same approach to resolving proxies, so more and more I think it's an environment-related issue.

@AndreiDegtiarev, would you be willing to take a trace on netcoreapp30 and share it with us? You can do so with a tool like PerfView, in which case when you do the collection you'd add *Microsoft-System-Net-Http into the "additional providers" text box. Or you can do so by copying/pasting https://github.com/dotnet/corefx/blob/master/src/Common/tests/System/Diagnostics/Tracing/ConsoleEventListener.cs into your app, and then at the beginning of your main method, adding a line:
C# using var listener = new System.Diagnostics.Tracing.ConsoleEventListener("Http");
after which when you run the app you should see a lot of diagnostics output to the console. You can also try out using the new dotnet trace tool, e.g. https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-trace-instructions.md, and would enable the same Microsoft-System-Net-Http provider, as with PerfView. Either way, you can then email me the results (stoub at microsoft) or share it in whatever other manner you feel most comfortable.

[stephentoub]

In perusing the Microsoft.Azure.Amqp library, I believe I see the problem. This code:
https://github.com/Azure/azure-sdk-for-net/blob/765ff59083a3dd611fbb9430be4d9437cbbd613b/sdk/servicebus/Microsoft.Azure.ServiceBus/src/ServiceBusConnection.cs#L314-L318
is going to result in WebRequest.DefaultWebProxy being passed down into the underlying implementation. On .NET Framework, that implementation uses WebRequests under the covers, and thus understands WebRequest's DefaultWebProxy. On .NET Core, WebRequests are considered legacy and aren't used by ClientWebSocket, and thus doesn't understand the sentinel value that is DefaultWebProxy.
cc: @davidsh

[Petermarcu]

Is there a ".NET Standard" value for proxy that can be used that will be respected by both .NET Core and .NET Framework?

[davidsh]

Is there a ".NET Standard" value for proxy that can be used that will be respected by both .NET Core and .NET Framework?

No.

We recommend callers use HttpClient and not HttpWebRequest. Not sure that is possible with the layered stack of libraries being used.

With HttpClient, the default properties of HttpClientHandler are for 'UseProxy=true' and 'Proxy=null'. And that combination means to use the 'system default' proxy. This differs from HttpWebRequest where there is a single property related to proxies and the .Proxy property is non-null by default. That non-null value is the 'system default' proxy. Using null for .Proxy with HttpWebRequest means not to use any proxy.

[stephentoub]

@davidsh, what should the Amqp library be doing then, given that it may run on different underlying stacks?

[davidsh]

@davidsh, what should the Amqp library be doing then, given that it may run on different underlying stacks?

By 'different stacks' do you mean different versions of .NET Framework or .NET Core?

If so, if Amqp uses HttpClient instead of HttpWebRequest I think the platform differences would work better regarding proxy support.

[stephentoub]

@davidsh, it's just using ClientWebSocket. On .NET Framework ClientWebSocket will end up respecting WebRequest.DefaultWebProxy because it uses HttpWebRequest to make the connection; on .NET Core, it uses either WinHTTP or SocketsHttpHandler, depending on version, neither of which respect WebRequest.DefaultWebProxy.

[davidsh]

@davidsh, it's just using ClientWebSocket.

Thanks for clarifying. Yes,
ClientWebSocketOptions.Proxy is the only proxy-related property for ClientWebSocket.

WebRequest.DefaultWebProxy is considered a legacy type/property. In .NET Core, it resides in the System.Net.Requests library. And we do not have any dependency from System.Net.WebSockets.ClientWebSocket to System.Net.Requests. We would not want to add that because it would create a circular dependency. System.Net.Requests depends on System.Net.Http already. And System.Net.WebSockets.ClientWebSocket depends on System.Net.Http.

Currently in .NET Core, System.Net.Http and thus SocketsHttpHandler, et. al. do not observe WebRequest.DefaultWebProxy at all.

In .NET Core 3.0, we added a new HttpClient.DefaultProxy static property (https://github.com/dotnet/corefx/pull/37333) where the intent was to support similar 'global settings' scenarios as WebRequest.DefaultWebProxy.

For .NET Core 3.0, a solution could be to set ClientWebSocketOptions.Proxy = HttpClient.DefaultProxy. That would tell the a particular clientwebsocket to use the 'system default' proxy. But I don't think we have a general solution for all versions of .NET Core.

[stephentoub]

@davidsh, the Amqp package doesn't have a netcoreapp asset.

What I hear you saying is there's no way on .NET Core prior to 3.0 to tell ClientWebSocket to use a default proxy. Is that correct?

And on .NET Core 3.0, if you're targeting netcoreapp3.0, it could be done by passing in HttpClient.DefaultProxy. Correct?

It seems then like the fix is for WebRequest.DefaultWebProxy to return HttpClient.DefaultProxy (and fix System.Net.Requests as-is needed). That way, any code using WebRequest.DefaultWebProxy today will "just work". And the Amqp package will be broken on .NET Core for default proxies until we deploy such a fix, so we should consider servicing it to release/3.0.

Also seems like we should augment ClientWebSocketOptions with better support for proxies, so we don't hit this same issue in the future should the underlying implementation change again.

[davidsh]

Also seems like we should augment ClientWebSocketOptions with better support for proxies, so we don't hit this same issue in the future should the underlying implementation change again.

From a design perspective, ClientWebSocketOptions was designed in the timeframe before HttpClient. So, it used the semantics of a single .Proxy property. ClientWebSocketOptions.Proxy sort of became broken when we moved to .NET Core.

I believe that on .NET Framework, the default value of ClientWebSocketOptions.Proxy is a non-null value where that value is the current value of WebRequest.DefaultWebProxy. And that value represents to use the 'system default' proxy (which may or may not actually result in using a proxy...because it depends on system settings). But on .NET Core, we ended up I think where ClientWebSocketOptions.Proxy has a default value of null. And that means don't use any proxy by default.

So, we should investigate this and determine how to reconcile .NET Core moving forward.

However, I don't think we need to add any more properties to ClientWebSocketOptions to get "better support for proxies". Having a single property, .Proxy, is fine as long as a 'system default' proxy can be represented by a object. And that would also support the scenarios where credentials need to be sent to a 'system default' proxy. Because that would inherit the IWebProxy.GetCredentials() of any proxy set into the property.

[davidsh]

And on .NET Core 3.0, if you're targeting netcoreapp3.0, it could be done by passing in HttpClient.DefaultProxy. Correct?

And specifically, yes, to this question.

[nemakam]

@davidsh , I see that PR https://github.com/dotnet/corefx/pull/41692 issue is marked with 5.0 milestone.
Meanwhile, for this issue, in .net core 3.0, would you suggest that we expose IWebProxy as a property on ServiceBusConnection which customers can then set it to HttpClient.DefaultProxy so that we need not wait for 5.0 release?

[davidsh]

@davidsh , I see that PR dotnet/corefx#41692 issue is marked with 5.0 milestone.

Changes are made first to master branch (5.0). I suspect that after that change is validated, then @stephentoub will create a release/3.0 PR for evaluation for servicing release of .NET Core 3.0.

[apraovjr]

@AndreiDegtiarev The fix would be released in .net-core 3.0.2 and 3.1 version

[jfggdl]

@AndreiDegtiarev, we are closing this issue for now as there is now visibility into in what version a fix will be available. Please feel free to request to open it again if you have a question.

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[AndreiDegtiarev]

I have tried netcoreapp3.1 and frm48. The problem is the same: frm48 works and netcoreapp3.1 doesn't