Reported by @ydekova in https://github.com/dotnet/corefx/issues/29942#issuecomment-397006087
The following code throws on .NET Core 2.1, and does not throw on .NET Core 2.0:
```c#
using System;
using System.Net.Http;
using System.Threading.Tasks;
namespace console2
{
class Program
{
static async Task Main(string[] args)
{
using (var httpClient = new HttpClient())
{
var response = await httpClient.GetAsync("https://outlook.office.com");
System.Console.WriteLine(response.StatusCode);
}
}
}
}
Unhandled Exception: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
at System.Net.Security.SslStream.<>c.
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask1 creationTask)
at System.Threading.Tasks.ValueTask1.get_Result()
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at console2.Program.Main(String[] args) in /tmp/console2/Program.cs:line 13
at console2.Program.
```
@ydekova's comment https://github.com/dotnet/corefx/issues/29942#issuecomment-397200251:
The SslPolicyError (errors param in the above code) is RemoteCertificateNameMismatch
When using the curlhandler on 2.1 (DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0), the code works.
The outlook.office.com certificate has as CN and in alternative names: 'Outlook.office.com'. So this may be due to the different casing:
Subject: C = US, ST = Washington, L = Redmond, O = Microsoft Corporation, CN = Outlook.office.com
X509v3 Subject Alternative Name:
DNS:Outlook.office.com, DNS:attachment.outlook.office.net, DNS:attachment.outlook.officeppe.net, DNS:bookings.office.com, DNS:delve.office.com, DNS:edge.outlook.office365.com, DNS:edgesdf.outlook.com, DNS:img.delve.office.com, DNS:outlook.live.com, DNS:outlook-sdf.live.com, DNS:outlook-sdf.office.com, DNS:sdfedge-pilot.outlook.com, DNS:substrate.office.com, DNS:substrate-sdf.office.com, DNS:afd-k-acdc-direct.office.com
cc: @bartonjs
Does this happen just on Linux or also on Windows?
https://www.ssllabs.com/ssltest/analyze.html?d=outlook.office.com&s=13.107.18.11&hideResults=on
Subject: Outlook.office.com
Common names: Outlook.office.com
Alternative names:
Outlook.office.com
attachment.outlook.office.net
attachment.outlook.officeppe.net
bookings.office.com
delve.office.com
edge.outlook.office365.com
edgesdf.outlook.com
img.delve.office.com
outlook.live.com
outlook-sdf.live.com
outlook-sdf.office.com
sdfedge-pilot.outlook.com
substrate.office.com
substrate-sdf.office.com
afd-k-acdc-direct.office.com
If it works in Windows (or worked in cURL), everyone else must use strnicmp, and I used memcmp, because DNS names are "always" lowercase.
because DNS names are "always" lowercase.
https://tools.ietf.org/html/rfc4343
Domain Name System (DNS) names are "case insensitive".
We should consider fixing this for servicing release/2.1.x as well.
cc: @karelz
While DNS is lowercase I thought that RFC3280/5280 had a constraint saying it needed to be in a normalized form. Unfortunately for me, it explicitly calls out that it's case-invariant. ::whistles innocently::
For master we should hold this work until dotnet/corefx#30376 is merged to avoid conflicts.
It happens only on Linux, at least in my case.
I ended up with this workaround for now. Not sure if it's a viable one, I'd be thankful if you comment.
```c#
var httpClientHandler = new HttpClientHandler();
httpClientHandler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => {
if (errors == SslPolicyErrors.None)
{
return true;
}
if (errors == SslPolicyErrors.RemoteCertificateNameMismatch)
{
// Try custom DNS name validation
var isValidDns = MyCustomValidation(cert, message.RequestUri.Authority);
if (!isValidDns)
{
throw new AuthenticationException($"Ssl certificate validation failed when trying to connect to {message.RequestUri}, Error: {SslPolicyErrors.RemoteCertificateNameMismatch}. Built-in validation failed and custom validation failed.");
}
return true;
}
throw new AuthenticationException($"Ssl certificate validation failed when trying to connect to {message.RequestUri}, Error: {errors}.");
}
var client = new HttpClient(httpClientHandler);
```
MyCustomValidation being mostly inspired by how it is implemented in mono.
checkServerIdentity method.
I would prefer to use the built-in logic though. I'm not familiar with how releases are happening when it comes to patch update so can you point me to some roadmap where I can check for this?
@ydekova what happens when you set DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0? How does your application behave?
With the code above, that makes it work for me on Fedora.
@tmds I set it in /etc/environment:
/etc$ cat environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
ASPNETCORE_ENVIRONMENT=Staging
DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0
I restarted the dot net application and there was no change. The same error as before.
@ydekova that's unexpected - @tmds do you think there are differences in libcurl behavior here?
@davidsh I agree that we should fix it in 2.1.x. Where would the fix go? Networking layer or do we have to convince @bartonjs to relax comparison in Security layer?
It's not really "relax" it's "match everyone else". ~4 line change in crypto native.
Let's do it then. Thanks @bartonjs!
@tmds do you think there are differences in libcurl behavior here?
curl does the case-insensitive compare.
I restarted the dot net application
@ydekova if you do this with /etc/environment you need to reboot your system. You can launch your app as DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 dotnet myapp.dll.
@tmds can you tell me what this setting (DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER) does exactly and how it will affect the app's behavior? Does it affect the SignalR support? Not sure if its OK to change it globally, we have quite a few net core apps running on this instance, all of them are now upgraded to 2.1
@ydekova just change it locally and run the app with the setting. That should not be a problem, should it?
The setting will disable our new networking stack implementation - see more here.
@karelz 'locally' it's windows so the problem does not exist. Aside from this, testing this on a Linux machine is not a problem yes, it does fix the problem on our test machine.
What I was asking is are there some functionalities which rely on the new handler in 2.1. (i.e. SignalR) or is the difference only related to performance? We haven't implemented our socket server yet but we are planning to, this is why I need to know.
Thanks
By locally I meant in env window, then run the app, so the env vars apply only "locally", not machine-wide.
There are subtle differences, but the main ones are performance and HTTP/2 support (which SocketsHttpHandler) does not have.
@karelz I would prefer to use the new sockets handler because otherwise, we encounter the memory leak problem. In my code above this would mean using "new SocketsHttpHandler()" instead of "new HttpClientHandler()" right?
@ydekova no, SocketsHttpHandler is default, unless you opt out via the env variable we suggested above. You don't need to new up SocketsHttpHandler, using HttpClientHandler will automatically use it under the hood.
we encounter the memory leak problem
Which memory leak? Is that reported above or another bug?
@karelz Thanks for the clarification. We had a failure of our production server (Ubuntu 16) yesterday some 12 hours after the upgrade and the most probable reason, for now, seems to be "out of memory".Still investigating but my first guess was that there might be something related to this issue https://github.com/dotnet/corefx/issues/28241 that could have been caused by my "hack". We have an API with some heavy integrations, making a lot of https requests to external systems.
dotnet is definitely the greediest process on this server when it comes to memory, but it was stable for quite some time before the upgrade.
Could there be something important I might be missing as configuration? Should I dispose of the Handler and the Client object explicitly?
Hard to say, the easiest thing is to use memory profiler to find out what's happening. We should take that discussion into separate issue. This one is tracking a proper fix of the issue you filed originally.
@karelz This actually is related to the current topic in a way. With my code above I couldn't find a way to use the HttpClientFactory and intercept the certificate validation since I can't access the HttpClientHandler object. Creating a new handler for each request definitely affects the performance in a bad way. On the other hand, I'm not sure if using a singleton HttpClient is recommended. What is the right approach in this situation?
Perhaps I could create a new handler for each endpoint (domain) that expires after some time?
@ydekova I know it is a bit related, but the main purpose of this issue is to address the bug in 2.1 servicing, not to troubleshoot problems with potential workarounds. Let's focus on that here.
BTW: I expect that you can create HttpClientFactory with your handler as well. Creating handler per request is very inefficient and not recommended. Using a singleton is fine - you may run into troubles if DNS changes, but you can workaround it on .NET Core 2.1 by setting SocketsHttpHandler.PooledConnectionLifetime or recycling the HttpClient on regular basis (e.g. every 2 minutes) - both are the techniques that HttpClientFactory uses.
Is there any quick way to fix this bug?
@xiazen, to sum-up there are two workarounds in this thread.
Depending on how serious you are about the certificate validation (and you should be) writing your own validation may not exactly be "quick".
And/or wait for our future servicing release (you can even dogfood early servicing branch builds if that helps).
Same issue for me, Windows Server 2012,
happens in Service Fabric cluster, trying to setup communication between services over https.
With HttpClientFactory default mode:
Error: url: https://psaqueries.mitp.services:8162/PsaQueriesAPI/ ,
ex: The SSL connection could not be established, see inner exception.,
data: System.Collections.ListDictionaryInternal
inner: The remote certificate is invalid according to the validation procedure.
inner data: System.Collections.ListDictionaryInternal at mitp.PSA.Facade.QueriesHandlers.CountReviewTicketsQueryHandler.Handle(CountReviewTicketsQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.PSA.Facade\QueriesHandlers\CountReviewTicketsQueryHandler.cs:line 44
at mitp.Queries.QueriesHandlers.GetReviewDetailsInfoQueryHandler.GetReviewTicketsCountAsync(GetReviewDetailsInfoQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.Queries\QueriesHandlers\GetReviewDetailsInfoQueryHandler.cs:line 34
at mitp.Queries.QueriesHandlers.GetReviewDetailsInfoQueryHandler.Handle(GetReviewDetailsInfoQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.Queries\QueriesHandlers\GetReviewDetailsInfoQueryHandler.cs:line 25
at mitp.WebApiCore.Controllers.Reviews.ReviewsController.GetReviewInfoAsync(Int32 reviewId) in C:\d\mitp\myITprocess\mitp.Services\mitp.WebApiCore\Controllers\Reviews\ReviewsController.cs:line 52
at lambda_method(Closure , Object )
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at mitp.WebApiCore.System.Middleware.MaintainCorsHeadersMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
With AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
Error: url: https://psaqueries.mitp.services:8162/PsaQueriesAPI/
ex: An error occurred while sending the request.,
data: System.Collections.ListDictionaryInternal
inner: Error 12175 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'A security error occurred'.
inner data: System.Collections.ListDictionaryInternal at mitp.PSA.Facade.QueriesHandlers.CountReviewTicketsQueryHandler.Handle(CountReviewTicketsQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.PSA.Facade\QueriesHandlers\CountReviewTicketsQueryHandler.cs:line 44
at mitp.Queries.QueriesHandlers.GetReviewDetailsInfoQueryHandler.GetReviewTicketsCountAsync(GetReviewDetailsInfoQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.Queries\QueriesHandlers\GetReviewDetailsInfoQueryHandler.cs:line 34
at mitp.Queries.QueriesHandlers.GetReviewDetailsInfoQueryHandler.Handle(GetReviewDetailsInfoQuery message, CancellationToken cancellationToken) in C:\d\mitp\myITprocess\mitp.Services\mitp.Queries\QueriesHandlers\GetReviewDetailsInfoQueryHandler.cs:line 25
at mitp.WebApiCore.Controllers.Reviews.ReviewsController.GetReviewInfoAsync(Int32 reviewId) in C:\d\mitp\myITprocess\mitp.Services\mitp.WebApiCore\Controllers\Reviews\ReviewsController.cs:line 52
at lambda_method(Closure , Object )
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at mitp.WebApiCore.System.Middleware.MaintainCorsHeadersMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)
It has been fixed in 2.1 servicing in PR dotnet/corefx#30553, closing.
It should be part of 2.1.3 release.
I have web server with wild card certificate ( *.mdrinc.com) running on port 1060.
When I try to reach server using HttpClient I get "_The remote certificate is invalid according to the validation procedure_" error . With HttpClient handler with ServerCertificateCustomValidationCallback I see "_SslPolicyErrors.RemoteCertificateNameMismatch_".
Same program works with AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.2.100
Commit: b9f2fa0ca8
Runtime Environment:
OS Name: ubuntu
OS Version: 18.04
OS Platform: Linux
RID: ubuntu.18.04-x64
Base Path: /usr/share/dotnet/sdk/2.2.100/
Host (useful for support):
Version: 2.2.0
Commit: 1249f08fed
.NET Core SDKs installed:
2.2.100 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]_
[Subject]
[email protected], CN=*.mdrinc.com, OU=Dev, O=mdrinc, L=Bangalore, S=Karnataka, C=IN
Simple Name: *.mdrinc.com
Email Name: [email protected]
DNS Name: *.mdrinc.com
[Issuer]
[email protected], CN=CA.HCPF.COM, OU=CA, O=mdrinc, L=Bangalore, S=Karnataka, C=IN
Simple Name: CA.HCPF.COM
Email Name: [email protected]
DNS Name: CA.HCPF.COM
[Serial Number]
01
[Not Before]
04/12/18 2:17:36 PM
[Not After]
03/12/20 2:17:36 PM
https://gist.github.com/madhub/39c980ff010ba3888e54e8f1527ced6b#file-ssltest-cs
```c#
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using System.Text;
class Program
{
public class MyHttpClientHandler : HttpClientHandler
{
public MyHttpClientHandler()
{
this.ServerCertificateCustomValidationCallback = delegate (HttpRequestMessage message,
X509Certificate2 x509Cer2,
X509Chain chain,
SslPolicyErrors errors)
{
bool validationStatus = true;
if ((errors & SslPolicyErrors.RemoteCertificateChainErrors) != 0)
{
Console.WriteLine("SslPolicyErrors.RemoteCertificateChainErrors");
validationStatus = false;
}
if ((errors & SslPolicyErrors.RemoteCertificateNameMismatch) != 0)
{
Console.WriteLine("SslPolicyErrors.RemoteCertificateNameMismatch");
validationStatus = false;
}
else if ((errors & SslPolicyErrors.None) != 0)
{
Console.WriteLine("SslPolicyErrors.None");
validationStatus = true;
}
Console.WriteLine(x509Cer2.ToString(true));
return validationStatus;
};
}
}
// this works
public static void WithAppContextSwitch(string hostUrl)
{
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
try
{
using (var httpClient = new HttpClient())
{
var response = httpClient.GetAsync(hostUrl).GetAwaiter().GetResult();
System.Console.WriteLine(response.StatusCode);
}
}
catch (Exception exp)
{
Console.WriteLine("Exception " + exp);
}
}
// throws RemoteCertificateNameMismatch exception
public static void WithOutAppContextSwitch(string hostUrl)
{
try
{
var httpHandler = new MyHttpClientHandler();
using (var httpClient = new HttpClient(httpHandler))
{
var response = httpClient.GetAsync(hostUrl).GetAwaiter().GetResult();
System.Console.WriteLine(response.StatusCode);
}
}
catch (Exception exp)
{
Console.WriteLine("Exception " + exp);
}
}
static void Main(string[] args)
{
string hostUrl = "https://msrv4.mdrinc.com:1060";
// this works
WithAppContextSwitch(hostUrl);
// throws RemoteCertificateNameMismatch exception
WithOutAppContextSwitch(hostUrl);
}
}
```
@madhub This issue is already closed. If you have a new bug to report, please open up a new issue. Thanks.
@ydekova
checkServerIdentity
can you please provide your full code workarounds for me how to solve this problem in dotnet core 2.2.
Most helpful comment
We should consider fixing this for servicing release/2.1.x as well.
cc: @karelz