Aspnetcore: HttpClient Exception GetAsync HTTPS when deploy code to K8s

[coderstrong]

Describe the bug

When using HttpClient request GET to Https api I get exception only deploy to K8s. When build docker local on my computer and run it OK not has exception. I try use curl on terminal K8s call api OK.

2020-02-25 16:28:55.950 +07:00 [ERR] Error Trace

System.Threading.Tasks.TaskCanceledException: The operation was canceled.

---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)

at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)

at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)

--- End of stack trace from previous location where exception was thrown ---

at System.Net.Security.SslStream.ThrowIfExceptional()

at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)

at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)

at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)

at System.Net.Security.SslStream.<>c.b__65_1(IAsyncResult iar)

at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Test.Controllers.HomeController.Index() in /src/Test/Controllers/HomeController.cs:line 29

To Reproduce

C# using (var client = new HttpClient()) { var _test = await client.GetAsync("https://domain.d"); if (_test.IsSuccessStatusCode) return Ok(await _test.Content.ReadAsStringAsync()); }
Build docker and deploy to K8s

Further technical details

• ASP.NET Core 3.1 Assembly System.Net.Http, Version=4.2.2.0

[javiercn]

@coderstrong thanks for contacting us.

I imagine you are using HTTPS within your K8s cluster? It is likely that the issue is that the Cluster CA is not configured in the pod/application to be a trusted certificate.

Yo can go about it in a few ways:

• Add a custom certificate validation callback:
  
  
  
  • https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclienthandler.servercertificatecustomvalidationcallback?view=netframework-4.8
  
  
  • That should allow you to validate the SSL certificate chain goes up to the CA certificate chain.
  
  
• Install the CA Certificate in the pod in a way that .NET Core believes it is a trusted cert.
  
  
  
  • Not sure how you would go about this or if it is even possible. I recommend you ask at dotnet/runtime to see if it is possible.
  
  

[javiercn]

/cc @rynowak @davidfowl in case they have other ideas.

[coderstrong]

When i use curl call API log is
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
But curl can bypass with curl params -k. Use Go call API can bypass, but use HttpClient try bypass is failse how can bypass this case

[blowdart]

Can you clarify something for me, is the app inside the pod making a call to another pod, or to another service within the same pod, or something different?

As @javiercn rightly points out this is a typical HTTPS error, and it's not really docker related, it would happen if you put these on physical machines too if each machine was only using self signed certs.

[coderstrong]

Hi @javiercn. I tried add custom certificate validation callback but it not working, I think it is a bug from
HttpClient
```c#
using (var handler = new HttpClientHandler())
{
// OR use httpClientHandler.ServerCertificateCustomValidationCallback = (message, cert, chain, errors) => { return true; };
handler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
using (var client = new HttpClient(handler))
{
var _test = client.GetAsync("https://domain.d").Result;

if (_test.IsSuccessStatusCode)
  _logger.LogInformation("Response: " + _test.Content.ReadAsStringAsync().Result);

}
}
// all not working

Good news I try Install the CA Certificate in image docker it working. In Dockerfile I config 

COPY ["DigiCertSHA2SecureServerCA.crt", "/usr/share/ca-certificates/extras/"]
RUN echo "extras/DigiCertSHA2SecureServerCA.crt" >> /etc/ca-certificates.conf && update-ca-certificates -v -f
```
But if custom certificate validation callback it work then will be better

Thank @javiercn and @blowdart for support!

[blowdart]

If this is an actual certificate, as seems to be the case from it being from digicert then installing the CA certificate is the right way to do this. You should not override validation, that's for cases where you are using self signed certs.