Runtime: SSL connection could not be established(The remote certificate is invalid according to the validation procedure)

[viveknaragude]

Environment:-
Docker: Windows using Linux containers
OS: Window 10
Microsoft.AspNetCore.App:2.1.1
Microsoft.NETCore.App: 2.10
Docker Image: microsoft/dotnet:2.1-aspnetcore-runtime

Hi ,
I have no idea why this happens but this is an issue that appears when i call webapi hosted in docker container also i follow Hosting ASP.NET Core Images with Docker over HTTPS but not working .

Please help me if i miss anything or fix this issue.

Generate Log:-

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: '[PII is hidden]'.
 ---> System.IO.IOException: IDX20804: Unable to retrieve document from: '[PII is hidden]'.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

[karelz]

Is it happening on the client side?
It looks like there is some invalid certificate. Did you check it?
Are you able to reproduce the same problem with brand new HelloWorld-style app? If not, can you create minimal repro that anyone can try?

cc @bartonjs @wfurt

[bartonjs]

This sounds like https://github.com/dotnet/corefx/pull/32455, which is/will-be fixed in 2.1.6.

In the short term, setting an environment variable SSL_CERT_DIR=/dev/null has a good chance of being a functional workaround.

[viveknaragude]

Not sure happening on which side but its on server side
Brand new HelloWorld-style app working well(Create new webapi using vs2017)
Also try to setting an environment variable SSL_CERT_DIR=/dev/null but not working
2.1.6 image available for docker ?

[karelz]

In that case I think we will need a repro to make further progress. Something we can run locally.
Take your app and try to minimize it. The call stack is from client side, which means you can focus on outgoing requests. Ideally create a repro from console app, without any ASP.NET parts.

[viveknaragude]

Hi @karelz @bartonjs @wfurt
Problem in this code

1.startup.cs(ConfigureServices method))
```c#
services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.Audience = "aud";
options.Authority = "https://demo.identityserver.io:44333";
options.TokenValidationParameters.ValidIssuers = "issuers";
});

2)startup.cs (Configure method)
```c#
         app.UseAuthentication();

Actually your Authority url is https and your code is working on Linux container in docker then this Exception is occur is any way to handle this exception (Microsoft.AspNetCore.Authentication)

[karelz]

@viveknaragude do you have a repro we can run locally from any machine?

[karelz]

@viveknaragude ping?

[viveknaragude]

Hi @karelz

Its fix

Actually the case is if your .netcore application hosting on IIS server bind with ssl certificate and on that same machine docker is install and other .detcore application hosting using kestrel server in docker container they access iis hosted application url inside second app like (https:///...) that time error is occur its only happen on Linux container not on windows container.

Solution for me:
Add same certificate(.crt) file used by IIS hosting application in Linux container

[viveknaragude]

@karelz ping?

[wfurt]

I'm not sure that is actual problem @viveknaragude. Unless your app uses certificate signed by somebody trusted by the container you need to establish the trust somehow. If anything that seems like container configuration problem. Do I miss something?

[karelz]

Its fix
Actually the case is if your .netcore application hosting on IIS server bind with ssl certificate and on that same machine docker is install and other .detcore application hosting using kestrel server in docker container they access iis hosted application url inside second app like (https:///...) that time error is occur its only happen on Linux container not on windows container.

I didn't quite understand what you meant here.
Check for configuration errors. If you think it is problem in the platform, please create repro steps from scratch (ideally with minimal dependencies - e.g. no containers, etc.).

[gauravagarwal28]

I am encountering the same issue.

Hi @karelz

Its fix

Actually the case is if your .netcore application hosting on IIS server bind with ssl certificate and on that same machine docker is install and other .detcore application hosting using kestrel server in docker container they access iis hosted application url inside second app like (https:///...) that time error is occur its only happen on Linux container not on windows container.

Solution for me:
Add same certificate(.crt) file used by IIS hosting application in Linux container

However, this fix is not working for me.

[karelz]

There is not enough information to make the issue actionable, closing.
If there is a clear repro for us to look at, feel free to reopen (or create new issue with full details). Thanks!

[ewassef]

Here is a simple repro:

private static readonly HttpClient Client = new HttpClient(new HttpClientHandler { UseCookies = false });
var req = new HttpRequestMessage();
req.RequestUri = new Uri("https://www.cnn.com/");
var response = await Client.SendAsync(req);

This is giving me the same error..

[wfurt]

What is your environment @ewassef ??? I did quite test with your fragment and everything works fine for me.

Can you check that your system time is correct and "issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3" is in your trust store?
(if this is container it needs to be within that container)

You can also try different client (like wget or curl)

[ewassef]

@wfurt
Im on a windows machine and this snippet works if you run a .net core 2.2 console application. If you try this from a mvc core running kestrel , it fails everytime. I am configured to run in Linux docker containers, but this occurs when debugging on a windows 10 machine

The site comes up in all browsers and in the console app like i mentioned... only in a kestrel based mvc app

```c#
public class Program
{
public static void Main(string[] args)
{
CreateWebHostBuilder(args).Build().Run();
}

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseContentRoot(Directory.GetCurrentDirectory())
            .UseKestrel(opts => { })
            .UseStartup<Startup>();
}

and the code is in the startup.cs in an app.Run :
```c#
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            app.Run(async ctx =>
            { ....

[karelz]

In that case it is something specific to your environment setup in debug mode.
Check your firewall settings. Try it on freshly installed VM to see what else could cause it.

[ewassef]

@karelz What do you recommend i look at? I am running the console app and the web app on the same machine and one works while the other doesn't. This isnt a VM, but i can try one if you think the problem can be isolated that way somehow.

I even tried running the mvc app in a docker container based off of microsoft/dotnet:2.2-aspnetcore-runtime container and have the same issue.

This looks to be tightly coupled with using the HttpClient in an MVC host. the same code works if i add :
handler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
(Which is expected, but not a suitable solution).

I believe the the mvc code is perhaps injecting its own ServerCertificateCustomValidationCallback which has issues while the netcoreapp version does not.. i am combing through the source code to see if i can find something, but your help would be very appreciated

[davidsh]

@Tratcher Can you assist here? Perhaps this issue needs to be moved to ASP.NET repo? The problems are tightly coupled with ASP.NET Core. Thx.

[Tratcher]

@ewassef AspNetCore/MVC host does not mess with the ServerCertificateCustomValidationCallback. This may be an issue with the container infrastructure. Try simplifying the web app repro by removing all of the code in Program.Main and calling HttpClient directly from there.

[wfurt]

I'm not too familiar with ASP.NET @ewassef but if you post complete repro I can run, I can see if I get repro as well. That would show if there is something with the combination or if there is something truly environmental. I would suggest to stay out of Docker for now as it decreases complexity.

BTW your second example uses https://demo.identityserver.io:44333
What CA do you use to sign the certificate?

[Cephei]

I'm having the same issue running a dotnet core app. Targeting framework 2.2. When debugging in VS Code everything is fine. My HttpClient (within the Azure DocumentDB Client) can connect to my CosmosDB emulator.

When I run the app within a Docker container using microsoft/dotnet:2.2.103-sdkto build and runtime microsoft/dotnet:2.2.1-runtime, The DocumentClient fails, stating: HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

[wfurt]

That seems like different problem @Cephei . It looks like container does not have proper trust established. What server do you connect to and what certificate does it use?

[Cephei]

@wfurt My mistake. I'm connecting to a local Windows VM. The VM is running CosmosDB emulator over port 8081. I've taken the necessary steps to export the emulator's certificate and imported into my Mac's Keychain. I've assigned it to Trust Always. I can view the emulator's Data Explorer just fine in Chrome - no certificate issues; it states it's trusted.

I can't reason why the app, when run within a Docker container cannot establish the SSL connection. I'm assuming Docker must somehow trust the host's certificate stores? I should note that this used to work just fine. Perhaps this is an issue with Docker for Mac.

[wfurt]

Docker provides isolation - that is one key benefit. Importing certificate to host trust store (Mac, Windows or Linux) does nothing for app running in the container.
The app cannot (and should not) access the host.
What OS do you run in the container? You need to add the CA certificate there @Cephei

[rammaram06]

I'm trying to build a .net core project using dotnet core sdk image getting same error..

I tried adding the CA cert but no luck

Import-Certificate -FilePath C:/temp/CA-Bundle.crt -CertStoreLocation Cert:/LocalMachine/Root/

The SSL connection could not be established, see inner exception.
[09:12:01][Step 2/4]     The remote certificate is invalid according to the validation procedure.
[09:12:01][Step 2/4]   Retrying 'FindPackagesByIdAsyncCore' for source 'https://artifactory.xxx.xx/artifactory/api/nuget/nuget/FindPackagesById()?id='DocumentFormat.OpenXml'&semVerLevel=2.0.0'.

[ewassef]

I think the docker stuff might be causing some confusion, but I am not in Docker ... i am running MVC core app using the exe directly on a windows machine. This code is running directly in the PRogram.cs already and the same code works in a .net core 2.2 console application.

the code variants were created via the built in templates in visual studio 2017 and still have the default plumbing

[Tratcher]

@ewassef can you upload that app for confirmation?

[ewassef]

Heres the mvc site... thanks for taking a look
HttpClientSample.zip

[Tratcher]

@ewassef you example works fine for me running either IIS Express or Kestrel from VS. Note you did have docker enabled for your project as a 3rd option. I couldn't get it to even run docker, but when I created a new empty web template with docker enabled and pasted your sample in that also worked.

This definitely smells of an environmental issue.

[wfurt]

That command only imports certificate but that does not necessarily means it would be trusted as CA @rammaram06 Depending on OS/Version you need to import to OS trusted CA store.
Also this was originally discussion about Docker.
It may be better to open separate issues for non-docker cases.

[twsl]

Can confirm the issue for self signed certificates in docker containers.
Create two ASP.NET Core 2.2 projects and let VS create two self signed developer certificates (or do it manually and adjust the secrets.json), then request any site via their external ip, same error

[bartonjs]

@twsI Any self-signed certificate would certainly be considered untrusted (unless you manually trusted it)...

[alexanderbikk]

Hi @twsI
I have exactly the same issue. How do you solve it?

Hi, @bartonjs
Do you have any suggestions to do it for local environment?

[alexschultz]

same problem here @twsI

ASP.Net Core 2.2 app running in Docker via Docker-Compose trying to connect to local host Cosmos emulator using host IP address.

I feel like this shouldn't be this difficult! I've tried everything I can think of. I even tried creating an nginx proxy container so I could connect over http and not https. I can get to the web gui for the emulator through the proxy, I can even connect through the proxy if I run the app locally and point the DocumentClient to the proxy container instead of localhost. But for some reason when I try to connect to the emulator from within a container, it just hangs and eventually it cancels the async task. I've also tried to override the default HttpHandler in the DocumentClient to ignore SSL validation errors but I get the same issue.

Is there some reason why the emulator can only run over ssl? Seems like a lot of people are running into this problem because there is no way to run the emulator without https. Another alternative solution would be to be able to run the emulator within a linux container.

[twsl]

@alexanderbikk @alexschultz I removed https support and decided to handle them outsite of the docker environment. this is far from perfect, but it's working.
The ideal solution would be an easy way to specify, which certificates to accept - even if they are self signed.

[gelihu]

I am writing a .net core console application which connects to a RabbitMQ broker. TLS is enabled on the RabbitMQ server and it expects authentication. Clients can be authenticated using Username/Password. I connected to server using OpenSSL to check TLS version, it is SSL3/TLS1.0 and it mentioned some valid certificate issuers too. I added a certificate(CRT format) issued by one of the valid issuers to root while building the Docker image. But, when I ran the container, it is throwing this kind of exception: (The remote certificate is invalid according to the validation procedure.) ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure..

I even generated a self signed certificate with the expected subject or issuer information. I even changed security protocol manually like this System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls;

Could you please tell me some scenarios when this kind of exception could be thrown?

Edit: It works on my machine as a console app, but when I containerize the app this happens!

[davidsh]

I even generated a self signed certificate with the expected subject or issuer information. I even changed security protocol manually like this System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls;
Could you please tell me some scenarios when this kind of exception could be thrown?
Edit: It works on my machine as a console app, but when I containerize the app this happens!

First, I don't recommend commenting on an old closed issue. It you are having problems with the current version of .NET Core, you should open a new issue.

Second, the ServicePointManager class is effectively a no-op in .NET Core. So, modifying the .SecurityProtocol property isn't going to do anything.

You mention that you are using TLS/SSL. But are you using the SslStream class or the HttpClient class?

Getting this error message:
"System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.."

can have a lot of different root causes. It could be the result of different problems with the received server TLS certificate. The best way to figure that out is to add some code to your application and get the detailed errors from the callback.

See dotnet/corefx#35897 for examples of diagnosing this error.

[wfurt]

how do you establish trust in docker @gelihu ? Self-signed certificates will not be trusted by default. It may be worth of opening new issue with details as @davidsh suggested.

[davidsh]

Also, the biggest thing I've seen is that these problems are typically not trust chain issues. The error message simply reflects that the certificate didn't pass the checks.

It could be an invalid date range, name mismatch or even key usage mismatch. The X509v3 KeyUsage field is especially problematic on Linux versus Windows. Linux due to a recent OpenSsl bug requires that self-signed certificates have an extra flag set (Certificate Signing)in the KeyUsage field.

[gelihu]

Thanks for the response. Actually, Server has a self-generated certificate in certificates chain and the root is untrusted. Exception was specific to that particular server. I modified CertificateValidationCallback to validate for now. It is SslStream by the way.

[viveknaragude]

Hi everyone
Add same certificate(.crt) file in Linux container & add this two line in your docker file
ADD SSL.crt /usr/local/share/ca-certificates/SSL.crt
RUN update-ca-certificates

I hope its help you....

[saitejaprattipati]

Hi i am new to docker can you please tell me how to add .crt file in docker linux container. I have added two lines mentioned above in docker file.

[AblEdge]

it might be helpful
call-a-grpc-service-with-an-untrustedinvalid-certificate