Runtime: The remote certificate is invalid according to the validation procedure

[lukos]

Running a dotnet core 2.2 web api on Ubuntu 18.04 Docker image (mcr.microsoft.com/dotnet/core/aspnet:2.2-bionic). I am using MailKit (https://github.com/jstedfast/MailKit) to send an email and it works fine when I run locally on Windows 10 but fails on Linux:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure

Since it works on Windows OK, it seems that there is perhaps an OpenSSL issue, maybe due to 1.0 vs 1.1? MailKit calls into SslStream.AuthenticateAsClient, which is the start of the stack that fails. Since this is an email relay, I need some other way of debugging that rather than just that https works generally (which it does with curl).

I have tried the following to no avail:
1) Downloaded and updated the root CA stack using update-ca-certificates (no difference)
2) Used curl from the container itself to access the hello world lets encrypt web site to see if there is a problem with LE specifically (works fine)
3) Set environment variables for SSL_CERT_DIR (/usr/lib/ssl/certs) and SSL_CERT_FILE (/usr/lib/ssl/cert.pem) after symlinking the downloaded CA file into /usr/lib/ssl (no difference).

Can someone please give me some more debugging tips? Thanks.

[vcsjones]

I'm not familiar with MailKit, but after looking at their docs, it looks like they use the same ServerCertificateValidationCallback that's used elsewhere in .NET. You could then theoretically implement a validation callback that does additional logging, which should let you examine any validation errors. Something like:

static void Main(string[] args) {
    MailKit.Net.Smtp.SmtpClient client = default; //setup client
    client.ServerCertificateValidationCallback = (sender, cert, chain, errors) => {
        // Log errors
        // Log chain
        // Log cert
        return errors == SslPolicyErrors.None;
    };
    // Do thing with client.
}

You could also attempt to connect to the mail server using openssl directly and see what it produces, something like:

openssl s_client -connect smtp.example.com:587 -starttls smtp

[lukos]

So I tried with openssl and that looked to work OK. I didn't really know what I was doing afterwards but I could start the EHLO process with no issues.

I logged the results of the ServerCertificateValidationCallback, which shows RemoteCertificateChainErrors, the chain is shown below and it looks complete i.e. normal cert, intermediate and root but I don't know if there is an easy way to see this chain more like a browser shows is? I also returned "true" to ignore the errors and the message is sent without a problem.

[Subject]
  CN=ms9.ssmx.net

[Issuer]
  CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US

[Serial Number]
  031FC5DBAC17B9EB28B798EC5120EF57608E

[Not Before]
  11/15/2019 16:04:59

[Not After]
  02/13/2020 16:04:59

[Thumbprint]
  CE402F9055B9888E00F2525CC3F5DF0835D67BCA


[Subject]
  CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US

[Issuer]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Serial Number]
  0A0141420000015385736A0B85ECA708

[Not Before]
  03/17/2016 16:40:46

[Not After]
  03/17/2021 16:40:46

[Thumbprint]
  E6A3B45B062D509B3382282D196EFE97D5956CCB


[Subject]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Issuer]
  CN=DST Root CA X3, O=Digital Signature Trust Co.

[Serial Number]
  44AFB080D6A327BA893039862EF8406B

[Not Before]
  09/30/2000 21:12:19

[Not After]
  09/30/2021 14:01:15

[Thumbprint]
  DAC9024F54D8F6DF94935FB1732638CA6AD77C13

[lukos]

OK, so now I have some more information. I logged the chain element status and I saw this for my top level certificate:

unable to get certificate CRL

This should give me more to dig into.

[vcsjones]

If you are getting RemoteCertificateChainErrors, then take a look at chain.ChainStatus array to determine what parts of the chain is invalid.

[lukos]

Hi @vcsjones that's the error I posted above.

So apparently Lets Encrypt do not include a CRL for the server cert but use OCSP stapling instead so perhaps a better question is: Should the SSL connection be failing due to a CRL missing in the cert and not just fallback to OCSP?

Of course I can leave the server check bypassed or try and ignore that specific error but that sounds a bit smelly to me - I am hoping there is a proper way to get around this?

[davidsh]

Can you try using .NET Core 3.0? There were a lot of fixes to TLS/SSL handling and perhaps one of those fixes might solve your problem.

[wfurt]

Ping @lukos? 2.2 will be out of support this month.

[bartonjs]

In .NET Core 2.2 we didn't have support for OCSP on Linux. We added OCSP on Linux in .NET Core 3.0, so Let's Encrypt chains should cleanly build with revocation now.

If you're seeing otherwise, let us know.

[pawod]

I'm still gettting following errors targeting netcoreapp3.1 :

unable to get certificate CRL
unable to get local issuer certificate
Subject: CN=smtp.gmail.com, O=Google LLC, L=Mountain View, S=California, C=US
Issuer: CN=GTS CA 1O1, O=Google Trust Services, C=US

Minimal code example for repro with MailKit:

var client = new SmtpClient();
client.Connect("smtp.gmail.com", 587, SecureSocketOptions.StartTls);
client.Authenticate(_options.MailAccountName, _options.MailAccountPassword);
var message = new MimeMessage();
message.From.Add(author);
message.ReplyTo.Add(author);
message.To.Add(recipient);
message.Subject = subject;
message.Body = new TextPart(MimeKit.Text.TextFormat.Plain) { Text = text };
await client.SendAsync(message);

when checking against openssl everything is fine with the certs

openssl s_client -CApath /etc/ssl/certs/ -connect smtp.gmail.com:587 -starttls smtp
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
   i:C = US, O = Google Trust Services, CN = GTS CA 1O1
 1 s:C = US, O = Google Trust Services, CN = GTS CA 1O1
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFizCCBHOgAwIBAgIRALAfsBFF7jp2CAAAAAAqsrIwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMDAxMjEwODE0NTdaFw0yMDA0MTQwODE0
NTdaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
Ew1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgTExDMRcwFQYDVQQDEw5z
bXRwLmdtYWlsLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKg4
68gPXFngWV99XtDhpohHtfmkCFriYf6TJnc8kh0t8VnRjjwoDEt0uEkVyiTkAkjZ
cliqHcKf06h87fGmoA4FtnsKuBWhPeYPU76QR88tSmwDRI1TMcRJjRZmkRAypK+E
H3fHySbudMtdUnm5/G7WoWDtxKp8lAG8L2CHwgmDpgi2n/9TnQ2iZpu+L2SgDbbm
RXdSTo5RhkfDVZ8mVXqT61AFLX62x+0+GzoAh50Ye5145mC1ru7nH89Qno6xf6n+
XhTFmlJNbrORUCD5sHsehkEI4D/e+8O5ndIY7yz9G3uYwGhSm7kUBL5BLtYYH/1H
PyyN2WFiJ064xwcIKNECAwEAAaOCAlQwggJQMA4GA1UdDwEB/wQEAwIFoDATBgNV
HSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTbio0y2MGh
pZVCc2D6W2F1vaVdczAfBgNVHSMEGDAWgBSY0fhuEOvPm+xgnxiQG6DrfQn9KzBk
BggrBgEFBQcBAQRYMFYwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLnBraS5nb29n
L2d0czFvMTArBggrBgEFBQcwAoYfaHR0cDovL3BraS5nb29nL2dzcjIvR1RTMU8x
LmNydDAZBgNVHREEEjAQgg5zbXRwLmdtYWlsLmNvbTAhBgNVHSAEGjAYMAgGBmeB
DAECAjAMBgorBgEEAdZ5AgUDMC8GA1UdHwQoMCYwJKAioCCGHmh0dHA6Ly9jcmwu
cGtpLmdvb2cvR1RTMU8xLmNybDCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2ALIe
BcyLos2KIE6HZvkruYolIGdr2vpw57JJUy3vi5BeAAABb8diMFoAAAQDAEcwRQIg
Z/KVgKafdXDy+ApADniOxmKR/vwjwkGX3vFoSXQSKlkCIQCLumhFIcD0T1QA4auf
2pD2tVXvfQPRdoKihSfgVRCcgQB2AF6nc/nfVsDntTZIfdBJ4DJ6kZoMhKESEoQY
dZaBcUVYAAABb8diMH4AAAQDAEcwRQIhAJjOvO183KgUETiIpAt99NAvZPrY1KOp
xF49fkGWic/pAiBdLafQlkyz25SQvXu8rcz7pFH5x04L4cW4WtTdm6WpBDANBgkq
hkiG9w0BAQsFAAOCAQEAUVjgtKD1pxkisLV9VStkhyqpgd9JJ/l1W92AcF/RZ3Yf
/72ARCDXpfha3IMmVpqfmKUP03Yo7xM3GBHjgPxRtq1K/Hd8xk/ovVZ0ihugCwf2
8FXpEpcbuPm/QBdL9uZg8B+kT3UBsYggN9qrEvXLfODL6HwRHNMN1K5xaJyeQOf8
JZ8HX/JwTQS7iHGLajv9cT3ABUq4En8hy3KdGscV+fPs/qLdmYCDHpTmCYxW7G7q
Q0NKvXY0waiffFy0hxSnhMLoU4n/b4Kiv1AeWb1ZgtO4IY8O5f6wdQIXD/fcEjEr
UDMjAw4Ooy4Szy8gw9eXpot8yaFeGqwqyQML4MF28w==
-----END CERTIFICATE-----
subject=C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com

issuer=C = US, O = Google Trust Services, CN = GTS CA 1O1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3271 bytes and written 429 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 SMTPUTF8

help(?)

[davidsh]

@bartonjs

[jeroenheijmans]

Landed here after finding that this dotnet/aspnetcore issue and its resolutions (not its Stack Overflow thread on dotnet dev-certs https --trust) did not yet work for me on Ubuntu 20.04.

Adding some info here, as it might help with an easy repro. What I did was:

1. Install Ubuntu 20.04, 64bit
2. Install dotnet sdk (--version gives 3.1.300)
3. Clone my repository with IdentityServer4 and ASP.NET Core Web API in one project
4. Add IdentityModelEventSource.ShowPII = true; // DEV ONLY!! to Configure(...)
5. Run dotnet dev-certs https --trust
6. Run it with dotnet run
7. Open https://localhost:5001 in Firefox

Because the instructions in Stack Overflow did not (yet) work for me somehow (even after sudo update-ca-certificates the verification openssl verify localhost.crt keeps giving me the same error as before), the story continues like this for me:

1. You get an SSL/TLS warning, via "Advanced" accept "the risk" and continue with the untrusted cert
2. In my sample app click _"Fetch token for administrator" (this is still fine, the browser cert exception is in place)
3. Click button "GET /endpoint-specific-security/admin-role-only"

Then:

• Result: the API is called, which via HTTP calls the IdentityServer instance on the same https://localhost:5001 but this fails with:

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:5001/.well-known/openid-configuration'.
 ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:5001/.well-known/openid-configuration'.
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

To my limited understanding it seems that my API is calling another "server" and is not trusting that server's SSL certificate. So I presume this is the same problem as OP has, only in their case it is "MailKit" that is running a cert that's not okay for the calling client (similar to my 'client' calling into IDS4)?

I'm adding this information because:

• _either_ (and most likely) my problem is specific to my case, and for most others @davidsh's instructions (and this SO answer) in the other thread will be a good workaround
• _or_ (less likely) my problem is a common one, and the repro above might help track down the issue

I truly hope all this information in my post helps and is not completely off topic. If it _is_ off-topic, then please ignore!

---

What I did (to no avail) to try and fix the certificate (might be a valid workaround for OP even though it didn't work for me?):

1. Create and cd to ~/tmp
2. Create localhost.conffrom this gist
3. openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout localhost.key -out localhost.crt -config localhost.conf
4. openssl pkcs12 -export -out localhost.pfx -inkey localhost.key -in localhost.crt
5. sudo cp localhost.crt /usr/local/share/ca-certificates
6. sudo update-ca-certificates ("1 added")
7. export ASPNETCORE_Kestrel__Certificates__Default__Password=password
8. export ASPNETCORE_Kestrel__Certificates__Default__Path=~/tmp/localhost.pfx

Then dotnet run my repository again (it picks up the .pfx because it'll give an error if the password is wrong), but still get the certificate warnings, both in Mozilla Firefox 76.0.1 and in Google Chrome 83.0.

[wfurt]

I don't think the dotnet dev-certs https --trust works on Linux. If you want system components to trust it, you need to modify system trust @jeroenheijmans . The last sequence looks reasonable but can you post your localhost.crt?
You can also verify the trust with either curl or openssl s_client -connect localhost:5001

[jeroenheijmans]

Thx for the response @wfurt! I appreciate the offer to look at my specific situation, but I don't want to dilute this GitHub issue with super-specific debugging of what is likely my own specific problem.

Your tips were super helpful already though, grabbing the contents of the .crt and the output from openssl should give me enough details to ask a question on Stack Overflow or AskUbuntu about my specific problem.

[pawod]

I'm still gettting following errors targeting netcoreapp3.1 :

unable to get certificate CRL
unable to get local issuer certificate
Subject: CN=smtp.gmail.com, O=Google LLC, L=Mountain View, S=California, C=US
Issuer: CN=GTS CA 1O1, O=Google Trust Services, C=US

Minimal code example for repro with MailKit:

var client = new SmtpClient();
client.Connect("smtp.gmail.com", 587, SecureSocketOptions.StartTls);
client.Authenticate(_options.MailAccountName, _options.MailAccountPassword);
var message = new MimeMessage();
message.From.Add(author);
message.ReplyTo.Add(author);
message.To.Add(recipient);
message.Subject = subject;
message.Body = new TextPart(MimeKit.Text.TextFormat.Plain) { Text = text };
await client.SendAsync(message);

when checking against openssl everything is fine with the certs

openssl s_client -CApath /etc/ssl/certs/ -connect smtp.gmail.com:587 -starttls smtp
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com
   i:C = US, O = Google Trust Services, CN = GTS CA 1O1
 1 s:C = US, O = Google Trust Services, CN = GTS CA 1O1
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFizCCBHOgAwIBAgIRALAfsBFF7jp2CAAAAAAqsrIwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMDAxMjEwODE0NTdaFw0yMDA0MTQwODE0
NTdaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
Ew1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgTExDMRcwFQYDVQQDEw5z
bXRwLmdtYWlsLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKg4
68gPXFngWV99XtDhpohHtfmkCFriYf6TJnc8kh0t8VnRjjwoDEt0uEkVyiTkAkjZ
cliqHcKf06h87fGmoA4FtnsKuBWhPeYPU76QR88tSmwDRI1TMcRJjRZmkRAypK+E
H3fHySbudMtdUnm5/G7WoWDtxKp8lAG8L2CHwgmDpgi2n/9TnQ2iZpu+L2SgDbbm
RXdSTo5RhkfDVZ8mVXqT61AFLX62x+0+GzoAh50Ye5145mC1ru7nH89Qno6xf6n+
XhTFmlJNbrORUCD5sHsehkEI4D/e+8O5ndIY7yz9G3uYwGhSm7kUBL5BLtYYH/1H
PyyN2WFiJ064xwcIKNECAwEAAaOCAlQwggJQMA4GA1UdDwEB/wQEAwIFoDATBgNV
HSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTbio0y2MGh
pZVCc2D6W2F1vaVdczAfBgNVHSMEGDAWgBSY0fhuEOvPm+xgnxiQG6DrfQn9KzBk
BggrBgEFBQcBAQRYMFYwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLnBraS5nb29n
L2d0czFvMTArBggrBgEFBQcwAoYfaHR0cDovL3BraS5nb29nL2dzcjIvR1RTMU8x
LmNydDAZBgNVHREEEjAQgg5zbXRwLmdtYWlsLmNvbTAhBgNVHSAEGjAYMAgGBmeB
DAECAjAMBgorBgEEAdZ5AgUDMC8GA1UdHwQoMCYwJKAioCCGHmh0dHA6Ly9jcmwu
cGtpLmdvb2cvR1RTMU8xLmNybDCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2ALIe
BcyLos2KIE6HZvkruYolIGdr2vpw57JJUy3vi5BeAAABb8diMFoAAAQDAEcwRQIg
Z/KVgKafdXDy+ApADniOxmKR/vwjwkGX3vFoSXQSKlkCIQCLumhFIcD0T1QA4auf
2pD2tVXvfQPRdoKihSfgVRCcgQB2AF6nc/nfVsDntTZIfdBJ4DJ6kZoMhKESEoQY
dZaBcUVYAAABb8diMH4AAAQDAEcwRQIhAJjOvO183KgUETiIpAt99NAvZPrY1KOp
xF49fkGWic/pAiBdLafQlkyz25SQvXu8rcz7pFH5x04L4cW4WtTdm6WpBDANBgkq
hkiG9w0BAQsFAAOCAQEAUVjgtKD1pxkisLV9VStkhyqpgd9JJ/l1W92AcF/RZ3Yf
/72ARCDXpfha3IMmVpqfmKUP03Yo7xM3GBHjgPxRtq1K/Hd8xk/ovVZ0ihugCwf2
8FXpEpcbuPm/QBdL9uZg8B+kT3UBsYggN9qrEvXLfODL6HwRHNMN1K5xaJyeQOf8
JZ8HX/JwTQS7iHGLajv9cT3ABUq4En8hy3KdGscV+fPs/qLdmYCDHpTmCYxW7G7q
Q0NKvXY0waiffFy0hxSnhMLoU4n/b4Kiv1AeWb1ZgtO4IY8O5f6wdQIXD/fcEjEr
UDMjAw4Ooy4Szy8gw9eXpot8yaFeGqwqyQML4MF28w==
-----END CERTIFICATE-----
subject=C = US, ST = California, L = Mountain View, O = Google LLC, CN = smtp.gmail.com

issuer=C = US, O = Google Trust Services, CN = GTS CA 1O1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3271 bytes and written 429 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 SMTPUTF8

help(?)

I finally figured out why this is not working on Linux. The netcore framework does not read from the OS's ca-certificates folder, but from

~/.dotnet/corefx/cryptography/x509stores/ca

Therefore the app did not trust ANY certificates. So downoaded Google's sample PEM file from this page, converted it into a P7B file to make sure the entire certificate chain remains intact

openssl crl2pkcs7 -nocrl -certfile google-roots.pem -out google-roots.p7b

and added the entire chain during app startup

using (var store = new X509Store(StoreName.CertificateAuthority, StoreLocation.CurrentUser))
{
    store.Open(OpenFlags.ReadWrite);

    options.CertificateChainFiles.ToList().ForEach(file =>
    {
        var collection = new X509Certificate2Collection();
        collection.Import(file);
        store.AddRange(collection);
    });
}

[bartonjs]

The ca store directory (and the CertificateAuthority store from code) is just a cache for intermediate certificates, it's not anything that provides trust (that comes from the root directory/Root store, as well as the system root store). So that import changing anything would be rather unexpected, unless that bundle contains some intermediate certificates which weren't provided by the TLS server on the handshake.

[Amberg]

Same issue here with a wildcard Certificate issued by letsencrypt (*.stg.foo.com) :
stackoverflow

Setup:

.NET Core 3.1.5
Ubuntu 20.04 LTS

CURL works:

curl https://admin.stg.foo.com/ -v
ALPN, offering h2
...
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server accepted to use h2
Server certificate:
subject: CN=.stg.foo.com
start date: Jul 8 04:50:10 2020 GMT
expire date: Oct 6 04:50:10 2020 GMT
subjectAltName: host "admin.stg.foo.com" matched cert's "*.stg.foo.com"
issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
SSL certificate verify ok.

The certificate seems to be valid. I can access the URL with my browser, curl or openssl s_client
but

HttpClient fails with: The remote certificate is invalid according to the validation procedure

      using (HttpClient client = new HttpClient())
        {
            var result = await client.GetAsync("https://admin.stg.foo.com/.well-known/openid-configuration");
            Console.WriteLine(await result.Content.ReadAsStringAsync());
            result.EnsureSuccessStatusCode();
        }

  Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNe

[bartonjs]

@Amberg If you have a stable repro, can you register a custom verification callback to report

a) the SslPolicyErrors value
b) chain.ChainStatus.Aggregate(X509ChainStatusFlags.NoError, (f, s) => f | s.Status)

An alternate, more expressive, (b) would be

C# foreach (X509ChainElement element in chain.ChainElements) { Console.WriteLine($"{element.Certificate.Subject}: {element.ChainElementStatus.Aggregate(X509ChainStatusFlags.NoError, (f, s) => f | s.Status)}"); }

[bartonjs]

BTW, when I checked against admin.stg.foo.com, I got a cert for "loadbalancer.localdomain", so it makes sense that one would fail.

[Amberg]

@bartonjs

The correct url is foo.stg.cluyo.ch

The custom verification callback reports:

CN=*.stg.cluyo.ch: NoError
CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US: NoError
CN=DST Root CA X3, O=Digital Signature Trust Co.: NoError

openssl s_client foo.stg.cluyo.ch:443

openssl s_client foo.stg.cluyo.ch:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.stg.cluyo.ch
verify return:1
---
Certificate chain
 0 s:CN = *.stg.cluyo.ch
   i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFUzCCBDugAwIBAgISA+WNqMuQMx9M9sDEux8ZsWODMA0GCSqGSIb3DQEBCwUA
..
wwC7CLmBUuyJXcnf8liKE2Nmzw0eM8M/XdCOiBKUkAOJyQDYj0ZaF6uUMovyoJ7A
CTefvvLu2JzxZzMQa744gRL+7qLgTjs=
-----END CERTIFICATE-----
subject=CN = *.stg.cluyo.ch

issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3221 bytes and written 401 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0AE737A12FE813583D7B65BB8BB71A6E1C489AD99235B2FE3F476F2CA3E745B8
    Session-ID-ctx:
    Master-Key: A3B7CB6AC18DFC43BA8A7384324AE6F4FCCC3567E02BE1232C6A18E1E1F296598A18238863A26F86E9EC27FB686ED57A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
..

    Start Time: 1594260076
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

[bartonjs]

Weird. On 3.1 we use OpenSSL's X509_check_host function. I see it as verifying from Ubuntu 18.04.

To fully match what's going on from .NET, the equivalent check would be

openssl s_client -connect foo.stg.cluyo.ch:443 -servername foo.stg.cluyo.ch -verify_hostname foo.stg.cluyo.ch

At the bottom it should still say "Verify return code: 0 (ok)", which is what I get (though, honestly, I'd prefer it said "62 (Hostname mismatch)", since that'd show it was the library).

There is a /slight/ difference, in that we specify X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS, but that shouldn't apply.

The only thing weird I see is that your CN has an asterisk in it. Often the CN will use a single full hostname but the Subject Alternative Names extension will contain the CN hostname as well as a wildcard.

[Amberg]

I have found the issue:

I had a subdomain name with an underscore (which is not a valid host name) "foo_dev.stg.cluyo.ch"
If I use a - instead everything works as expected.

And I added the host name without wildcard stg.cluyo.ch to the certificate

Some Browsers and "curl" seems to be less strict in this case

[wfurt]

The restriction about '_' comes from OpenSSL AFAIK. I'm wondering if there are some extra flags we can use to suppress it.

[vcsjones]

The restriction about '_' comes from OpenSSL AFAIK. I'm wondering if there are some extra flags we can use to suppress it.

My conclusion when investigating this in #35880 is that, no, this is not configurable.