Aspnetcore: dev-certificates on Linux -- how to get dotnet-to-dotnet comms to work?

[fluffynuts]

Description

I have an asp.net core application with a self-hosted IdentityServer component. I can convince a browser (Chrome and Firefox) to visit the site, but I'm having trouble getting the IdentityServer component to be able to talk back to localhost due to certificate errors, specifically:

fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44303/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44303/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

I realise that certificate installation can differ from distro to distro, however, I've tried this first on my (more native) Gentoo, and also in an Ubuntu vm:

1. export the dev cert to a pfx with dotnet dev-certs https -ep /tmp/dnc.pfx -p SomePassword
2. extract the crt from the pfx with openssl pkcs12 -in /tmp/dnc.pfx -clcerts -nokeys -out /tmp/dnc.crt
3. copy to /usr/local/share/ca-certificates and run update-ca-certificates. In both cases (Gentoo and Ubuntu), see a message about 1 added certificate
   3.5 in the case of Gentoo, I was also able to use certutil to install the crt into my ~/.pki/nssdb; in the case of Ubuntu, this dir does not exist and I'm not sure how to bootstrap it
4. Fire up the asp.net site with angular frontend -- on Gentoo, where certutil worked, Chrome happily visits the site; on both (iirc) I had to add a security exception for Firefox and store, but could get to the site.
5. Attempts to log in fail because all auth requests go through the primary api, which then attempts to contact the same host/port (localhost:44303) for IdentityServer comms -- and that's where the failure occurs with the message above: I can get no further

To Reproduce

Set up an asp.net core project with self-hosted IdentityServer on Linux, such that the config for identity server points back to the same hosting application, using dev certs. Any route requiring authorization will result in a 500 error with the above logging in place. Using dotnet dev-certs https --trust on Windows, same code, no problem.

Expected behavior

I expect to be able to get dotnet-to-dotnet comms working with the dev cert. I'm probably missing something )':

Ubuntu dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a

Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/

Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62

.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Gentoo dotnet info:
.NET Core SDK (reflecting any global.json):
Version: 2.2.103
Commit: 8edbc2570a

Runtime Environment:
OS Name: ubuntu
OS Version: 18.10
OS Platform: Linux
RID: ubuntu.18.10-x64
Base Path: /usr/share/dotnet/sdk/2.2.103/

Host (useful for support):
Version: 2.2.1
Commit: 878dd11e62

.NET Core SDKs installed:
2.2.103 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.2.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

I have also tried with dotnet binaries from https://dotnet.microsoft.com/download/thank-you/dotnet-sdk-2.2.103-linux-x64-binaries

[fluffynuts]

If it would be helpful, I can possibly set up a self-hosting IdentityServer project at some point, given a little time. I can't supply the full code I'm working on because it's for a client.

[maq1823]

@fluffynuts Were you able to resolve this in any way?
I have the same exact problem.

[fluffynuts]

@maq1823 no, I didn't )': I've read that this is an issue within dotnet itself (specifically to do with cert trusting on linux), but I don't know any more about it. I eventually had to stop development under linux and boot into windows as this was a show-stopper preventing any progress on the project.
AFAIK, the only solution right now would be to have a proper certificate issued from an authority and use a hosts file redirection for that domain.

[nickcoad]

I'm having the same issue - I can find a lot of details for how to enable host-to-container SSL trust, but not for container-to-container trust.

[sntnupl]

Facing the same issue. I am trying to setup ASP.NET Core 3.0 Web Api, with self-hosted OpenIddict middleware, for handling OAuth token generation.

In Windows, things work fine, even with dotnet generated TLS certificate.
But things fall apart in Linux environment - I tried with WSL (ubuntu 18.04), as well as in a Ubuntu 18.04 VM. I created local self-signed certificate and setup the system to trust it - but still the issue persists.

There is no issue while trying to access the application URL, Token URL etc.
Once I get the token, and then make a cURL call to one of the protected API endpoints, application tries to internally talk to https://localhost:44320/.well-known/openid-configuration to get the OAuth configuration, and BOOM, it fails estabilish TLS session, as it thinks that the remote certificate is inavlid.

fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)

This is blocking development/deployment of my project in Linux environment. Would really appreciate some one looking/responding to it.

[poke]

Is this about development time or when deploying for production? The dev certs probably shouldn't be used for production purposes.

[sntnupl]

@poke development time. Unless the use cases work we can't be even thinking about prod.

[blowdart]

https://github.com/dotnet/corefx/issues/40725

You need to trust it in libcurl I believe. Regardless, it's a corefx issue not an aspnet one.

[sntnupl]

@blowdart I retried by asking libcurl to trust the self signed certificate:

curl -X --insecure --cacert path/to/self-signed.crt GET \
  https://localhost:44320/api/somesecureaction \
  -H 'Accept: application/json' \
  -H 'Authorization: Bearer <token>'

But the issue still happens:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 --insecure https://localhost:44320/api/Account/UserData  
fail: Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler[3]
      Exception occurred while processing message.
System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:44320/.well-known/openid-configuration'. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslStream.ThrowIfExceptional()
   at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   --- End of inner exception stack trace ---
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()

The problem does not seem to related to libcUrl though, since the following works fine:

curl -X GET https://localhost:44320/.well-known/openid-configuration

The problem happens when JWT middleware internally tries to fetch config from OAuth endpoint, by making HTTPS call to: https://localhost:44320/.well-known/openid-configuration.
I have already setup ASPNET to use my self-signed cert:

       public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder => {
                    webBuilder.UseKestrel(options => {

                        var certificate = new X509Certificate2("localhost.pfx", "<password>");
                        options.Listen(IPAddress.Loopback, 44320, listenOptions => {
                            listenOptions.UseHttps(certificate);
                        });
                    });
                    webBuilder.UseStartup<Startup>();
                });

But somehow that does not stop JWT middleware to throw the error.

[blowdart]

@terrajobst this appears to be a corefx problem rather than aspnet core. Can someone explain how to get corefx to trust a self signed cert?

[terrajobst]

this appears to be a corefx problem rather than aspnet core

That's fair but before we move it let's first verify (because we can't transfer directly and we'll lose some context).

@davidsh @bartonjs could one of you take a look?

[davidsh]

@davidsh @bartonjs could one of you take a look?

I don't understand what the repro is exactly here. Can you attach the dev certificate so we can look at it?

Can someone explain how to get corefx to trust a self signed cert?

The problem is most likely not a "trust" issue. We have see that installing self-signed certificates into the trust list on Linux doesn't work because the certificate itself is usually missing some X509v3 attributes that Linux (due to OpenSssl) require but Windows doesn't. OpenSsl on Linux requires that self-signed certificates have the "Cert Signing" attribute on them. This is because a self-signed, trusted certificate has to be both a valid CA certificate (requires 'Cert Signing') as well as a "leaf node" certificate.

[sntnupl]

@davidsh please find attached the dev certificate file with which the issue happens.
dev-cert.zip

It would also be very helpful if you could point to any article where the proper step for generating self-signed certificates are provided, that would make linux/openssl happy.

[meron1122]

I have this issue too. But for me is issue:

System.IO.IOException: The decryption operation failed, see inner exception. ---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL. ---> Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.Decrypt(SafeSslHandle context, Byte[] outBuffer, Int32 offset, Int32 count, SslErrorCode& errorCode)
   at System.Net.Security.SslStreamPal.EncryptDecryptHelper(SafeDeleteContext securityContext, ReadOnlyMemory`1 input, Int32 offset, Int32 size, Boolean encrypt, Byte[]& output, Int32& resultSize)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Adapter.Internal.AdaptedPipeline.ReadInputAsync(Stream stream)
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask`1& awaitable)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

[davidsh]

It would also be very helpful if you could point to any article where the proper step for generating self-signed certificates are provided, that would make linux/openssl happy.

So, I've researched this and determined that on Linux, self-signed certificates will not work unless you have "Certificate Signing" as an additional attribute on the KeyUsage field.

This limitation is not a .NET Core thing. It is an OpenSsl change (between 1.0.2e and 1.0.2g) where self-signed certificates need to have this additional attribute set on them. Even when you install the certificate (public key portion) using 'update-ca-certificates' on Linux, the certificate will still not be "trusted" if the "Certificate Signing" bit is missing.

There is an open issue still on the OpenSsl repo about this problem: https://github.com/openssl/openssl/issues/1418

This is a client-side problem. For example, I created an ASP.NET Core website using kestrel and hosted it on my Windows box. I first used the default "ASP.NET Core HTTPS development certificate" for the website. I then installed the certificate (public key portion) into the trusted certificates list on my Linux machine using 'update-ca-certificates'. When I used 'curl' to send a request to the website, I got an error.

When I changed the website to use a self-signed certificate that has the added "Certificate Signing" attribute, then 'curl' worked from the Linux machine.

I got the same results using HttpClient from the Linux machine. I got errors about the certificate when I used the default ASP.NET Core development certificate. But when I updated the certificate to include "Certificate Signing" then HttpClient worked again.

As a result of this investigation, we need to fix the 'dotnet dev-certs' tool so that it will generate a self-signed certificate that can be used on both Windows and Linux properly. @Tratcher where is the source code to this tool? Perhaps I can submit a PR to fix that.

Instead of using the default ASP.NET development certificate, you can generate your own self-signed certificate using Windows PowerShell or Linux OpenSsl tools. Just make sure to specify the 'Certificate Signing' attribute in addition to other things.

For example, this is the command I used to generate a self-signed certificate on Windows.

New-SelfSignedCertificate -DnsName "localhost" -KeyUsage CertSign, DigitalSignature, KeyEncipherment -FriendlyName "Better ASP.NET Core HTTPS development certificate" -CertStoreLocation "Cert:\CurrentUser\My"

I then exported the public key portion and installed that on my Linux machine and ran 'update-ca-certificates' to put it into the trusted list.

[davidsh]

cc: @wfurt

[wfurt]

As @davidsh mentioned, OpenSSL is more picky about correct flags. If curl does not work without -k (insecure) option, .Net will not work either. I look at one @sntnupl and it does not seems to have what is needed.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            15:01:f7:60:0f:09:82:85:5a:e6:ab:1e:ba:1e:a4:16:5b:1d:9e:06
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=New York, L=Rochester, O=localhost, OU=Development, CN=localhost
        Validity
            Not Before: Sep 20 01:17:47 2019 GMT
            Not After : Sep 19 01:17:47 2020 GMT
        Subject: C=US, ST=New York, L=Rochester, O=localhost, OU=Development, CN=localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d0:26:ea:7d:0c:a2:67:81:d1:4b:41:2c:2f:ac:
                    e3:f9:81:77:96:4c:df:dc:df:20:39:d5:e6:1c:4a:
                    b3:34:60:a3:77:b2:e2:c0:3c:a6:25:59:0b:90:6f:
                    6b:9d:50:8c:d1:b3:45:95:e0:f6:30:b6:d3:5e:4c:
                    f2:a6:8c:a6:1c:25:f8:42:b2:e7:cb:88:dc:95:6f:
                    f2:37:5c:dc:40:62:cb:b1:57:16:1b:0b:03:63:90:
                    e4:3f:13:74:1c:e7:ff:97:e9:67:57:f5:7f:e0:ac:
                    84:1d:23:5d:03:c9:ea:04:4f:60:75:8a:77:8f:b0:
                    94:b4:27:4e:08:c5:6d:f3:1f:e5:02:a4:3b:74:a2:
                    6e:7d:b2:a3:e8:96:19:a3:94:2e:49:f6:a1:e0:b1:
                    46:ce:9e:c9:21:b8:ea:28:4a:25:ad:3d:6c:c0:1b:
                    88:12:5a:f3:1d:d5:fc:6b:e7:b5:f6:ef:66:b6:01:
                    fa:ef:71:b0:ed:d9:71:65:98:22:64:09:4e:e8:c4:
                    ba:73:c0:bb:31:24:57:6c:69:cc:18:a4:c1:8e:eb:
                    70:e3:5f:a5:2a:8f:31:df:4b:a4:2f:d4:a2:26:fc:
                    77:9c:c0:01:f1:7a:55:29:70:c1:4e:76:5c:f5:f0:
                    6c:cb:5d:14:b7:c0:07:08:1a:89:cd:91:ba:67:96:
                    bb:0b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:localhost, DNS:127.0.0.1
    Signature Algorithm: sha256WithRSAEncryption

There is longer discussion about this here: https://github.com/dotnet/corefx/issues/37516
I check certificate generated by asp.net on my system and I get:

        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication
            X509v3 Subject Alternative Name: critical
                DNS:localhost
            1.3.6.1.4.1.311.84.1.1:

my suggestions would be to clear old certificate, create new set, verify v3 extensions and verify that curl can connect without -k.

[davidsh]

my suggestions would be to clear old certificate, create new set, verify v3 extensions and verify that curl can connect without -k.

As I discussed above, the current 'dotnet dev-certs' tool is generating a self-signed certificate that works only on Windows but can't work on Linux due to OpenSsl requirements that self-signed certificates have the additional 'Certificate Signing' bit set on them. So, "clear old certificate, create new set" will not work at all unless that new set contains the additional 'Certificate Signing' bit.

And as you pointed out, the current ASP.NET 'dotnet dev-certs' tool is not creating a certificate that can make Linux clients happy.

    X509v3 Key Usage: critical
        Digital Signature, Key Encipherment

That Key Usage attribute is missing the 'Certificate Signing' bit.

So, the best fix for everyone is to 1) understand the problems that OpenSsl on Linux has regarding self-signed certificates, and 2) change our 'dotnet dev-certs' tool so that the automatically generated ASP.NET Core development certificate has the proper attributes so that it works on both Windows and Linux. That will reduce pain for developers.

cc: @bartonjs

[Tratcher]

The source is here:
https://github.com/aspnet/AspNetCore/blob/8e05fb0cf450afdf629b275435c69c836ae8df68/src/Shared/CertificateGeneration/CertificateManager.cs
https://github.com/aspnet/AspNetCore/blob/8e05fb0cf450afdf629b275435c69c836ae8df68/src/Tools/dotnet-dev-certs/src/Program.cs

@blowdart https://github.com/aspnet/AspNetCore/issues/7246#issuecomment-537752663

[blowdart]

If this were to change it should only be on Linux. Creating a trusted cert which can issue other certs goes beyond self signed, it's creating a root CA and that is dangerous. I'd argue OpenSSL isn't being correct here, but it's easier for us to cope with their bad security decision than it is to get it rolled back, but if we do so we don't want to expose Windows users to additional risks just because of OpenSSL.

As for Linux to Windows communication, the dev certificate is limited to localhost anyway, so it should be rejected for cross machine communication, unless you're disabling subject validation which is not something we want to encourage at all, so I view that as out of scope and where folks need to start considering their own development certificate infrastructure and creation.

[davidsh]

If this were to change it should only be on Linux. Creating a trusted cert which can issue other certs goes beyond self signed, it's creating a root CA and that is dangerous.

What's more interesting is that the certificate itself has a Basic Constraint that says it isn't a CA but an end-entity certificate.

It's just that OpenSsl has some logic that says any self-signed certificate can't be "trusted" unless it has the "Certificate Signing" attribute in the KeyUsages field.

I'm going to explore other possibilities for creating the self-signed certificate to see what works on both Windows and Linux.

Perhaps if the Basic Constraints were changed from
"Subject Type=End Entity, Path Length Constraint=None"

to
" Subject Type=End Entity, Path Length Constraint=0"

then it would further reduce risk if someone tried to use this development certificate to issue further certificates from it.

[sntnupl]

@davidsh thanks for your reply.
Following your advice, I created crt file with KeyUsage field having Certificate Signing" attribute.

My development platform is linux, so I used openssl to do the same.
Here's what the config file looked like:

[req]
default_bits       = 2048
default_keyfile    = localhost.key
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_ca

[req_distinguished_name]
countryName                 = Country Name (2 letter code)
countryName_default         = US
stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = New York
localityName                = Locality Name (eg, city)
localityName_default        = Rochester
organizationName            = Organization Name (eg, company)
organizationName_default    = localhost
organizationalUnitName      = organizationalunit
organizationalUnitName_default = Development
commonName                  = Common Name (e.g. server FQDN or YOUR name)
commonName_default          = localhost
commonName_max              = 64

[req_ext]
subjectAltName = @alt_names

[v3_ca]
basicConstraints = CA:FALSE
subjectAltName = @alt_names
keyUsage       = keyCertSign, nonRepudiation, digitalSignature, keyEncipherment

[alt_names]
DNS.1   = localhost
DNS.2   = 127.0.0.1

I transferred the crt file to windows machine, and checked the KeyUsage field, and Certificate Sigining was indeed added.

However, the issue continues to happen. Same exception as I mentioned in my preview comments.
I used curl with --insecure field, but it didn't help.

Can there be any other attribute missing?
Cert file attached.
dev-cert-updated.zip

[davidsh]

@davidsh thanks for your reply.
Following your advice, I created crt file with KeyUsage field having Certificate Signing" attribute.

How did you generate the self-signed certificate exactly? What was the openssl command you used? What version of openssl do you have?

Is the self-signed certificate (*.PFX file with private key) installed on the Linux machine? How/where did you install that? Is the Linux machine the server machine?

Is the self-signed certificate public key portion (*.CRT file) installed in the machine trusted CA store with commands similar to?

sudo cp ~/dev-cert.crt /usr/local/share/ca-certificates/dev-cert.crt
sudo update-ca-certificates

[sntnupl]

How did you generate the self-signed certificate exactly? What was the openssl command you used? What version of openssl do you have?

I used the following commands (localhost.conf commented above)

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout localhost.key -out localhost.crt -config localhost.conf -passin pass:crypticpassword

sudo openssl pkcs12 -export -out localhost.pfx -inkey localhost.key -in localhost.crt


Delete previously installed certificate named "localhost"
certutil -D -d sql:${HOME}/.pki/nssdb -n "localhost"

Install new certificate
certutil -D -d sql:${HOME}/.pki/nssdb -n "localhost"

I am using OpenSSL version 1.1.1

Is the self-signed certificate (*.PFX file with private key) installed on the Linux machine? How/where did you install that?

Yes, using the commands above.

Is the Linux machine the server machine?

Yes.

Is the self-signed certificate public key portion (*.CRT file) installed in the machine trusted CA store

Not installed but I call curl like this:

curl -X GET --insecure --cacert ~/.aspnet/localhost.crt https://localhost:44320/api/Account/User   -H 'Accept: application/json'   -H 'Authorization: Bearer <Access_Token>'

[davidsh]

https://localhost:44320

Is this an ASP.NET Core server? How is the ASP.NET Core Server finding the self-signed certificate (w/ private key)?

I've never tried your particular set of instructions, i.e. using 'certutil' on the Linux machine.

What happens if a Windows machine tries to access this Linux http://localhost:44320 server? And assume that you have installed the self-signed certificate (public key portion) into the Windows trusted root store. Will that work without any browser errors?

[sntnupl]

Yes, it is a ASP.NET Core server.

public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder => {
                    webBuilder.UseKestrel(options => {
                        var certificate = new X509Certificate2("localhost.pfx", "crypticpassword");
                        options.Listen(IPAddress.Loopback, 44320, listenOptions => {
                            listenOptions.UseHttps(certificate);
                        });

                        options.AddServerHeader = false;
                    });
                    webBuilder.UseStartup<Startup>();
                });

Above code shows how the server finds the certificate.
I did not try connecting a windows client to this server. But I had, a few weeks back, follow similar steps to generate a self signed cert in windows (using this Microsoft blog/tutorial), and same code worked fine windows client w/ windows server.

Also, as I mentioned above,
curl -X GET --insecure https://localhost:44320/.well-known/openid-configuration works fine.

I call another api endpoint to get access token, and then, as I make an api call to access a protected end point,
ASP.NET Core JWT middleware internally tries to fetch config from OAuth endpoint, by making HTTPS call to: https://localhost:44320/.well-known/openid-configuration. The issue happens here. Only in Linux.

[wfurt]

Using curl --insecure is invalid test. You need to get to point where trust is esteablished and curl just works. Without it, you will keep getting validation errors.

[davidsh]

So, I did a test where I created a sef-signed certificate on Windows using PowerShell. This self-signed certificate has the 'Certificate Signing' attribute on it.

PS C:\Users\davidsh> New-SelfSignedCertificate -DnsName "localhost" -TextExtension @("2.5.29.37={text}1.3.6.1.5.5.7.3.1","2.5.29.19= {critical}{text}false") -KeyUsage CertSign, DigitalSignature, KeyEncipherment -FriendlyName "V2 ASP.NET Core HTTPS development certificate" -CertStoreLocation "Cert:\CurrentUser\My"

I then ran an ASP.NET Core web server on Windows using this self-signed certificate.

I was able to access it from a Linux WSL window using 'curl' and making sure the self-signed certificate was trusted (via passing the .crt via a parameter to curl).

davidsh@davidsh2:/mnt/d/dotnet/web1$ curl --cacert ./localhost-v2.crt --verbose https://localhost:5001/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: ./localhost-v2.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost
*  start date: Oct 11 17:27:55 2019 GMT
*  expire date: Oct 11 17:47:55 2020 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=localhost
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffbe609b40)
> GET / HTTP/2
> Host: localhost:5001
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< date: Fri, 11 Oct 2019 17:44:35 GMT
< server: Kestrel
<
* Connection #0 to host localhost left intact
Hello World!davidsh@davidsh2:/mnt/d/dotnet/web1$

I will try your openssl instructions to generate the self-signed certificate on Linux itself. And then try to run the Linux server and use curl against it.

But our respective repros are not actually the same because you have a Linux solution where you have the ASP.NET server do an outbound request back to itself over TLS. Can you post a simple complete repro for that?

We have too many variables at work here and we need to get a standardized repro.

[davidsh]

I am using OpenSSL version 1.1.1

Also, what distro are you using? Can you show the 'dotnet --info' output?

[davidsh]

I did another test using only Linux machines (actually 2 separate Linux distros using WSL on Windows).

Server:

Runtime Environment:
 OS Name:     debian
 OS Version:  9
 OS Platform: Linux
 RID:         debian.9-x64
OpenSSL 1.1.0j  20 Nov 2018

Then I created a new ASP.NET Core application:

dotnet new web --name web1

I generated the public/private key using your 'localhost.conf' and 'openssl' instructions above.

I set the following environment variables so that ASP.NET Core would use my *.pfx file I generated:

export ASPNETCORE_Kestrel__Certificates__Default__Password=password
export ASPNETCORE_Kestrel__Certificates__Default__Path=localhost.pfx

Then I started the server with "dotnet run"

My client machine is a second WSL machine:

Hello World!davidsh@davidsh2:/mnt/d/dotnet/web1$ more /etc/os*
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"

OpenSSL 1.1.0g  2 Nov 2017

Then I used 'curl' and was able to do an HTTPS request to my first Linux machine. The '/mnt/d/localhost.crt' is the public key of the self-signed certificate that I made on the first Linux WSL machine.

davidsh@davidsh2:/mnt/d/dotnet/web1$ curl --cacert /mnt/d/localhost.crt https://localhost:5001/
Hello World!davidsh@davidsh2:/mnt/d/dotnet/web1$    

Update

I then did a modification to the ASP.NET Core server I created above on the first machine. I added another mapped endpoint. This new endpoint will do an outbound request using HttpClient back to itself.

        endpoints.MapGet("/outbound", async context =>
        {
            using var client = new HttpClient();
            string result = await client.GetStringAsync("https://localhost:5001/");
            await context.Response.WriteAsync($"Outbound result: {result}\n");
        });

I also made sure to install the self-signed certificate (public key) portion into the machine's trusted store on this first Linux WSL machine.

davidsh@davidsh2:~/dotnet/web1$ sudo cp localhost.crt /usr/local/share/ca-certificates
davidsh@davidsh2:~/dotnet/web1$ ls /usr/local/share/ca-certificates
localhost.crt
davidsh@davidsh2:~/dotnet/web1$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

Then I repeated the experiment using the second Linux WSL machine and using curl.

davidsh@davidsh2:/mnt/d/dotnet/web1$ curl --cacert /mnt/d/localhost.crt https://localhost:5001/outbound
Outbound result: Hello World!

Finally, to simply things on the second Linux WSL machine, I then installed the self-signed certificate (public key) portion into its trusted store (similar to what I did on the first Linux WSL machine). Now, I can use a simpler 'curl' command since the certificate is now trusted on this machine:

davidsh@davidsh2:/mnt/d/dotnet/web1$ sudo cp /mnt/d/localhost.crt /usr/local/share/ca-certificates
davidsh@davidsh2:/mnt/d/dotnet/web1$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
davidsh@davidsh2:/mnt/d/dotnet/web1$ curl https://localhost:5001/outbound
Outbound result: Hello World!
davidsh@davidsh2:/mnt/d/dotnet/web1$                                                                                                                                                    

So, this repro is similar to yours in that the Linux ASP.NET Core server does a TLS request back to itself.

Try reproducing my results using your own Linux machines. It should work.

[sntnupl]

@davidsh Many thanks for your detailed explanation!
I am pleased to inform that after installing the self-signed certificate (public key) portion into the machine's trusted store, I am able to make my code work 😀

[meron1122]

It's resolved?

[davidsh]

It's resolved?

No. The automatic dev certificate that the 'dotnet dev-certs' tool generates still doesn't work on Linux since it is missing the 'Certificate Signing' attribute. A PR is still required to fix that (I have plans to put up a PR soon).

However, the manual instructions in this discussion thread above show steps to create a self-signed certificate that does work on Linux.

[Lord-Amateur]

I managed to make a self-signed certificate using the method above, and set the environment variable to point to the certificate. But when I run the application, the old default certificate is used instead of the new one by the web-application. Can you show me the changes to be made in the code of the application so that it uses the newly made self-signed certificate is used instead of the old one?(I am asking about the method to point out to krestrel the new self-signed certificate)

[davidsh]

and set the environment variable to point to the certificate. But when I run the application, the old default certificate is used instead of the new one by the web-application.

Not sure why the environment variables aren't working. That is an option that Kestrel usually looks at to specify the PFX file for the server (instead of using the default ASP.NET "dev" certificate). Can you show the exact environment variables you're using?

I've used something like this in my code to use a specific PFX file for the Kestrel server:

```c#
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseKestrel(options =>
                        {
                            options.Listen(IPAddress.Loopback, 5000);
                            options.Listen(IPAddress.Loopback, 5001, listenOptions =>
                            {
                                listenOptions.UseHttps("certificate.pfx", "password");
                            });
                        });
                webBuilder.UseStartup<Startup>();
            });
}

```

[Lord-Amateur]

It appears that the program is unable to find the pfx file even if I specified it in the envrironment variables. The error looks like
Using launch settings from '/home/lord-amateur/RazorPagesMovie/Properties/launchSettings.json' [Profile 'RazorPagesMovie']... Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:2006D080:BIO routines:BIO_new_file:no such file at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle) at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password) at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.UseHttps(ListenOptions listenOptions, String fileName, String password) at RazorPagesMovie.Program.<>c.<CreateHostBuilder>b__1_2(ListenOptions listenOptions) in /home/lord-amateur/RazorPagesMovie/Program.cs:line 29

log.txt
(the whole error log is attached)

I used the same Export commands to add to the environment variables.
checking the environment variables:
printenv | grep Kestrel
gives:
ASPNETCORE_Kestrel__Certificates__Default__Password=mypassword
ASPNETCORE_Kestrel__Certificates__Default__Path=localhost.pfx
The localhost.pfx file is located at my home folder

[beefo]

@Lord-Amateur I had a similar error due the file permissions.

[davidsh]

ASPNETCORE_Kestrel__Certificates__Default__Path=localhost.pfx
The localhost.pfx file is located at my home folder

What is your 'home' folder? And how does it relate to where the project files are?

Since you aren't specifying an absolute path for the PFX file, it is assumed to be in the same folder as the binaries for the application. So, you should make sure you copy it there during the build process. Or perhaps use an absolute path for the PFX file when you set the environment variable.

[Lord-Amateur]

I changed the permissions of the file localhost.pfx and also relocated it to the project folder and then curl request test worked. Thanks for the advice.

[DamianEdwards]

Just FYI the underlying OpenSSL issue has been resolved and will ship in 1.1.1h (stable) and 3.0 later this year.

[micka190]

Just FYI the underlying OpenSSL issue has been resolved and will ship in 1.1.1h (stable) and 3.0 later this year.

Should updating to OpenSSL 1.1.1h automatically fix this issue, or will we need to wait for some underlying change in the .NET Core framework/tooling before it's ready?

Currently running the OpenSSL 1.1.1h build (as the linked issue mentions that the fix is already included in it), but I'm still having issues with the SSL connection.

[bartonjs]

Should updating to OpenSSL 1.1.1h automatically fix this issue

As far as I'm aware.

Currently running the OpenSSL 1.1.1h build

If you've built it locally (since it hasn't released yet) double check that your version is actually getting loaded. Local builds tend to end up in /usr/local/lib/ which has (IIRC) lower binding than the system library path. So you'd need to use LD_PRELOAD to pre-load both libssl.so.11 and libcrypto.so.11 from your OpenSSL 1.1.1h build (or install the libraries to the normal system path).

[micka190]

No dice, sadly.

I've installed the OpenSSL 1.1.1-stable version from the OpenSSL GitHub repo, following their instructions:

# From the repo's root directory.
./config
make
make test
sudo make install

Running the /usr/bin/openssl version -a command yields the following output:

OpenSSL 1.1.1f  31 Mar 2020 (Library: OpenSSL 1.1.1h-dev  xx XXX xxxx)
built on: Fri Aug 28 14:27:37 2020 UTC
platform: linux-x86_64
options:  bn(64,64) rc4(8x,int) des(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG
OPENSSLDIR: "/usr/local/ssl"
ENGINESDIR: "/usr/local/ssl/lib/engines-1.1"
Seeding source: os-specific

Which indicates that the file is still the old one.

Running the /usr/local/bin/openssl version -a command yields the following output:

OpenSSL 1.1.1h-dev  xx XXX xxxx
built on: Fri Aug 28 14:27:37 2020 UTC
platform: linux-x86_64
options:  bn(64,64) rc4(8x,int) des(int) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG
OPENSSLDIR: "/usr/local/ssl"
ENGINESDIR: "/usr/local/ssl/lib/engines-1.1"
Seeding source: os-specific

So that's where it got installed.

I move the current /usr/bin/openssl file to my home to keep it as a backup, then create a symlink to the newly created /usr/local/bin/openssl file to replace it. Now, running openssl version -a results in the 1.1.1h output.

I reboot, just in case.

If I try Identity Server 4's first quickstart example, it fails to establish an SSL connection (but it works fine on Windows).

I set LD_PRELOAD to /usr/local/lib/libcrypto.so.1.1:/usr/local/lib/libssl.so.1.1, and reboot (just in case).

I still get a failure to establish an SSL connection.

At this point I'm thinking of just trying to see if I can't get a Unix config and a Windows config and just use a hard-coded certificate with a script to generate them on Unix machines as part of our dev build process.

I don't know. Maybe I'm missing something here? Some people mentioned running sudo ldconfig -v, but that doesn't seem to link libraries that are in my /usr/local/ directory.

[wfurt]

can you check /proc/<PID>/maps just to be sure? The .Net really does not care about the openssl executable. You can leave it as is. I would set LD_LIBRARY_PATH=/usr/local/lib to avoid interactions between preload and dlopen.

Also the loader would honor CLR_OPENSSL_VERSION_OVERRIDE environment. To be sure you can readme the libraries (or create symlink to something like libcrypto.so.1.1.my and do CLR_OPENSSL_VERSION_OVERRIDE=1.1.my

[micka190]

can you check /proc/<PID>/maps just to be sure?

I'm guessing you mean the dotnet results from ps -A? If I look in them using less, I can see that they seem to interact with /usr/local/lib/libcrypto.so.1.1 and /usr/local/lib/libssl.so.1.1, which should be the 1.1.1h install.

LD_LIBRARY_PATH=/usr/local/lib

I've tried exporting this environment variable (both with and without LD_PRELOAD being exported), and it doesn't seem to change anything. Same thing with creating a symlink to libcrypto.so.11.my and setting CLR_OPENSSL_VERSION_OVERRIDE=1.1.my.

[wfurt]

The question is if you see any other copy of libel or libcrypto. If not, than the .NET is running against the version you built.
If you build is right that would suggest that the 1.1.1h did not resolve your issue @micka190

[micka190]

Yeah, using grep to find all instances of libcrypto and libssl just gives me a list of calls to /usr/local/lib/libcrypto.so.1.1 and /usr/local/lib/libssl.so.1.1, which are the 1.1.1h ones.

Looks like I may have an issue unrelated to OpenSSL 1.1.1h... :disappointed:

[wfurt]

What is your issue @micka190? Can you post certificate and repro here (or create separate repo)
Do you have control over the cert creation? There is trivial workaround AFAIK by setting proper flags.

Also note the : OPENSSLDIR: "/usr/local/ssl"
Your version will probably look for trusted CAs in different directory and update-ca will not work if you want to add trust for your self-signet cert.

[micka190]

@wfurt, Essentially: we're trying to get a Blazor app going, but we need authentication for our API. So we're going through the Identity Server 4 documentation to get us started. Or at least we're trying to. We can't get past the first sample, because of issues with SSL. Running it on Windows works fine, but most of us are using Ubuntu 20.04 as our daily driver (the actual distro, not WSL).

In terms of code, that sample link has it all. We gave up and downloaded their sample code from their GitHub repo to make sure we weren't missing something. Works great on Windows, SSL connection issues on Ubuntu machines...

All we get is "The SSL connection could not be established, see inner exception.." But we're unable to view the inner exception. My understanding is that IS4 catches it in some deeply nested internal method and just adds the message to whatever throws this exception.

Since it only occurs on Ubuntu, and not on Windows, I assumed that it was probably related to this issue, since it's occurring in development, when using dotnet dev-certs https's certificates (which from my understanding are self-signed).

[wfurt]

And this you make the cert trusted? Does it work when you run curl -v https://XXX/
Also you can always write simple console app with HttpClient. If you use 5.0 for that, it should have somewhat better error reporting on handshake failures.

Can you also check the certificate has the KeyUssage mentioned above?
You can always use your own certificate instead of the default.

[micka190]

I stepped out of the office for today. I did try it with curl, and I was getting security errors related to SSL (don't remember the exact wording), but I seem to recall that it was about self-signed certificates. I'll verify the exact output on Monday. I'll also take a look at using the 5.0 preview SDK with HttpClient to see if I can't narrow down the cause a bit more and check the KeyUsage.

You can always use your own certificate instead of the default.

I did try that using some of the above examples, but I think I got some stuff mixed up in all the examples because even specifically providing a self-generated certificate (be it through ASP .NET Core environment variables, appsettings.json configs, or through the actual code) was resulting in the same SSL error as I'm getting now (which kind of made me think it had to do with the OpenSSL version initially).

[micka190]

@wfurt Alright, so I'm back in my office.

Running curl -v https://localhost:5001 returns this:

*   Trying 127.0.0.1:5001...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Which is probably related to the OPENSSLDIR: "/usr/local/ssl" potential issue you pointed out earlier. I'll see if I can get the dotnet dev-certs https certificate inside that directory.