Runtime: SslStream only sends chain certs if they're in the Windows Cert Store

[cocowalla]

I'm using the RabbitMQ C# Client, which under the hood uses SslStream. I'm having an issue where clients are unable to authenticate using x509 certificates if intermediate certificates are involved - such a chain looks like:

Root CA -> Issuing CA -> Issued Client Certificate

Using Wireshark I can see that when authenticating as a client, SslStream is sending only the leaf certificate, which is causing a certificate handshake error. However, if the Root CA and Issuing CA are added to the Windows Certificate Store as trusted roots, then SslStream sends all 3 certificates, and RabbitMQ is happy.

The certificate I'm using as the client cert is a PKCS#12 file that contains the whole chain (as X509Certificate2). So, the question is if there is any way to force SslStream to send the whole chain when authenticating as a client, even if the chain certs are not in the Windows Certificate Store?

[davidsh]

cc: @bartonjs

[Clockwork-Muse]

...except you're using the file, not a cert from the store, as the client id, right? Very probably it's not loading the entire chain from the file (it only actually loads one), so the leaf has a reference value to the issuer cert (thumbprint? not sure), but no idea what the actual cert is.
If I remember right, you have to load the entire set of certs into a runtime store, then grab the leaf certificate from that. The reason it works if it's in the windows cert store is that it does do lookup there by default.

[cocowalla]

@Clockwork-Muse I'm currently selecting the certificate from an X509CertificateCollection that contains the full chain.

I figured the solution might involve creating some kind of temporary X509Store, but if I create a unique one for my app and select a certificate from it, it's still the case that only the leaf is sent - it only seems to send the whole chain if the others certs are in the My store (strangely, not the Machine store).

If you have any info on how to create a temporary store only visible to the app and have SslStream send the whole chain, some guidance would be much appreciated.

[cocowalla]

I've checked from Linux, and observed the same behavior - the chain of intermediate certs is sent if I use the My certificate store, but _not_ if I create a unique one with new X509Store("whatever").

I guess this is happening in SecureChannel, but that is a bit of a whale, and I'm struggling to find _where_ in the code this actually takes please?

[cocowalla]

I've managed to get it working as expected on Linux, by using the SSL_CERT_FILE environment variable:

SSL_CERT_FILE=/opt/my-app/etc/ca-bundle.crt ./My.App

When doing this, SslStream is sending the intermediate certificate as well as the leaf, and RabbitMQ is happy.

Is there anything like this for Windows? @bartonjs, I guess if anyone would know it would be you? ;)

[Clockwork-Muse]

@cocowalla - well, _often_, SSL_CERT_FILE/SSL_CERT_DIR are set to a global location, so it ends up being the equivalent of the Windows certificate store (I'm currently cursing Python, because it uses about 3/4 different sets of certs during setup pulled in from different packages). I guess the closest equivalent would be certs that only certain users would have permission to access (which you'd normally want to do for the private key anyways).

I thought I'd discovered a way to load/send the entire chain when dealing with a different issue, but I can't recreate it now, possibly I just imagined it.

What's your usecase here, that you wouldn't be adding the certs to the store anyways?

[cocowalla]

Regarding SSL_CERT_FILE/SSL_CERT_DIR, these certs are only for use by this particular application, so the env var is set just for the application.

My usecase was a (failed!) attempt to keep things simple by using the same approach on both Windows and Linux (keys stored in the filesystem, protected by ACLs). I've decided to give up; I'll use SSL_CERT_FILE on Linux, and the My root and intermediate stores on Windows.

I do however still think that SslStream should have some means of providing chain certs from ephemeral stores or a X509Certificate2Collection.

[Clockwork-Muse]

While I agree that it would be nice if SslStream would send the entire cert (which would probably require the actual cert domain types keeping their reference, or whatever), there's a problem with that: You're sidestepping the normal mechanism for chain resolution on the platforms. If the cert end up in a resource bundle or something, it ends up being more difficult for the end user to configure.

[cocowalla]

I don't really see it as side-stepping anything; more _augmenting_ the existing mechanism. The My store would still be used, but the ephemeral store or X509Certificate2Collection the cert came from would additionally be used to locate chain certs. I imagine most devs would expect this behaviour; if you load a chain from a PKCS12 file, it seems counter-intuitive to only use the leaf cert and completely ignore the chain certs.

For your last point, I'd argue that most users would be comfortable replacing a file, and probably haven't even _heard_ of the Windows Certificate Store ;)

[bartonjs]

I don't really know the SChannel APIs (which provide TLS for us on Windows), but I think that they only take the single certificate, then internally do the chain building for sending the intermediates.

The Linux and macOS TLS APIs are a little more raw, so they might be more amenable to such a feature, but I wouldn't add a new feature that doesn't work on Windows (71% of the usage of .NET Core, as of last summer).

[cocowalla]

@bartonjs ah, that explains why I couldn't find anything in SecureChannel that sends the intermediate certs - I didn't realise it was using an SChannel API that's hard-coded to check against specific stores.

I agree it makes no sense to add this unless it works across all platforms.

[gingters]

I like to bring this issue back to the table.

We have a aspnet core based service that should be deployed onto clients windows computers. They are most likely not publicly available on the internet, and the http API that our service provides should be secured. We have a self-signed root CA and an intermediate CA and this issues a certificate for each installation. This way we can also do client-certificate auth with our service as the client.

We really want to avoid to install our root and intermediate certificates into the trusted root CA store of the clients computers. We deem this a bad security practice and really want to avoid it.

Technically it is sufficient to use certificate pinning to our root CA cert within our dedicated client application. There is no need that any computer should be needed to completely trust our cert.

So there needs to be a way to tell kestrel on windows (which uses SslStream) to not only send the actual server certificate we configured but the complete chain, _without_ installing the chain in the trusted root store of the computer.

This is pretty counter intuitive if you create a .pfx file containing the complete chain (the server cert, the servers private key as well as both the root and intermediate CA certs), tell kestrel to use this file, and then, when its not working, find out that the X509Certificate2 only represents a single cert from the file and not the chain, and there is no other way to tell the system where to find the certificates that it should send along except for exposing the computer of your client to a risk that usually should be prevented at all costs.

So, what is the actual plan to overcome this issue? Is there anything we could do to help get this sorted out?

[davidsh]

So, what is the actual plan to overcome this issue? Is there anything we could do to help get this sorted out?

I agree with your thoughts about the importance of this scenario. It should "just work".

As far as making progress on this issue, we need to understand what the capabilities/API of the underlying TLS stacks (SCHANNEL for Windows, OpenSsl for Linux, MacOS) are. At this point, we aren't sure the platforms support this functionality.

[JesperTreetop]

I have almost exactly the same situation as @gingters . We would like to send the full chain if possible - and would also like to avoid placing root certificates in a user/computer-wide store for a variety of reasons. As mentioned, telling Kestrel to use a .pfx which provides a full chain and then having them conceptually "dropped on the floor" is not a good experience, and recent efforts (#31944) to be able to read formats that are often associated with carrying the full chain highlights this.

(For anyone in the same situation who needs to have clients validate a certificate presented without its chain: if you can prime a list of potential certificates and get a grip on their chains beforehand, that's a good workaround. We can't, because the set of potential certificates is large and volatile. Our workaround right now is for the client to connect, receive the certificate, disconnect, use a secondary server or endpoint to retrieve the full chain and then connect again with the answer in hand. The alternative is to stall synchronously in the certificate validator, which is not only an incredibly bad idea but can also look hostile and malicious to the server, and is liable to be tripped up by defenses or timeouts in Kestrel or Schannel. Asynchronous certificate validation might have fixed this if this step wasn't supposed not to have huge pauses in it in the first place.)

Schannel not supporting this is a reasonable explanation why, but considering alternate stacks can provide a solution it would be good to be able to opt into them. (And yes - I realize this is a big hammer, but it would also be able to pound in many nails.)

[knapsu]

Hi. We are having the exact same problem. Certificate _trust chain_ is broken because only the server leaf certificate is sent without the intermediate CA (and root CA). The PFX file had all the certs bundled. Very unexpected and problematic behavior.

As a workaround I am using HAProxy to terminate the SSL traffic and forward it to unencrypted port.

[wfurt]

the server part will be fixed in 5.0. There is now option to pass CertificateContext with all chain - perhaps loaded from pfx or PEM file. However, on Windows that will add intermediates to the store if needed - there is no other way how to deal with it as the handshake actually does not happen in the same process space. That was recommended by Windows platform developers.

The client part will need some more work and thinking.

It also seems like this morphed from client to server side back in 2019. Since this is probably too late for @cocowalla, I'm thinking about closing this and perhaps moving the server discussion to separate issue if needed - and the 5.0 behavior does not seem sufficient - #35844.

[JesperTreetop]

Good to see some progress and that this issue is given some thought and attention. (Less good that the workaround is all that can happen on Windows, but that's not the fault of the .NET team.)