Standard: The state of X509Certificate management with netstandard, xamarin, win & macOS

[manijak]

First, my apologies if this is the wrong channel to post this. I've been digging and trying to understand for weeks where the issue/limitation is to our problem, figured this would be the most suitable place to ask. We are not sure if this is a limitation with the frameworks, or the platform, or due to our limited knowledge, or a combination of all. But things are very confusing right now, and we just need proper guidance on what direction to take.

EDIT: Would appreciate if @bartonjs or @filipnavara could pitch in 👍

Background 💡

We are creating a small, cross-platform (win & macos), desktop application that will act as a websocket server running only on localhost (https). This application is a bridge between our web applications and the local machine hardware (e.g. printers). In order to provide a secure connection, SslStream requires a X509Certificate2 class with a PrivateKey.

The idea is to let the app itself generate a self-signed certificate with private key and use that to start the secure websocket service. After the certificate has been generated for the first time, we store it in the local certificate store and reuse it when the app restarts.

Project structure & frameworks 📦

This is how our solution is defined. SharedProject is a shared lib referenced by both AppWin & AppMac. The Win & Mac projects are startup-projects for their respective platform.

OurApp.sln
  |---- SharedProject (netstandard2.1)
        Creates & runs the websocket service, generates certificate (win), communicates with local hardware, ect

  |---- AppWin (netcore3.0)
        Startup project for Windows platform. Contains platform-specific code, if any.

  |---- AppMac (Xamarin.Mac)
        Startup project for macOS platform. Contains platform-specific code (generates certificate).

How it runs 🏃

In our SharedProject we have a class taking care of certificate generation and storing. The idea was that both the Win & Mac platforms could use this to manage the certificates. It uses the X509Certificate2 & CertificateRequest provided by the framework. Plus the X509Store to save and retrieve the certificate.

Windows ✔️
On Windows there are no issues. With minimum amount of code we are able to generate a self-signed certificate with private-key, store it, and run the service with it. Also re-using the certificate when the app restarts by retrieving the certificate works very well. The X509Store returns the certificate with the _private key_, and we can easily use it with our SslStream.

macOS ❌
Things are not so clear here... We could not use the above mentioned shared class to manage certificates (that works just fine on Windows):

• First issue was that we got a PlatformNotSupportedException when the application was trying to use the CertificateRequest.
• Second "issue" was that the X509Store did not store or retrieve certificates from the macOS keychain, but instead it uses some local mono certificate-store that the browsers do not see (at least in our tests). And it only stored the certificate (.cer) not the private-key that was included in the X509Certificate2 object. Plus it seemed difficult to "Trust" the stored certificate.

So we created a separate class to take care of certificate management for macOS inside the AppMac (Xamarin.Mac) project. By using BouncyCastle we generated an identical self-signed certificate with private-key and used that to run our secure websocket service:

• When launching our Secure Websocket service on macOS, the certificate used here is automatically added to the local macOS login keychain. For us this is great, since we could not use X509Store to add it there in the first place.
• The only way we could retrieve the certificate from the keychain was to use the SecKeyChain class and query for it. Thankfully the SecCertificate object has a ToX509Certificate2() function. But that still left us without the PrivateKey that is needed by SslStream.
• We also tried using SecKeyChain.FindIdentity to retrieve both, but even tho it is easy to convert SecCertificate to X509Certificate2, there seems to be no way to convert a SecKey to something useful like RSA or RSACryptoServiceProvider or something that can be embedded to the X509Certificate2

What to do? 😭

We are pretty much stuck when it comes to the macOS platform. There seems to be no way for us to retrieve our own X509Certificate2 with the private-key embedded. Not sure if this is because we are using Xamarin.Mac & netstandard2.1, perhaps things are easier on netcore3? But still, Xamarin & mono have been around for a long time, there should be a way for us to do this today. Must be something we are missing here and would appreciate if someone could guide us.

[filipnavara]

The X509Store and related classes on Xamarin/Mono are a mess. What you are seeing is unforunately the expected behavior now. I previously investigated several ways to deal with it:

• Porting CoreFX implementation to Mono (as part of System, System.Core and mscorlib assemblies). While part of that work was completed it was done mostly to fix bugs in X509Certificate[2] classes. There are few fundamental problems with this approach. Firstly, Mono uses BoringSSL/AppleCrypto while CoreFX uses OpenSSL/AppleCrypto. Secondly, Mono supports many platforms that CoreFX currently does not (iOS, for example). We worked to fix any incompatibilities in CoreFX code that would be problematic but eventually the project was dropped. I had a prototype Mono build for macOS that used CoreFX code for all these problematic classes but due to portability issues to other platforms it was not deemed viable approach. It was discussed at length here.
• Using CoreFX implementation on Xamarin/Mono as application assemblies. This approach is actually viable and works for us but we build everything from source and it's not as easy as it could be. If I find some free time I can try to make a minimal sample for this and publish it.
• Use Mono/netcore (.NET 5) instead. These builds use the whole CoreFX class library as-is. Unfortunately it's not production ready at the moment and the integration with Xamarin.Mac is basically non-existent. I do have some prototypes and I hope to have more information to share in about a month but it will take a while to get this production ready.

Overall, if you can confirm that the certificate related code actually work on .NET Core 3 applications on macOS it would be a good start. If that works I may be able to help you with some solution until .NET 5 comes along.

[manijak]

Thank you so much @filipnavara for the detailed explanation. At least it makes me feel more sane 😆. I don't have much experience with Mono/Xamarin in general, but would've thought that there was some decent way to work with the macos keychain & X509 by now. But it is what it is.

I will definitely take some time and try out the same code with .netcore 3, should've done that first 😬. Will report back.

Considering that .NET 5 is at least a few years away we will need to find a workaround or rethink the solution. For the time being only Chrome will allow us to access an unsecured websocket connection from a secure website. Just a matter of time before that changes.

Thanks again, at least we know where we stand. 👍

[someleshik]

Hello. I'm trying to make just the same scenario - macos app with ssl-enabled localhost https server.
Here's what I did find:

1. I'm using Ceen.Httpd as a light-weight http(s) server. It's working well on my scenarios, and it's available in NuGet repository
2. When my app starts, it loads certificate with a private key either from pfx file or from pem files.
   If the source of certificate is a pfx file - everything is just ok. If pem - then there's a problem in custom RSA private key reading/creating, but if your private key is not encrypted, and you know its low-level asn1 structure - you can easily read it. And, in that latter scenario - you can load certificate and assign a private key to it via

private X509Certificate2 ConstructCertificate(string cert, string key)
{
  X509Certificate2 result = new X509Certificate2(GetBytesFromPEM(cert, "CERTIFICATE"));
  RSACryptoServiceProvider prov = DecodeRSAPrivateKey(GetBytesFromPEM(key, "PRIVATE KEY"));
  return result.CopyWithPrivateKey(prov);
}

Where GetBytesFromPEM does Base64-decoding of string inside (BEGIN WHAT)/(END WHAT) and DecodeRSAPrivateKey copies bytes from ASN1 representation of a 2048-bit RSA key
The platform I'm working on is a .NET Core 3.0 and all is ok except one thing - somehow that private key and a certificate are being copied automatically to 'login' keychain, and, when app is being run outside of Visual Studio - it asks access to login keychain. It is the only side-effect I don't know how to hanlde.
I can share my source snippets if you like.

[ConnorL33t]

@filipnavara If you have time - I'd love to see the minimal sample.

[manijak]

@filipnavara Sorry for the late follow-up to this issue, it's been a busy month. But here we go.
I re-created a version of our Xamarin.Mac app as a netcore3 Console app. Here are the results:

• I decided to again try our original shared class to generate & store the certificate (same one we use for Windows platform). Like mentioned before this class uses the CertificateRequest() method to generate a certificate. Happy to say that this worked very well on macOS using netcore3 👍. There was only a minor exception on the first try because we tried to set FriendlyName on the certificate, and that is not supported on Unix. No big deal.

• Using the X509Store to store & retrieve the certificate also worked like a charm! The certificate was fetched with its privateKey without issues. Only thing to note here is that we had to use .My store compared to the Windows .Root store, otherwise we would get Access denied exception.

So it looks like netcore3 plays much better with macOS and the keychain. Now the question is, will it be possible to get this ported over to Xamarin somehow?

[filipnavara]

So it looks like netcore3 plays much better with macOS and the keychain. Now the question is, will it possible to get this ported over to Xamarin somehow?

Maybe. Hopefully I will have better answer by the end of the week. I am going to ask Mono people about the possible options.

[manijak]

I know it's the busy time of the year @filipnavara, but did you get any more info regarding the issue?

[andreybazhukov]

@manijak I have the same problem and I created 2 applications (Xamarin.Mac + .NETCore Console). .NETCore Console (self-contained) app is contained in Xamarin.Mac like Resources.
Xamarin.Mac application launches the second console application (Process.Start()) and executes code with the private key.

[manijak]

@andreybazhukov Interesting solution :) Did you only isolate the certificate related code to the console app or everything except the GUI? I'm just wondering how you managed to retrieve the X509 certificate (with privateKey) from one process and reuse it in the original (xamarin) process? (Our main issue is retrieving the certificate with private key and using it in the secure websocket service).

[borrrden]

Is there somewhere that documents all the differences between the platforms? I'm having a horrid time trying to make something that works between all the platforms I need to support (.NET Core, UWP, Xamarin iOS / Android). .NET Core seems to work pretty well, but Xamarin iOS is basically nonfunctional. I can save a certificate with a private key but the private key just vanishes (looking at the source, it seems this entire functionality is simply commented out). Or maybe some resources on subclassing X509Certificate2 so that the runtime can use it properly and I can manage the keys myself using interop calls, etc. I'm trying to avoid the behavior that the .NET runtimes seem to often defer to (just dump out the private key material to an arbitrary file somewhere) and instead interface with more secure mechanisms where possible (preferably in a way that the key material never even leaves secure enclaves where available).

[lihuiniu]

Cross-platform cryptography in .NET Core and .NET 5 | Microsoft Docs at https://docs.microsoft.com/en-us/dotnet/standard/security/cross-platform-cryptography