Microk8s: Certificates not trusted on macOS Catalina

[VonRehberg]

With latest macOS new requirements for trusted certificates got introduced (see https://support.apple.com/en-us/HT210176 )

One requirement is, that certificates should not be valid for more than 825 days, which collides with the -days 10000 option that is currently in place.

Are there plans to adopt to these changes?
Is there any possibility for a quickfix? Replacing all certs after a restart seems cumbersome.

[balchua]

@VonRehberg thank you for reporting this. Unfortunately as you can see it is written as part of the command when generating the certs.

https://github.com/ubuntu/microk8s/blob/69ea17d4c7a135573d24cfb320edff394b6b0f6d/microk8s-resources/actions/common/utils.sh#L327-L329 and so on.

@ktsakalozos is it possible to add a non-standard attribute to the csr.conf.template like cert_days, we can grep the value?
Why i mention to add it to the csr.conf.template so that the api service kicker will pick up the changes. I just dont know if openssl will complain about unknown properties.

[VonRehberg]

Is there a specific reason for not choosing a more reasonable period like 2 years?

[ktsakalozos]

@VonRehberg how does this issue manifests itself? What do you see failing? I am asking just for a way to reproduce the issue so we are all on the same page.

The reason we did not set on a more reasonable time period is that
a) we were not aware of the 825 day constraint and
b) we did not have in our roadmap to work on refreshing the certificates.

I believe we should comply with the 825 day requirement so we provide a good out of the box user experience. As a follow up we should work on assisting users in all aspects of certificate management eg issue, cycle, update, integrate with certificate management systems etc. @tvansteenburgh this would a good roadmap item.

@VonRehberg would you be interesting in updating the certificate creation in [1] to address you needs and provide some feedback if this fixes the problem? In [2] you can read how to build MicroK8s.

Thank you

[1] https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/common/utils.sh
[2] https://github.com/ubuntu/microk8s/blob/master/docs/build.md

[VonRehberg]

As you can see, the certificate is blocked on OS level and I can't work around it using Chrome or Safari. I need to manually add the public key to the keychain and adjust the trust level.

I will check out the build asap and keep you posted!

[VonRehberg]

Sorry, I can't get the build running in a reasonable time to verify on my machine. :-(

[VonRehberg]

I was able to setup a local node server with https. Using just the information from the csr.conf file was not fixing the issue, as the default hash was ignored. I had to explicitly instruct openssl to use sha256 by adding -sha256 option when signing the CSR.
With the extendedKeyUsage from csr.conf, setting the validity to 800 and sha256 I was able to connect to my local server without modifying the keychain.

[joedborg]

I think the real issue here is that the cert doesn't have a valid expiry date.

[joedborg]

The PR referenced above fixes the "ERR_CERT_REVOKED" error, but there still seems to be something else either wrong or not possible with Chrome.

[VonRehberg]

Isn't that the default for self-signed certs? At least now you're able to instruct chrome to connect anyway!?

[joedborg]

You certainly used to be able to; but (unless there's more issues with the cert than just self signed), I don't see the option.

This does allow you in Firefox, however, but that seems to use its own cert manager and not the MacOS Keychain.

[VonRehberg]

Can you try signing the cert with option "-sha256"?
For me this fixed the issue locally, as otherwise sha1 will be used, which is also blocked by mentioned update.

[joedborg]

Now making a build with sha256 (and 825 days). I'm guessing you only specified that on the openssl req calls?

I.e. https://github.com/ubuntu/microk8s/pull/970/commits/39a333e9a27d1e5a529ef4b94f4b45114e71bab2

@VonRehberg ^

[VonRehberg]

No, it should be specified in the signing process ("openssl x509 -req"). Although the certificate signing request contains the wish to be signed using sha256, the signing CA decides about the options. Seems like openssl on macOS still uses sha1 although it's no longer trusted.

Sorry for the late reply!

[joedborg]

No problem! I think Chrome is now always blocking self-signed certificates. It can be turned off with chrome://flags/#allow-insecure-localhost (change to Enable). We'll look at using letsencrypt in the future.

[ktsakalozos]

@VonRehberg @joedborg the related PR was merged. A build should be available on the edge channel within the day (sudo snap install microk8s --classic --channel=edge). Should we close this issue? Thank you both for your work on this.

[VonRehberg]

For me the dashboard is working now as expected. There's a certificate "warning" but I can proceed to the dashboard.

[mianos]

Installing: snap install microk8s --classic --channel=edge
I still get the following, very invalid, certificate:
5C 7D 9E 3E 9C B1 9C 92 33 2F 0E 38 1F 7C A8 EA
That seems to have been generated during the install: (Thursday, 11 June 2020 at 11:30:43 am Australian Eastern Standard Time)
Brand new install of Ubuntu and then the non edge version which I removed, then the edge version and I still get a certificate that neither Chrome or Safari will accept.

[ktsakalozos]

@mianos the certificate used by MicroK8s is self signed. Is this why the certificate is blocked?

[mianos]

@mianos the certificate used by MicroK8s is self signed. Is this why the certificate is blocked?

Yes, it is self signed but I think it also does not have enough details filled in to be accepted as valid by chrome or safari. I will fix the certificate issue myself by setting up a local CA, creating a cert and loading it into the client. I am mainly commenting here as I just installed the 'edge' snap and it is no way 'fixed' as far as I can see.

[numman-ali]

Chrome Quick-Fix:
Go to chrome://flags/#allow-insecure-localhost
And enable "Allow invalid certificates for resources loaded from localhost."

[jamespegg]

Are there any other workarounds or updates for this issue by any chance? I'm just trying microk8s but this bug is basically making that impossible right now.

At the very least there should be a warning the the micro8ks website that macOS 10.15 and above is not supported...