Cert-manager: Allow CA issuer secret rotation

[jroper]

Is your feature request related to a problem? Please describe.
The CA issuer will automatically rotate secrets for the secrets it generates, but what about rotating its own secret and certificate? What happens when its certificate expires? Anything trusting the secrets it generates will break. And if you give it a new secret? Well now either you have to instantly renew all certificates and make everything pick up those certificates at the same time. There has to be a better way.

Describe the solution you'd like
Provide the ability to configure multiple secrets (or at least, one secret, and additional certs). The additional certs are configured in k8s secrets as additional certs that a consumer should trust. Consumers can pick up these additional certs, and them to the list of root cas they trust. Now, it is possible to rotate ca certificates, without ever having something not trusting a certificate that should be trusted, with the following procedure:

• Generate the new private key/cert for the ca cert, and create a new secret for it.
• Add the secret to the ca issuers list of additional secrets. The ca should update all secrets it's generated to include this additional CA.
• Wait for long enough for all consumers to pick the additional CA up, so now everything trusts two different root CA certificates.
• Update the CA issuer to use the new secret as its primary secret, and move the old secret to its list of additional secrets.
• Wait for all existing issued certificates to expire/be rotated.
• Remove the old root CA certificate from the list of additional ca certificates in the issuer.

Describe alternatives you've considered
I don't know if there are any alternatives - as discussed above, there's no other way to rotate root certificates without risking an outage.

/kind feature

[munnerz]

Instead of the CA issuer generating its own certificate, you should use the self signed issuer type to automatically handle renewing the CA certificate.

That said, the CA issuer should have a mechanism for rotating all leafs under the root, as this is a generally useful thing.

The steps you've suggested do make sense, however I'm concerned that if cert-manager is too opinionated here then we'll build an API that works for nobody - I'd rather cert-manager provider API primitives that allow for different rotation strategies to be built, and then provide a sane default implementation perhaps like you've described.

Right now, trust distribution isn't handled by cert-manager - in some cases, reading the ca.crt from the generated secret is sufficient, but in many other cases the CA may be distributed through some other manual mechanism to end-user systems. If we rush ahead and make assumptions then I fear we'll build something that could cause more harm than good.

[retest-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to jetstack.
/lifecycle stale

[meyskens]

/remove-lifecycle stale

[meyskens]

/area ca
/priority important-longterm

[stefansedich]

We suffer the same issue, we are using kiam and have setup an issuer with a self signed CA cert, but if the CA cert renews, the old server and agent certs are no longer valid, often resulting in an outage forcing us to remove the agent and server cert secrets to let it generate new ones.

Would be greate if there could be a dependency chain or something where we could link the agent/server certs to the CA cert and have them renew when the CA cert does.

@munnerz is that the kind of thing you were thinking? how are people handling this today, we basically have a kiam outage every 3 months resulting in some downtime and manual intervention which is becoming painful, often the CA cert will renew leaving the agent/server certs invalid until they renew which may or may not be shortly after the CA cert.

[meyskens]

This is something we've been planning to do for a while but isn't easy to execute properly. It is on our longer term roadmap, however PRs are always welcome

[anguslees]

I think the solution here requires preserving both the old and new CA key for some overlapping period during the renewal. This also means publishing a "caBundle" rather than just a single CA in secret/ca.crt, etc.

1. t=0: We have only the original CA. Life is simple.
2. t=$renewalTime: Generate new CA key. We now have two CA keys (old and new), _and both are valid_. Clients should be given both so they can validate that remote key satisfies _either_ CA.
3. All new cert signatures should be performed with the "newest" key (or key with latest notAfter?)
4. Publish both CA keys to all clients. In cert-manager, I think this means including both CA certs in Secret.data ca.crt.
5. All (valid) certificates signed by original CA key need to be regenerated/re-signed by new CA key. I think this is best handled by making the CA rotation time longer than the leaf cert renewal time and let natural cert rotation do its thing - but it _could_ be handled by explicitly re-signing everything earlier than expected.
6. t=$notAfter: Original CA key has expired and can be deleted. Remove from ca.crt caBundle. Clients should ignore it anyway, because we're outside notAfter.
7. Goto 1.

For cert-manager, I think this means we need a way to keep both keys around somehow.
This implies we need either a "secret selector" rather than just a single "secretName". Alternatively, I _think_ we only need the old CA public key, so we could just preserve the old CA tls.crt somewhere in the ca/self-sign Issuer itself as a special-case, or preserve any existing ca.crts in the target Secret as part of leaf Cert re-generation...

(cainjector should obviously also be updated to inject multiple CA certs.)