Cert-manager: Kubernetes end-of-life version policy

[munnerz]

Is your feature request related to a problem? Please describe.

Currently we have a very loosely defined restriction as to what Kubernetes version(s) cert-manager is compatible with.

Because we have not clearly outlined the requirements, users sometimes run into issues where they cannot run cert-manager in their environments and they receive not-so-helpful error messages informing them that something isn't right (see: #2054), and we're also uncertain at what point we can introduce new features (https://github.com/jetstack/cert-manager/issues/2060#issuecomment-530261423).

Other instances of this coming up:

• Whether to enable the 'webhook' deployment by default
• Extending the webhook to also include a mutating webhook for defaulting
• Switching on CRD conversions (available from 1.15 in beta)
• Utilising new CRD options like preserveUnknownFields

Describe the solution you'd like

We should work out a policy for what Kubernetes versions we intend to support, and communicate this clearly to users.

This will allow us to push ahead implementing new features without having to wait around for an indeterminate about of time.

Given Kubernetes supports a release for 1y after it is pushed out, I would like to think we can stick to a similar support policy. This would mean we only ever support the last 4 Kubernetes versions that are available (so as of today that would be 1.15, 1.14, 1.13 and 1.12). In a week's time when Kubernetes 1.16 is cut, we would then technically be 'allowed' to drop support for 1.12.

Looking at GKE, the supported 'master' versions as of today:

validMasterVersions:
- 1.14.3-gke.11
- 1.13.9-gke.3
- 1.13.7-gke.24
- 1.13.7-gke.19
- 1.13.7-gke.8
- 1.13.6-gke.13
- 1.12.9-gke.16
- 1.12.9-gke.15
- 1.12.9-gke.13
- 1.12.9-gke.7
- 1.12.8-gke.12
- 1.12.8-gke.10
- 1.12.7-gke.26
- 1.12.7-gke.25

The oldest version here is 1.12, which actually falls neatly in-line with our versioning strategy.
It may be worth attempting to continue to support the 5th oldest release too (i.e. 1.11 as of today, 1.12 as of next week), given that GKE is still offering 1.12 as a valid version (and I don't think they intend to remove this for at least a little while after upstream have cut 1.16).

cc'ing a few people that have either built integrations around cert-manager (so this is directly relevant!) or that often provide critical feedback 😅

/cc @JoshVanL @mikebryant @DirectXMan12 @rimusz

[mikebryant]

I support having an explicit policy.

I think it's also worth considering the EKS version policy, as another major provider in this space. They will drop 1.11 on November 4th, and force an upgrade for all users.

It would be nice to support all versions of kubernetes offered by GKE, EKS etc.

I'm not too concerned how many versions cert-manager actually supports - having the policy written down is sufficient, and very helpful for enterprise users who can use that policy to support prioritising upgrades against feature development. (In particular given Lets Encrypt will be blocking older versions of cert-manager, which forces upgrades there too)

[stuart-warren]

stating supported k8s versions in release-notes would also be handy as that's where you tend to look for breaking changes when upgrading :)

[DirectXMan12]

From the KubeBuilder perspective, I think we're on board with four versions (adding wiggle room for a bit right after a release is probably a good idea -- practically very few people upgrade immediately, even on self deployed, AFAIK). That's more-or-less what we've been unofficially doing (and occasionally even a superset by accident), so I don't think it would break us.

[munnerz]

We've just run into #2200 which means our CRDs are no longer compatible with Kubernetes 1.11.

This sort of drives home to me that support for the N-5th version of k8s is not going to be easy to maintain/guarantee - technically, there could be incompatible changes between version N and N-5, which could be impossible to work around.

I propose we don't officially support N-5, although do keep periodic/CI jobs enabled for it so that we have an idea of when it breaks, and can communicate that to users clearly. This means it's "best effort", and we as a project are allowed to make the executive decision to drop support at any point if we need to.

For what it's worth, EKS is actually forcing upgrades upon users with Kubernetes 1.11 in November, and GKE does not offer 1.11 as an option as of quite a while ago (see the supported version list in the original post).

How does this sound @mikebryant @stuart-warren @DirectXMan12?

[DirectXMan12]

SGTM!

[grzechukol]

I see that you are focusing on Kubernetes only when it comes to backwards compatibility but what about OpenShift? Maybe it is worth to reconsider, just for the short switchover time from OpenShift 3.11 to OpenShift 4.X to change your support policy?

[munnerz]

ref #2187

[munnerz]

This is being documented in https://github.com/cert-manager/website/pull/15