Kops: Adopt a Formal Deprecation Policy

[rifelpet]

We have discussed dropping support for older versions of Kubernetes for a while now, and I'd like to formally establish a deprecation policy. I think there are a few points of interest:

• How many stable minor versions of Kops are supported? This implies eligibility for cherry-picks and new patch releases
• How many minor versions of Kubernetes should a given minor version of Kops support?
• Do we error if a user tries to use an older version of Kubernetes than is supported? Do we expose a feature flag to override this, letting the user know they will be "on their own"?
• How much advanced notice does Kops need to give for deprecating functionality before it is removed. This would mostly apply to API fields that represent deprecated/removed flags and feature gates on k8s components
  
  
  
  • Do we announce a deprecation warning when a user is using a flag that is deprecated for the k8s version they are using?
  
  
  • Do we remove the API field when the flag has been removed for all supported k8s versions? Would the Kops API version need to change in that case?
  
  
  • How could we surface deprecation warnings from flags / feature gates back to the user? One idea was to actually run a copy of the component binary and look for any warnings / errors related to these fields and feature gates, this way the user sees the issue immediately rather than having to troubleshoot failing nodes.
  
  
• How is the support notice and deprecation warning(s) given to users? Release notes + warning logs?
• For the initial deprecation, once we decide on a number of minor versions we'd have to decide whether the support will be dropped at the next minor release or if it will be dropped over time according to the decided range. Example in the case of 3 minor version support: should kops 1.16.0 drop support for k8s 1.12.0 (given that kops 1.15 could still receive patch releases if we decide on it) or do we announce at the kops 1.16.0 release that kops 1.19.0 will no longer support k8s 1.16.0?

Proposal

I think a reasonable starting point for discussion would be a variation of the Kubernetes support policy itself which states that 3 stable versions of k8s are supported at any given time. Traditionally Kops has only patched the latest 1 stable release and occasionally back ported 2 stable releases, but this is made up for by the extended k8s version support, so perhaps we settle with:

• Kops will support patch releases for the 2 latest stable Kops versions
• Kops will support the same minor version of k8s and the previous 2 k8s minor versions.

Example: When Kops 1.16.0 is released, we will backport bugfixes to Kops 1.16.X and Kops 1.15.X. Kops 1.16.X will support K8s 1.14.X - 1.16.X. Kops 1.15.X will support k8s 1.13.X - 1.15.X. This means that k8s 1.13 - 1.16 are supported by supported versions of kops, 1 minor version longer than the k8s official support.

Do we think that k8s support is too narrow? It is a rather significant change but I think most users stuck on unsupported k8s versions will probably not be using the newest Kops version either.

Any input or alternative viewpoints would be much appreciated. Thanks!

[mikesplain]

Thanks for writing this up @rifelpet, I 100% agree, we need a more formal versioning and deprecation policy.

For those unaware, our current policies are located in our readme here and docs site here.

We officially support the same versions of kops as kubernetes, aka, we support the 3 most recent versions of kops. I do think, since we specifically do not say kops version == k8s version, we should not alway gurantee all bug fixes are back ported. We do our best to back port the ones that make sense and are worth it. If members find a fix we have not back ported that they would like to backport, they should be welcome to do so.

As Kubernetes Operators I believe it is our responsibility to those who can, upgrade. To those hwo may be new to operating, plan to upgrade and plan to run your apps in a way they can be disrupted and moved occasionally.

I personally view almost all of the above as a sane policy that we should better document and make sure our users are aware of. I feel as though this is similar enough to what we've been doing however just needs a little extra highlighting in release notes and office hours. I think it would also be useful if in the long term, we could throw warnings to users running older versions of k8s and kops that are falling or have fallen out of maintenance. We can do some of that now but expanded support would be useful.

Thanks for your hard work on this Peter, great start!

[joshbranham]

I think the above recommendations, as well as the Kubrernetes CLI deprecation policy, cover the bases.

Per @mikesplain point, we should help encourage users to upgrade, as well as design their environments in a way that allows portability. I also tend to agree that supporting more versions just renders people less likely to perform the upgrades from older k8s and kops versions.

Maybe we can discuss in office hours? Otherwise, if you have a rough template already in mind, it seems a PR would be a good place to hash out specifics. Let me know if you want any help in that regard.

[hakman]

Thanks @rifelpet. Sounds pretty reasonable to me.

I would add that dropping support should also mean that would be a good practice to remove workarounds and pieces of code that are used only for Kubernetes versions that are no longer supported. This would make the code cleaner and easier for people to contribute.

Not sure if this the place and time but maybe deprecation should include policies about OS releases and container runtime versions.
OS releases that are EOL should be removed. For example Debian Jessie will EOL soon, on June 30, 2020. At the moment, Docker is no longer building packages for Jessie.
Docker versions that are no longer listed in those validated by for Kubernetes releases should be removed.

[rifelpet]

Yes, one of the benefits of a deprecation policy would be cleanup of legacy code paths and addons. grepping for IsKubernetesGTE and IsKubernetesLT would be a good starting point.

We could certainly consider extending the policy to OS releases as well, and since Kubernetes is tested on a specific set of Docker versions, Kops could drop support for older docker versions as they fall off Kops' list of supported k8s versions.

[hakman]

Considering the new deprecation policy, do we still need KOPS_RUN_OBSOLETE_VERSION ?

[johngmyers]

Yes, that is needed to run versions that are unsupported per the channel, but supported in that kops version per deprecation policy.

[hakman]

@johngmyers I understand the mechanism. Just seems a bit too much.

[johngmyers]

If we're going to continue to obsolete Kubernetes versions through the channel, we're going to need that escape hatch. With the deprecation policy obsoleting through kops versions, we might want to remove the code that obsoletes through the channel.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.