Sig-release: Backporting fixes and defining exceptions for non supported releases

[jimangel]

Currently the release team only supports & patches n-3 releases. Recently we came across a regression that was introduced in a post-release patch (while the version was still maintained). As discussed in the sig-release meeting on 1/15/2019, a fix will be backported to the non supported version. The specific criteria driving that decision was:

• CI was still available for the version
• The regression was introduced as a patch (and not part of the official release)

Reference PR: https://github.com/kubernetes/kubernetes/pull/72860#issuecomment-454072746

Next steps: We need to document the specific criteria (issue introduced post-release in a patch & CI availability) and choose the document location (somewhere in k/sig-release)

cc: @liggitt @justaugustus @spiffxp

[liggitt]

additional criteria (which should always be true of picks, but bears repeating):

• The issue being fixed is of sufficient severity
• The fix is well understood and contained (doesn’t introduce risk of additional regressions)

[mikebrow]

For consideration as an example in your tbd documentation:

The #74034 pick has been closed do to the n-3 release requirement.

Background:
There was a recent, very public, CVE associated with runc, that resulted in the newly patched runc allocating significant additional memory than it did pre-patch. Enough additional memory is being used in the patched runc to cause a regression with k8s e2e node tests without cherry pick #74034. Given that runc is being updated across all runtimes, docker, containerd, cri-o, ... the result is CI will now not be available for the unpatched 1.10.x releases of k8s when run on nodes with a patched runc.

That said, I can not warrant that there is no risk to additional regressions when running kubernetes 1.10.13 on the patched runc, nor no risk, at all, with using more pod memory in k8s CI. No risk is a .. very high bar.

....

Personal note: I'm still getting used to end of life support statements for less than one year old software releases, due to having as much as 4 releases per year (with significant changes, deprecations, etc..). This rapid release cycle plus n-3 support statement requires customers to upgrade kubernetes, a major element of their stack, at least twice a year or be left behind. Seems, to me from my long history with commercial software products, that there should be one level of support for high severity security related issues and high severity regressions (the greater of m-3 || n = months after release), and another for low severity issues.

tldr.. severity of the issue should be the major determining factor for exceptions for some number of months > 12 for a release.

Cheers, Mike @Random-Liu @yujuhong

[BenTheElder]

I hear what you are saying but if we should support releases for longer than we state we do currently... that's a distinct discussion (See EG the LTS working group). Other than changing that policy, generally the project should actually support what we claim to support, and the CI etc. should not support more than that. We have to draw a line somewhere.

. Given that runc is being updated across all runtimes, docker, containerd, cri-o, ... the result is CI will now not be available for the unpatched 1.10.x releases of k8s when run on nodes with a patched runc.

This is pretty much on the CRI teams to test and support this. We can't support arbitrarily old Kubernetes releases in our general CI because something related to Kubernetes might have a bug or regression someday. The CI tooling supports the same releases as the rest of the project, with a little wiggle room to transition as releases go out of support.

Kubernetes the project does not ship runc.

We expect some support for qualifying security patches to eg containerd against an old k8s release to be handled by vendors that are shipping these old versions, and the CRI vendors, just as we expect them to qualify and support their CNI implementations etc.

---

Also worth noting: the patch above nearly didn't happen for fear of opening up to "but we need to support versions we don't support". It was allowed because:

• it was exceedingly tiny and very unlikely to introduce new problems
• it fixed a regression we introduced in a patch release
• it was shortly after the release was supposed to be unsupported

[mikebrow]

All good... thx for the discussion. Interesting topic.

Cheers, Mike

[justaugustus]

/area release-eng release-team
/help
/priority important-soon
/milestone v1.15

[k8s-ci-robot]

@justaugustus:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/area release-eng release-team
/help
/priority important-soon
/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

/milestone v1.16

[jeefy]

/assign @jimangel

[jimangel]

PR is WIP, planning on landing a new doc here: https://github.com/kubernetes/community/tree/master/contributors/devel/sig-release today.

Please chime in if there's any disagreements about location before I PR. Otherwise we can iterate on the PR.

[alejandrox1]

/milestone v1.17

[guineveresaenger]

/priority important-longterm

[guineveresaenger]

/wg lts

for input for ETA on KEP, as well as information via contributor summit session

@kubernetes/wg-lts

[tpepper]

WG LTS is drafting a KEP proposing small changes to the support lifecycle. Multiple things are out-of-scope there, but the question of this fuzzy 1-2month period has been highlighted as needing resolved. KEP draft is looking for completion this week / next week, with next level discussion slated for contrib summit.

[guineveresaenger]

/remove area/release-team