kubebuilder and controller-* libraries should follow k8s release cadence

[tamalsaha]

Kubebuilder is becoming the common way to develop operators /controllers for Kubernetes. With the release of various new features of CRD, it seems that new annotations are added to kubebuilder instead of code-generator repo. But one issue with kubebuilder and its controller-runtime and controller-tools repos is that these are not released with the same cycle as Kubernetes. In recent times, these libraries have been updated months after the last Kubernetes release. This is not a criticism of the fine folks behind kubebuilder. As a developer of Kubernetes operators/controllers, we would like to see that these tools become part of the same release cadence as Kubernetes. This will us to experiment with the new features of CRDs and release own tools and products faster after a new release of Kubernetes.

/kind feature
/sig api-machinery
/sig architecture
/area custom-resources

[DirectXMan12]

Kubernetes' release cadence (and, more broadly, long release cadences) are not well suited to healthy development practices, IMO. They encourage stressful pre-code-freeze crunch times, delay features because they were not able to make some arbitrary calendar date, and generally cause much misery as folks battle between trying to ensure that a release is stable versus not having to wait months to get their feature in.

We should be striving to decouple release cadences, not couple them further.

If you were to argue that we should release KubeBuilder and friends more frequently, I'm all for that -- ideally, we're releasing new minor versions every few feature PRs (basically, whenever there's enough momentum).

If you were to argue that we've done a poor job staying on top of Kubernetes releases, and that we should upgrade more aggressively, I'd buy that (it's a bit more complicated due to compatibility, but that's not an insurmountable challenge).

However, as it stands, I'm strongly opposed to coupling to the Kubernetes release cadence -- it'll only hamper KubeBuilder, and I'm not convinced that it's the best way to deal with the version skew problem.

[DirectXMan12]

Following up on this:

In my experience (and based on feedback I've gotten), there are a few problems being bundled into one here:

1. As a controller-runtime user, I want to be able to make use of the latest Kubernetes API machinery features at any given point in time

2. As a controller-tools user, I want to be able to generate CRDs that take advantage of the latest api extension fields (in CRDs, webhook configs, etc)

3. As a controller author, I want reassurance that a given version of my API types and API machinery will work with my target Kubernetes versions

4. As a controller author, I want assurance that generated manifests will work well with my target Kubernetes versions

5. As a controller author, I want to be able to include controller-runtime and third-party libraries that both depend on Kubernetes API machinery

6. As a controller-{runtime,tools} user, I want to be able to make use of the latest controller-{runtime,tools} features, independent of Kubernetes features

(feel free to comment if there's a case I haven't covered)

[DirectXMan12]

Let's look at some solutions, and see where they break down. First, though, some terminology:

• generic library/types/etc: types that aren't inherently tied to a Kubernetes API. They're tied to "Kubernetes the API platform" and not "Kubernetes the cluster orchestrator"
• specific library/types/etc: types that are inherently tied to a Kubernetes API. They're tied to "Kubernetes the cluster orchestrator"
• non-versioned change: a change to a Kubernetes API that adds fields in such a way that it's impossible to detect if your manifest will be applied succesfully without looking at the exact schema for an Kubernetes (not API) version. For instance, CRDs added multi-version support without changing the API version. Creating a multi-version CRD will fail on an older cluster, despite it purporting to accept that API version.

[DirectXMan12]

1. As a controller-runtime user, I want to be able to make use of the latest Kubernetes API machinery features at any given point in time

Relevant: 1, 3, 5, 6.

TL;DR: "create a minimal, stable subset of API types and functionality for controllers and clients, and don't break that" looks most promising, but requires a lot of work.

Solution: keep controller-runtime always on the latest k8s.io/apimachinery

• Problem: k8s.io/apimachinery often changes incompatibly, meaning we need the latest k8s.io/client-go and k8s.io/api.
• Problem: k8s.io/client-go contains both specific and generic functionality.

Solution: keep controller-runtime always on the latest k8s.io/*

• 1: solved!
• 3: in conflict -- using a newer version of k8s.io/apimachinery could break things in your cluster (unlikely, but it could happen), but using a newer k8s.io/api version very well might mean you accidentally use features not yet support by the specific APIs, or can't make
  use of removed API types (less of an issue now-a-days, but still potentially vexing).
• 5: in conflict -- incompatible type changes mean that the types that controller-runtime depends on could break if you switch versions, and vice-versa with the third-party libraries
• 6: in conflict due to 3 & 5

Solution: version k8s.io/apimachinery with stable versions, make a generic stable client-go

• Problem: API machinery is shared across Kubernetes, and often has to change in ways that aren't compatible, but normally wouldn't affect end users.
• Problem: Kubernetes has historically not been interested in stable versions for Go libraries, as it hampers development speed.
• Problem: Practically, there are breaking changes every release, so every release would be a major change.

Solution: version k8s.io/* with major module import path versions every release

• Problem: this means that people have to update their import path when they want to get new features, since API machinery features are often not backported

Solution: create a minimal, stable subset of API types and functionality for controllers and clients, and don't break that

• 1: Solved!
• 3: while we'd have to be careful to not break API machinery versions under the hood, this'd make it a lot easier to "bring your own API version"
• 5: this should make it possible to use whatever API version you want, and even bring your own generated clients, etc, as long as they depend on the same major version of this
• 6: solved, due to 1, 3, & 5

Caveat: This requires extra effort on the Kubernetes side to not break this, and figure out what it includes.

Solution: version controller-runtime with Kubernetes

• 1: Solved!
• 3: Solved!
• 5: Impossible -- if a library requires a specific version of CR, you're out of luck
• 6: Impossible -- you can't upgrade CR without also upgrading your Kubernetes version

Solution: Use completely our own interfaces, fork API machinery

NB: this is written mostly as a strawman -- there are a number of impracticalities here, not to mention it's not very good for both sets of code. I don't think we'd do this.

• 1: Solved, but only if you look at it sideways -- onus is on the KB maintainers to cherry-pick changes over, likely resulting in increased latency
• 3: Solved
• 5: Weird -- you can't generate interface implementations for types that you don't own, and it's likely that libraries are designed to work with the existing Go types, so generating a new set would cause issues. However, if the libraries were design to work with CR, it would be less of an issue

• 6: Solved

• Problem: this is a lot of work, almost impractical

• Problem: this would require extra code generation for existing types, "forking" existing API types

[DirectXMan12]

1. As a controller-tools user, I want to be able to generate CRDs that take advantage of the latest api extension fields (in CRDs, webhook configs, etc)

Relevant: 2, 4, 6.

TL;DR: there's nothing that makes everyone happy here that I can see.

Solution: Always update to the latest Kubernetes version in controller-tools, keep compatibility

• 2: solved!
• 4: potentially in conflict, but only potentially: we'd need to be very careful to not add new features that were on by default, to avoid hitting non-versioned changes in the Kubernetes API
• 6: not solved -- upgrading controller-tools requires working with manifests generated according to newer versions -- rely on KubeBuilder maintainers to make sure things don't break

Solution: version controller-tools with Kubernetes without regards to compatibility

• 2: Solved!
• 4: Solved!
• 6: Impossible -- you can't upgrade controller-tools without also upgrading your Kubernetes version

[DirectXMan12]

1. As a controller author, I want reassurance that a given version of my API types and API machinery will work with my target Kubernetes versions

1. As a controller author, I want assurance that generated manifests will work well with my target Kubernetes versions

Mostly discussed above -- these are constraints to consider when figuring how how to solve the problems introduced by 1 and 2.

1. As a controller author, I want to be able to include controller-runtime and third-party libraries that both depend on Kubernetes API machinery

The solution from 1 ("create a minimal, stable subset of API types and functionality for controllers and clients, and don't break that")

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[DirectXMan12]

We're getting progress! deads2k is putting together a KEP for solution 1 :tada:

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.