Enhancements: CRD Installation Mechanism

[saad-ali]

Feature Description

• One-line feature description (can be used as a release note):

How should CRDs that Kubernetes depends be deployed?

Kubernetes is moving towards a model where new API objects are not added to the core API but are instead defined and installed as CRDs. When core controllers depend on a CRD, some component in core must install and manage the CRD. Having the controller install it or using addon manager both have drawbacks. This issue tracks coming up with a solution that will address

Feature info:

• Primary contact (assignee): TBD
• Responsible SIGs: /sig api-machinery ? /sig cluster-lifecycle ?
• Design proposal link (community repo): TODO
• Link to e2e and/or unit tests: TODO
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @saad-ali
• Approver (likely from SIG/area to which feature belongs): @lavalamp @liggitt
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y): 1.15?
  
  
  • Beta release target (x.y): 1.16?
  
  
  • Stable release target (x.y): 1.17?
  
  

[verult]

@saad-ali we may also want to add more validation through external admission webhooks (preserving set and map properties in CSINodeInfo fields, for example). @msau42 @jsafrane

[yastij]

kubernetes/community#1937 could benefit from such controller

[justaugustus]

/kind feature
/stage alpha
/assign @saad-ali

Feel free to reassign once an assignee is determined.

[yastij]

cc @kubernetes/api-approvers

[saad-ali]

Short term mitigation for SIG-Storage CRDs until this feature is ready:

• Disable CSI plugin initialization for headless mode, and move CSI CRD initialization from runtime to CSI plugin init time
• Block CSI plugin initialization on CRD creation and CRD sync (if feature is enabled).

Long term plan:

• Will discuss with API Machinery and report back

[AishSundar]

@saad-ali how much work is left for this to land in Alpha in 1.13 and how confident of getting this in for 1.13?

[claurence]

@saad-ali is there an update on how much work is left for making 1.13? Enhancement freeze is tomorrow COB. If there is no communication or update on the PR, this is going to be pulled from the milestone as it doesn't fit with our "stability" theme. If there is no communication after COB tomorrow, an exception will be required to add it back to the milestone. Please let me know where we stand. Thanks!

[saad-ali]

I sync'd with SIG API Machinery last week: they said to follow up with SIG Cluster Lifecycle. I have an agenda item on the SIG Cluster Lifecycle agenda tommorow (10/23) to discuss.

My bet is that this item will not get picked up for 1.13. Will update after syncing with SIG Cluster Lifecycle tomorrow.

Regardless, as far as SIG storage is concerned we will just use the existing add on manager to install the CRDs we need so we should be unblocked.

[timothysc]

/cc @kubernetes/sig-cluster-lifecycle

[roberthbailey]

/sig cluster-lifecycle

[AishSundar]

From the SIG Cluster meeting notes looks like it was agreed to move this out to 1.14. Is that right @saad-ali @roberthbailey @timothysc ? If so should we be tracking any other issues/PRs as a workaround? if so @saad-ali can you point to any pending work?

[saad-ali]

I spoke with Cluster Lifecycle. They agreed that the problem needs to be addressed, and will take ownership of the project long term. But they do not have the bandwidth to address it in this release. So we can remove this from the 1.13.

Short term recommendation was to use the addon manager. I will create a PR for that and link it back to this bug.

[saad-ali]

Removed the following "Requirements" from the description:

• CRD YAML files MUST be checked in as YAML strings instead of typed go structs.
• CRD YAML with validation SHOULD be auto-generated from types.go, if possible.
• Owners of CRD must be api-reviewers.
• Handle race condition between CRD installation and initialization in kube binaries. [Ideally CRD should be installed prior to initialization of kube binaries. Since we want to minimize the amount of time we block kube binary initialization on waiting for CRD to exist.]
• CRD Informer should gracefully handle uninstallation and reinstallation of CRD.

These are issues that we need to think about when we modify Kubernetes components to use CRDs but not necessarily requirements for how the CRDs gets deployed on k8s.

[spiffxp]

/milestone clear
@kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-storage-feature-requests @kubernetes/sig-architecture-feature-requests FYI discussed during SIG Cluster Lifecycle today, and they are not planning on prioritizing for this release cycle. Agree it's important, and it feels like it's roughly in the same arena as addon management, but there's no forcing function on their end.

First step seems like there needs to be a proper KEP that is run through SIG Arch

If you feel this needs to be a priority for this release cycle, please discuss

[luxas]

I think that before proceeding further with this and even starting to think about beta/GA for the related CSI CRDs, we need a KEP where we can discuss this in more detail. It's not clear to me how this will work, but I don't think this is a SIG Cluster Lifecycle issue, as this shall be handled by core as-is, not the ecosystem.

[kacole2]

I'm the Enhancement Lead for 1.15. I don't see a KEP so it doesn't look like this feature can be tracked.

If something changes please PM me.

[msau42]

With respect to CSI, we unblocked ourselves from this issue by moving back to in-tree APIs.

Do @kubernetes/sig-api-machinery-feature-requests or @kubernetes/sig-cluster-lifecycle still want to keep this issue around?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[rbitia]

Hi @saad-ali , I'm the 1.16 Enhancement Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If it's not graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[msau42]

sig-storage doesn't need this feature anymore. We don't have in-tree controllers that depend on CRDs.
/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

sig-storage doesn't need this feature anymore. We don't have in-tree controllers that depend on CRDs.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.