Enhancements: Subresources for Custom Resources

[nikhita]

Feature Description

• One-line feature description (can be used as a release note): Support for Subresources for Custom Resources.
• Primary contact (assignee): @sttts, @nikhita
• Responsible SIGs: SIG API Machinery
• KEP:
  
  
  
  • https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20180415-crds-to-ga.md
  
  
• Design proposal link (community repo):
  
  
  
  • https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/customresources-subresources.md
  
  
• Link to e2e and/or unit tests:
  
  
  
  • https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/test/integration/subresources_test.go
  
  
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  
  
  
  • @sttts
  
  
  • @deads2k
  
  
  • @lavalamp
  
  
  • @liggitt
  
  
• Approver (likely from SIG/area to which feature belongs):
  
  
  
  • @sttts
  
  
  • @deads2k
  
  
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (1.10)
  
  
  • Beta release target (1.11)
  
  
  • Stable release target (TBD)
  
  

---

Originally a part of the main features issue for CRDs: https://github.com/kubernetes/features/issues/95
Design Proposal PR: https://github.com/kubernetes/community/pull/913
Implementation PR (alpha): https://github.com/kubernetes/kubernetes/pull/55168
Docs PR (alpha): https://github.com/kubernetes/website/pull/7439
Implementation PR (beta): https://github.com/kubernetes/kubernetes/pull/63598
Docs PR (beta): https://github.com/kubernetes/website/pull/8519

/kind feature
/sig api-machinery
/stage beta

[nikhita]

/cc @idvoretskyi @justaugustus

[nikhita]

@idvoretskyi since you asked for a new issue to be created (and for it not to be a part of https://github.com/kubernetes/features/issues/95).

[nikhita]

Not sure how to add the stage label. /stage beta didn't work.

[nikhita]

Have updated the feature tracking spreadsheet to include this.

[justaugustus]

Thanks for adding this, @nikhita!

[guineveresaenger]

@nikhita - for release team purposes, would you mind giving a summary of current completion criteria for this issue? :)

There's a bit of a k/test-infra rabbit hole on this and it's unclear which PRs are necessary to fix this bug, and which ones arose in context.

[nikhita]

current completion criteria for this issue

@guineveresaenger I'm confused. :) can you elaborate on what you mean by completion criteria? is it things required to be done for this to be stable?

There's a bit of a k/test-infra rabbit hole on this and it's unclear which PRs are necessary to fix this bug, and which ones arose in context.

I'm still confused :) do you mean PRs that were implemented for this feature?

[guineveresaenger]

Apologies for being unclear - yes, I would like to have a summary of the work currently underway that will implement this feature, since some of them aren't tagged with a 1.11 milestone, and there's a bunch of them. :)

[nikhita]

@guineveresaenger :)

For 1.11, this feature was promoted to beta. Along with it moving to beta, there has been one bug fix and one feature addition. The feature addition is split between 2 PRs.

• https://github.com/kubernetes/kubernetes/issues/62786: issue for promoting to beta
• https://github.com/kubernetes/kubernetes/pull/63385: disallow update to all fields, not just spec and metadata, during /status updates (bug fix)
• https://github.com/kubernetes/kubernetes/pull/63533: allow required at root of schema when status subresource is used (first PR for feature)
• https://github.com/kubernetes/kubernetes/pull/64766: allow description at root of schema when status subresource is used (second PR for feature)
• https://github.com/kubernetes/kubernetes/pull/63598: PR for promoting the feature gate to beta
• https://github.com/kubernetes/website/pull/8519: first docs PR
• https://github.com/kubernetes/website/pull/8918: second docs PR (contains minor update)

[nikhita]

Trying something for automation.

/remove-stage beta

[fejta]

/remove-stage beta

[fejta]

/stage beta

[nikhita]

@idvoretskyi @justaugustus for 1.12 tracking purposes: there will be some small additions and maybe bug fixes but this will remain in beta for 1.12 (so no change from the features perspective).

[kacole2]

@nikhita This feature was worked on in the previous milestone, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.12. If not, confirm this will remain in beta for 1.12.

---

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

[nikhita]

If not, confirm this will remain in beta for 1.12.

Yes, this will remain in beta for 1.12.

[justaugustus]

Thanks for the update, @nikhita!

[nikhita]

This feature is going to remain in beta for 1.12. Listing changes to this feature between 1.11 and 1.12 so that it's easier to track for anyone following along (will update this list if anything else comes up):

• https://github.com/kubernetes/kubernetes/pull/65357: More fields are allowed at the root of the CRD OpenAPI schema now (when status is enabled). https://github.com/kubernetes/website/pull/9973 is the docs PR.

• https://github.com/kubernetes/kubernetes/issues/67235: If subresources were being used _and_ the CRD contained unconventional pluralization, custom resources could not be created. https://github.com/kubernetes/kubernetes/pull/66249 is the bug fix and it has been cherry-picked to 1.10 in https://github.com/kubernetes/kubernetes/pull/67393 (present in v1.10.7) and to 1.11 in https://github.com/kubernetes/kubernetes/pull/67240 (will be present in v1.11.3).

• https://github.com/kubernetes/kubernetes/issues/67541: resourceVersion for custom resources was incremented even on no-op updates. This affected the main resource and the status and scale subresources. https://github.com/kubernetes/kubernetes/pull/67562 is the bug fix and https://github.com/kubernetes/kubernetes/pull/67616 is the cherry-pick to 1.11 (should be present in v1.11.3).

[kacole2]

@nikhita is this remaining in beta for 1.13 as well?

This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:
Docs (open placeholder PRs): 11/8
Code Slush: 11/9
Code Freeze Begins: 11/15
Docs Complete and Reviewed: 11/27

[nikhita]

@nikhita is this remaining in beta for 1.13 as well?

Yes, this is remaining in beta for 1.13 too.

[nikhita]

This feature is going to remain in beta for 1.13. Listing changes to this feature between 1.12 and 1.13 so that it's easier to track for anyone following along (will update this list if anything else comes up):

1. The behaviour of how the .metadata.generation value of a custom resource is incremented has been changed in https://github.com/kubernetes/kubernetes/pull/69059. https://github.com/kubernetes/website/pull/10705 is the docs PR. Now, the behaviour is as follows:

   • A custom resource is considered to participate in the spec/status convention if and only if the CustomResourceSubresources feature gate is enabled and the CRD has the field .spec.subresources.status={} (this is same as before).

   • If the custom resource participates in the spec/status convention, the .metadata.generation value is incremented for all changes, except for changes to .metadata or .status (before, it was only incremented if .spec changed).

   • If the custom resource does _not_ participate in the spec/status convention, the .metadata.generation value is incremented for all changes, except for changes to .metadata (before, it did not increment at all and was initialized to the value 1).

[lawrencejones]

Hey @nikhita! Firstly thank you for all the work you've been doing on this. It's really great to see how fast custom resource definitions are being developed, especially as they provide so much power to developers who use the kubernetes platform.

One thing that was really useful to see in the scope of your proposal were the non-goals, specifically:

Allow defining arbitrary subresources i.e. subresources except /status and /scale.

I reckon doing this was absolutely the right first step, and probably chunked the work into something much more achievable.

I was wondering though, is the community keen to extend CRDs in future to allow arbitrary custom subresources? Opening the definition up to provide a more generic subresource, especially given how that would integrate with the cluster's RBAC rules would solve several problems for us.

I have a suspicion this isn't the place to ask, but I didn't know where else to put my question. If you could help me figure out where to get more information about my query I'd really appreciate the help. Thanks!

[nikhita]

I was wondering though, is the community keen to extend CRDs in future to allow arbitrary custom subresources?

@lawrencejones I don't think we intended to support custom subresources (the idea is to use an aggregated apiserver if someone needs to use it). Having said that, we don't really have a formal consensus from the community at this point. I have created https://github.com/kubernetes/kubernetes/issues/72637 for discussing this.

[lawrencejones]

Thanks @nikhita, I'll track this in the thread you've created 🤲

[claurence]

@nikhita Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[nikhita]

I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release.

@claurence None, this feature will remain in beta for 1.14 as well.

[kacole2]

@nikhita anything targeted for 1.15 here? KEP still needed here to progress as well.

[nikhita]

@nikhita anything targeted for 1.15 here?

No, this will remain in beta.

[liggitt]

No, this will remain in beta.

I actually expected this to graduate so that it was not separately feature-gated. I'm not aware of any outstanding requirements for GA for this

[nikhita]

Discussed about this in the #sig-api-machinery channel on Slack: https://kubernetes.slack.com/archives/C0EG7JC6T/p1555344911023900

This will remain in beta until the main CRD API graduates to GA i.e. this will remain in beta for 1.15.

cc @roycaihw

[sttts]

Docs PR for 1.15 (change that scale subresource can be under .spec): https://github.com/kubernetes/website/pull/14583

@nikhita can you update the issue description?

[liggitt]

/milestone v1.16
/stage stable

[mariantalla]

Hey @sttts / @nikhita - From looking at labels it seems that the plan is for this feature to go stable in 1.16, is that right? If yes, I'll go ahead and add it to the 1.16 Tracking Spreadsheet.

If I've got this wrong and it's not graduating, please let me know I will remove it from the milestone.

Also, please list any remaining k/k PRs in this issue so they can be tracked properly.

As a reminder, 1.16 milestone dates: Enhancement Freeze 7/30 and Code Freeze 8/29.

Thanks!

[mariantalla]

...and one last thing: friendly reminder that there's no KEP linked to the feature, and it will need one to go GA.

cc @nikhita @sttts @liggitt

[liggitt]

friendly reminder that there's no KEP linked to the feature, and it will need one to go GA.

added

[simplytunde]

Hey, @sttts @nikhita I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[kacole2]

@nikhita @sttts @liggitt Enhancement Freeze is tomorrow and the KEP is currently in a Provisional state. Anyway to get this modified to Implementable for 1.16 release?

[liggitt]

Anyway to get this modified to _Implementable_ for 1.16 release?

Yes, that is getting finalized today.

[kacole2]

@sttts @liggitt @nikhita code freeze for 1.16 is on Thursday 8/29. Are there any outstanding k/k PRs that still need to be merged for this to go Stable?

[kacole2]

@sttts @liggitt @nikhita today is code freeze. are there any outstanding PRs? If I do not receive a response this will be dropped from v1.16

[liggitt]

there are no outstanding PRs

[liggitt]

docs ready for review in https://github.com/kubernetes/website/pull/15982

[liggitt]

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

Released as stable in v1.16.0

Post-GA work tracked in https://github.com/orgs/kubernetes/projects/28

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.