Enhancements: Limit node access to API

[saad-ali]

Limit node access to API

• One-line feature description (can be used as a release note):
  
  
  
  • A new Node authorization mode and NodeRestriction admission plugin, when used in combination, limit nodes' access to specific APIs, so that they may only modify their own Node API object, only modify Pod objects bound to themselves, and only retrieve secrets and configmaps referenced by pods bound to themselves.
  
  
• Primary contact (assignee):
  
  
  
  • @liggitt, @saad-ali
  
  
• Responsible SIGs:
  
  
  
  • sig/auth
  
  
• KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/0000-20170814-bounding-self-labeling-kubelets.md
• Design proposal link (community repo):
  
  
  
  • https://github.com/kubernetes/community/pull/585
  
  
  • https://github.com/kubernetes/community/pull/911
  
  
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  
  
  
  • @deads2k
  
  
  • @timstclair
  
  
  • @ericchiang
  
  
• Approver (likely from SIG/area to which feature belongs):
  
  
  
  • @smarterclayton
  
  
• Feature target (which target equals to which milestone):
  
  
  
  • 1.7
  
  
  • node authorizer and noderestriction admission beta release
  
  
  • 1.13
  
  
  • continued beta work
  
  
  • restrict node label addition (https://github.com/kubernetes/community/pull/911)
  
  
  • 1.14
  
  
  • restrict node address self-modification
  
  
  • stable release
  
  

[liggitt]

c.f. https://github.com/kubernetes/kubernetes/issues/40476

[liggitt]

c.f. https://github.com/kubernetes/community/pull/585

[luxas]

Anything yet to do @liggitt to get this to stable or is it kind of stable already?

[liggitt]

I'm looking at label/taint control in the NodeRestriction admission plugin for 1.8, then I expect this to be complete.

[tallclair]

/cc @davidopp - thinking about how to control taints & tolerations for a multitenant environment.

[liggitt]

this is specifically about limits which labels nodes can add to themselves, and which taints they can remove from themselves

[liggitt]

removing this from the 1.8 milestone and returning to beta. work to limit node control over labels/taints/addresses will continue in 1.10 timeframe - proposal at https://github.com/kubernetes/community/pull/911

[idvoretskyi]

/kind feature

[idvoretskyi]

@liggitt am I right that it's still beta for 1.10?

[liggitt]

correct, it is beta until kubernetes/community#911 is resolved

work on that is occurring in the 1.10 timeframe

[idvoretskyi]

@liggitt great, thanks

[adohe-zz]

@liggitt I am looking at this feature description limit nodes' access to specific APIs, so that they may only modify their own Node API object, only modify Pod objects bound to themselves, and only retrieve secrets and configmaps referenced by pods bound to themselves, IIUC if I enable this feature, the node access to API will be limited to quite narrow scope, and in a multitenant environment, this will bring more high level security.

[Bradamant3]

@liggitt does this need docs for 1.10? Maybe not? (can't quite tell from comments) Could you please update the 1.10 tracking spreadsheet and any related docs PR? Thanks!

[liggitt]

@liggitt does this need docs for 1.10? Maybe not? (can't quite tell from comments) Could you please update the 1.10 tracking spreadsheet and any related docs PR? Thanks!

no updates in 1.10

[justaugustus]

@mikedanese @liggitt
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[mdlinville]

@liggitt @mikedanese please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

[justaugustus]

@liggitt @mikedanese -- We're doing one more sweep of the 1.11 Features tracking spreadsheet.
Would you mind filling in any incomplete / blank fields for this feature's line item?

[kacole2]

@saad-ali This feature was worked on in the previous milestone, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.12 since you've marked it as beta in 1.12. This still has the 1.11 milestone as well so we need to update it accordingly.

If there are any updates, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

[tallclair]

Still beta in 1.12, with some incremental improvements.

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[liggitt]

Yes, limiting self-labeling capabilities is being worked on for 1.13

[kacole2]

@liggitt still considered beta or graduating to stable?

[liggitt]

Updated milestones to reflect the work targeted for 1.13. Node address reporting remains before this will be promoted to stable (looking toward 1.14 for that)

[claurence]

@liggitt Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[kacole2]

Hello @liggitt @saad-ali , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. Notes say this should have gone stable in 1.14. Is that the case? This will also require inclusion into a KEP to go into 1.15.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[liggitt]

There is already a KEP for this at https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/0000-20170814-bounding-self-labeling-kubelets.md

We are waiting out a deprecation period for the kubelet, and will finish this out in 1.16

[kacole2]

Hi @liggitt @saad-ali , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[liggitt]

This remains in progress in 1.16. The v1.16 portion of the timeline in https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/0000-20170814-bounding-self-labeling-kubelets.md#implementation-timeline will be delivered.

[palnabarun]

Hey there @liggitt , 1.17 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancement-tracking). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[liggitt]

We are waiting out the skew period before changing the API server. The next work is planned for 1.19

[palnabarun]

Thank you @liggitt for the updates. I have removed this from the Enhancements Tracking sheet.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[liggitt]

1.19-targeted work: https://github.com/kubernetes/kubernetes/pull/90307

[kikisdeliveryservice]

Hi @saad-ali @liggitt ,

1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

The KEP PR must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria.

The current release schedule is:

Monday, April 13: Week 1 - Release cycle begins
Tuesday, May 19: Week 6 - Enhancements Freeze
Thursday, June 25: Week 11 - Code Freeze
Thursday, July 9: Week 14 - Docs must be completed and reviewed
Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

Please let me know and I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[liggitt]

Yes, this is completed in 1.19. Relevant PRs:

• https://github.com/kubernetes/kubernetes/pull/90307
• https://github.com/kubernetes/enhancements/pull/1737

[kikisdeliveryservice]

@liggitt just to clarify will this be GA in 1.19?

[kikisdeliveryservice]

/milestone v1.19

[mikedanese]

This didn't follow alpha, beta, GA cycle, but basically it's GA now.

[kikisdeliveryservice]

@mikedanese @liggitt

I notice that the underlying KEP does not have a Test Plan, can you please update the KEP to have one by the Enhancements Freeze (Tuesday May 19th EOD PST) as this is required for inclusion in 1.19 release.

As an additional note, #1620 merged recently, adding production readiness review questions to the KEP template. We are not making it mandatory for the 1.19 release cycle, but it would be great if the PRR questionnaire is filled since the KEP needs to be updated to include the test plan. You can find the template at https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

If you have any questions please let me know.

Thanks!

[kikisdeliveryservice]

As this predates the current KEP format and the linked issues do have tests, we will be marking this as tracked.

[tallclair]

There is partial e2e coverage for this feature, but the tests are currently broken and not run by any of our test jobs, and don't cover the label or taint restrictions. See https://github.com/kubernetes/kubernetes/pull/82551.

[liggitt]

The unit and integration tests exercise all the API restrictions implemented by the feature. This feature does not lend itself to e2e test environments which do not control the kube-apiserver configuration (so cannot guarantee the feature is in use) and cannot easily create false Node objects in cloud environments where the node controller detects and deletes the dummy Node object.

[tallclair]

Do you think we should just delete the e2e tests?

[liggitt]

It's unclear to me what the goal of the e2e tests are currently. To verify functionality, integration tests seem much better suited to me. Are they trying to validate that the feature is enabled?

[annajung]

Hi @liggitt 👋 1.19 docs shadow here! Does this enhancement work planned for 1.19 require new or modification to docs?

Friendly reminder that if new/modification to docs are required, a placeholder PR against k/website (branch dev-1.19) are needed by Friday, June 12.

[liggitt]

No docs updates are required, https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction already describes the user-facing aspects of the feature.

[annajung]

Thank you! I will update the tracking sheet accordingly

[kikisdeliveryservice]

Hi @liggitt !

To follow-up on the email sent to k-dev today, I wanted to let you know that Code Freeze has been extended to Thursday, July 9th. You can see the revised schedule here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.19

We expect all PRs to be merged by that time. Please let me know if you have any questions. 😄

Best,
Kirsten

[mikedanese]

This is done.