Enhancements: Dynamic Kubelet Configuration

[mtaufen]

Dynamic Kubelet Configuration

• One-line feature description (can be used as a release note):
  New Kubelet configurations can be rolled out in a live cluster
• Primary contact (assignee): @mtaufen
• Responsible SIGs: sig-node
• Design proposal (community repo)
• Documentation:
  
  
  
  • Low-level manual workflow
  
  
• Tests:
  
  
  
  • https://k8s-testgrid.appspot.com/sig-node-kubelet#kubelet-serial-gce-e2e&include-filter-by-regex=DynamicKubeletConfig
  
  
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @liggitt @thockin @dchen1107 @dashpole
• Approver (likely from SIG/area to which feature belongs): @dchen1107
• Feature targets (which target equals to which milestone):
  
  
  
  • Alpha release target (1.8):
  
  
  • [x] Foundational primitives and Kubelet behavior implemented behind an Alpha feature gate
  
  
  • 1.9:
  
  
  • [x] Incremental improvements working towards 1.10 goals
  
  
  • 1.10:
  
  
  • [x] Improved testing, see https://github.com/kubernetes/kubernetes/issues/50217
  
  
  • [x] Kubelet configuration type reviewed and moved to beta
  
  
  • [x] Kubelet loading configuration from a file moved to beta
  
  
  • Beta release target (1.11):
  
  
  • [x] Node authorizer automatically allows access to the source specified by Node.Spec.ConfigSource https://github.com/kubernetes/kubernetes/pull/60100
  
  
  • [x] Unpack remote configs to directories on the node: https://github.com/kubernetes/kubernetes/pull/59692
  
  
  • [x] Make the Kubelet config key explicit:
    
    
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/59847
    
    
    
  
  
  • [x] Simplify skew scenarios/debugging by respecting ConfigMap mutations
    
    
    
    
    
    • [x] Issue: https://github.com/kubernetes/kubernetes/issues/61643
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/63221
    
    
    
  
  
  • [x] Expose a structured status for dynamic config, rather than a node condition:
    
    
    
    
    
    • [x] Issue: https://github.com/kubernetes/kubernetes/issues/56896
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/63314
    
    
    
  
  
  • [x] Expose metrics for Kubelet config:
    
    
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/57527
    
    
    
  
  
  • [x] Kubelet config validation should respect target feature gate combination
    
    
    
    
    
    • [x] Issue: https://github.com/kubernetes/kubernetes/issues/63305
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/63409
    
    
    
  
  
  • [x] Improve test coverage of the file utils (pkg/kubelet/kubeletconfig/util/files/files.go):
    
    
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/63496
    
    
    
  
  
  • [x] Document the structure of the file tree under --dynamic-config-dir, other docs updates for 1.11
    
    
    
    
    
    • [x] PR: https://github.com/kubernetes/website/pull/8766
    
    
    
  
  
  • [x] Document per-field advise for cluster administrators and k8s-as-a-service providers
    
    
    
    
    
    • [x] PR: https://github.com/kubernetes/kubernetes/pull/64354
    
    
    
  
  
  • [x] Dynamic kubelet configuration moved to beta
  
  

[mtaufen]

Alpha PR: https://github.com/kubernetes/kubernetes/pull/46254

[dchen1107]

Punt this to v1.8.

[luxas]

@mtaufen Any user/admin-facing docs ready for this feature?

[mtaufen]

No, I really need to write something to teach people how to play with the alpha. What's the preferred place for these docs again?

[luxas]

@mtaufen https://kubernetes.io/docs (https://github.com/kubernetes/kubernetes.github.io)
I'll definitely read what it came to look like finally

[mtaufen]

@luxas https://github.com/kubernetes/kubernetes.github.io/pull/5523

[idvoretskyi]

@mtaufen can you update the features tracking board with the relevant data?

Thanks.

[mtaufen]

Done

[pipejakob]

Looks like the proposal has moved to:

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/dynamic-kubelet-configuration.md

@mtaufen Want to update the link in the description?

[mtaufen]

Done, thanks!

[luxas]

Updated the labels here, @mtaufen please share the beta graduation requirements here for visibility

[mtaufen]

Deprecated and experimental fields removed from the KubeletConfiguration type in https://github.com/kubernetes/kubernetes/pull/53088. Next steps: push the type to beta, chase down loose ends (testing, node authorizer, etc.) and push dynamic config to beta.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[mtaufen]

/lifecycle frozen

[idvoretskyi]

@mtaufen can you comment on the actual feature status?

[mtaufen]

still pushing for beta in 1.10

[idvoretskyi]

Ref https://github.com/kubernetes/kubernetes/pull/55718

[timothysc]

From https://github.com/kubernetes/kubernetes/pull/60412 it appears beta is a push to 1.11

[mtaufen]

@timothysc that is correct. I was hoping the node authorizer checkbox for v1.10 could at least make it, but that seems increasingly unlikely given the time constraints...

[luxas]

This is then wrongly flagged beta in http://blog.kubernetes.io/2018/03/first-beta-version-of-kubernetes-1-10.html, as this label is beta here, swapping.
cc @nickchase (thanks for all your hard work on the article, seems to have been a communication issue here)

[mtaufen]

Thanks for catching that @luxas.
@nickchase what do we need to do to update the post?

[justaugustus]

@mtaufen
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[mtaufen]

@justaugustus does stage indicate the target for the milestone, or the current status?

[justaugustus]

@mtaufen the stage represents the target for the current milestone (v1.11).

[mtaufen]

Thanks, updated.

[mdlinville]

@mtaufen please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

[mtaufen]

@mistyhacks done, thanks!

[kacole2]

@mtaufen This feature was worked on in the previous milestone, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.12 since there is nothing in the original post. This still has the 1.11 milestone as well so we need to update it accordingly.

If there are any updates, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

[mtaufen]

No plans to graduate for 1.12.

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[mtaufen]

@kacole2 no plans to graduate in 1.13.
Are the milestones already correct for this case (assuming yes given tracked/no)?

[kacole2]

@mtaufen thank you. everything is correct for labels. thanks!

[luxas]

Are there plans to graduate this in v1.14?

[mtaufen]

I'd like to finish the flag migration for KubeletConfiguration, then take a step back and think about what we might want to change, then get KubeletConfiguration to GA, then finally consider how to move Dynamic Kubelet Config forward.

I'm hoping to make some significant progress on the flag migrations this quarter, and then start the discussions of what we might want to change as part of our component standardization work.

[claurence]

@mtaufen Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[mtaufen]

Hi @claurence,
We don't expect anything specific beyond the ongoing flag migrations (which are piecemeal and best-effort). Some more general things may be surfaced in KEPs by the Component Standard WG.

[kacole2]

Hello @mtaufen, I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will require a formal KEP to be included. The community post will not suffice unfortunately.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[mtaufen]

@kacole2 No plans for 1.15. Thanks for asking!

[kacole2]

Hi @mtaufen , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[mtaufen]

No further plans this release, thanks!

On Tue, Jul 9, 2019 at 8:31 AM Kendrick Coleman [e-mail]
wrote:

Hi @mtaufen https://github.com/mtaufen , I'm the 1.16 Enhancement
Lead/Shadow. Is this feature going to be graduating alpha/beta/stable
stages in 1.16? Please let me know so it can be added to the 1.16
Tracking Spreadsheet http://bit.ly/k8s116-enhancement-tracking. If
not's graduating, I will remove it from the milestone and change the
tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs
in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state
with Graduation Criteria explaining each alpha/beta/stable stages
requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/enhancements/issues/281?email_source=notifications&email_token=AAG4TQJABPUD2RVQX4FUHWTP6SVLTA5CNFSM4DJGDYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZQUNBQ#issuecomment-509691526,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG4TQMYXUSQEBH5PI4KFELP6SVLTANCNFSM4DJGDYXQ
.

--
Michael Taufen
Google SWE

[annajung]

Hello @mtaufen , 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see *if this enhancement will be graduating to alpha/beta/stable in 1.17?

*
Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Thank you!

🔔Friendly Reminder

• The current release schedule is
  
  
  
  • Monday, September 23 - Release Cycle Begins
  
  
  • Tuesday, October 15, EOD PST - Enhancements Freeze
  
  
  • Thursday, November 14, EOD PST - Code Freeze
  
  
  • Tuesday, November 19 - Docs must be completed and reviewed
  
  
  • Monday, December 9 - Kubernetes 1.17.0 Released
  
  

• A Kubernetes Enhancement Proposal (KEP) must meet the following criteria before Enhancement Freeze to be accepted into the release

  • PR is merged in
  • In an implementable state
  • Include test plan and graduation criteria

• All relevant k/k PRs should be listed in this issue

[mtaufen]

Thanks Anna for reaching out. There is no ongoing work on this feature for
1.17.

On Tue, Oct 1, 2019 at 6:13 PM Anna [e-mail] wrote:

Hello @mtaufen https://github.com/mtaufen , 1.17 Enhancement Shadow
here! 🙂

I wanted to reach out to see *if this enhancement will be graduating to
alpha/beta/stable in 1.17? *
Please let me know so that this enhancement can be added to 1.17 tracking
sheet https://bit.ly/k8s117-enhancement-tracking.

Thank you!

🔔Friendly Reminder

-

The current release schedule is

• Monday, September 23 - Release Cycle Begins
  
  
  
  • Tuesday, October 15, EOD PST - Enhancements Freeze
  
  
  • Thursday, November 14, EOD PST - Code Freeze
  
  
  • Tuesday, November 19 - Docs must be completed and reviewed
  
  
  • Monday, December 9 - Kubernetes 1.17.0 Released
    
    -
  
  

A Kubernetes Enhancement Proposal (KEP) must meet the following
criteria before Enhancement Freeze to be accepted into the release

• PR is merged in
  
  
  
  • In an implementable state
  
  
  • Include test plans and graduation criterias
    
    -
  
  

All relevant k/k PRs should be listed in this issue

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/enhancements/issues/281?email_source=notifications&email_token=AAG4TQMZ7AXQHRBIH2H5VLTQMPYTRA5CNFSM4DJGDYX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEADHHCA#issuecomment-537293704,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG4TQINRZ4YPBGZVZQ4OB3QMPYTRANCNFSM4DJGDYXQ
.

--
Michael Taufen
Google SWE

[annajung]

@mtaufen Thank you for getting back to me! I will remove this from the tracking sheet 👍

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mtaufen]

/remove-lifecycle stale
/lifecycle frozen
/priority awaiting-more-evidence

For all future Enhancement Leads/Shadows: This feature is not being actively worked on at the moment. Our focus is pushing the underlying ComponentConfig APIs to a more stable state across core components. Once we get there, we will re-evaluate the usefulness of Dynamic Kubelet Config in its current state and consider updating the feature's design. Directly using the Kubelet's current ComponentConfig API results in a number of sharp edges for the dynamic config feature.

We will update this issue when we are ready to focus on improving the relationship to dynamic config. Until then, it is safe to assume that the dynamic config feature will not be graduating in whatever release you are working on. Thanks!

[jeremyrickard]

Thanks for the update @mtaufen.

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[frittentheke]

@mtaufen are you still considering to pick this up again?

[xlgao-zju]

@mtaufen are you still considering to pick this up again?

+1 for this. Do we still work on this?

[mtaufen]

I don't currently have time to work on it. But I'm happy to
consult if someone wants to move it forward.

The approach has a lot of sharp edges right now, because
it just exposes the Kubelet's whole config, and many of those
fields don't make sense to be dynamic. That should probably
be discussed before moving it forward to GA.

[aisensiy]

So https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ this document actually does not work?

[xlgao-zju]

So https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ this document actually does not work?

@aisensiy It works fine for me.

[aisensiy]

So https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/ this document actually does not work?

@aisensiy It works fine for me.

My fault. I did not set the dynamic-config-dir argument.

[kikisdeliveryservice]

Hi @mtaufen

Enhancements Lead here. Any plans for this in 1.20?

Thanks,
Kirsten

[mtaufen]

Not that I know of for 1.20. SIG-Node has been discussing deprecating this feature. @derekwaynecarr do you have a proposed roadmap for that work?

[xlgao-zju]

will we have an alternative feature? since we are using this feature now. @mtaufen

[mtaufen]

We don't have a plan for a native alternative that I am aware of; after
deprecation it would be delegated to platform providers.
ComponentConfig will continue to be available.
Derek may have more details, can you post your use-case here so we have it
documented and take it into consideration?

On Mon, Sep 14, 2020 at 7:48 PM Xianglin Gao [e-mail]
wrote:

will we have an alternative feature? since we are using this feature now.
@mtaufen https://github.com/mtaufen

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/enhancements/issues/281#issuecomment-692429434,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAG4TQNIDZE7SZ2IVF6JEXLSF3IXFANCNFSM4DJGDYXQ
.

--
Michael Taufen
Google SWE

[xlgao-zju]

@mtaufen our k8s cluster will have several node pools(a node pool is a batch of nodes witch have same configs), and each node pool will have it's own kubelet dynamic config configmap. when we wanna change the kubelet config (such as system-reserved, kube-reserved, eviction-hard ) of a node pool, we will just change the configmap of the ComponentConfig. that is convenient.

but without the kubelet dynamic config feature, we will have to update the kubelet config file according to the configmap on each nodes of the node pool.

so, I think we'd better have a native alternative feature, before we deprecate this one.