Enhancements: Kubelet Client TLS Certificate Rotation

[jcbsmpsn]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubelet client certificate bootstrap and auto-renewal
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/266-kubelet-client-certificate-bootstrap-rotation
• Primary contact (assignee): @liggitt
• Responsible SIGs: sig-auth, sig-node
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target: 1.7
  
  
  • Beta release target: 1.8
  
  
  • Stable release target: 1.19
  
  

[idvoretskyi]

@jcbsmpsn please, provide us with the design proposal link.

[idvoretskyi]

@jcbsmpsn please, provide us with the design proposal link and docs PR link (and update the features tracking spreadsheet with it).
/cc @kubernetes/sig-auth-feature-requests @timstclair

[luxas]

@jcbsmpsn Can you please update this feature's status for v1.8?
AFAIK, beta is targeted, right?

[idvoretskyi]

@jcbsmpsn @kubernetes/sig-auth-feature-requests @luxas can you confirm that this feature is still on track for 1.8?

[ericchiang]

@idvoretskyi Yep! Client certificate rotation will be beta in 1.8 and a release note has been added in the release note draft.

[apsinha]

@jcbsmpsn an addition to the docs for this feature would be very useful for users. Is it already documented?

[jcbsmpsn]

Related documentation updates: https://github.com/kubernetes/kubernetes.github.io/pull/5639

[cjcullen]

cc @alexcope

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mikedanese]

/lifecycle frozen

[justaugustus]

@jcbsmpsn @mikedanese @kubernetes/sig-auth-feature-requests
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[justaugustus]

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y)
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

Set the following:

• Description
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[luxas]

@kubernetes/sig-auth-feature-requests will this feature graduate to stable in v1.14?

[claurence]

@jcbsmpsn Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[kacole2]

Hello @jcbsmpsn , I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also need a KEP to be included.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[kacole2]

Hi @jcbsmpsn , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[palnabarun]

Hey there @jcbsmpsn , 1.17 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancement-tracking). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Please note that all enhancements should have a KEP, the KEP PR should be merged, the KEP should be in an implementable state, have a testing plan and graduation criteria.

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[johnbelamaric]

@mikedanese is this something that you expect any progress on in the 1.18 cycle? I am enhancements shadow for the release team and so we need to know if it should be tracked.

[mikedanese]

This is blocked on the Certificates API GA. No change in 1.18.

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[discostur]

@mikedanese is this real that in k8s 1.17.2 there is still no kubelet server certificate rotation? i have upgraded just to 1.17 and i see only client certificate rotation working ...

$ll /var/lib/kubelet/pki/
insgesamt 20
-rw------- 1 root root 1143 26. Mär 2018  kubelet-client-2018-03-26-22-42-46.pem
-rw------- 1 root root 1143 11. Feb 2019  kubelet-client-2019-02-11-12-06-27.pem
-rw------- 1 root root 1143 10. Feb 11:22 kubelet-client-2020-02-10-11-22-37.pem
lrwxrwxrwx 1 root root   59 10. Feb 11:22 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2020-02-10-11-22-37.pem
-rw-r--r-- 1 root root 1196 14. Feb 2018  kubelet.crt
-rw------- 1 root root 1679 14. Feb 2018  kubelet.key

Regarding the documentation the feature should already be active by default?

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#certificate-rotation

RotateKubeletClientCertificate and RotateKubeletServerCertificate feature flags on the kubelet and are enabled by default.

[liggitt]

The serving certificate feature is active, but is opt-in, since it requires a certificate signing request approval process be set up by the cluster deployer (there is not sufficient information available to the kube-controller-manager to know whether it should approve a serving certificate request from a node for a given IP or DNS name)

[liggitt]

/assign
/milestone v1.19

The CSR API is targeting v1 in 1.19

[kikisdeliveryservice]

Hi @liggitt !

1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

The KEP PR must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria.

The current release schedule is:

Monday, April 13: Week 1 - Release cycle begins
Tuesday, May 19: Week 6 - Enhancements Freeze
Thursday, June 25: Week 11 - Code Freeze
Thursday, July 9: Week 14 - Docs must be completed and reviewed
Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

Please let me know and I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[liggitt]

Yes, this is planned to graduate in 1.19.

The original design and feature pre-dated the KEP process, so https://github.com/kubernetes/enhancements/pull/1756 has been opened to convert it to KEP format.

[kikisdeliveryservice]

/milestone v1.19

[liggitt]

KEP format merged at https://github.com/kubernetes/enhancements/tree/master/keps/sig-auth/266-kubelet-client-certificate-bootstrap-rotation

[palnabarun]

@liggitt -- Thank you for the update. I have updated the tracking sheet accordingly. :+1:

[annajung]

Hi @liggitt 👋 1.19 docs shadow here! Does this enhancement work planned for 1.19 require new or modification to docs?

Friendly reminder that if new/modification to docs are required, a placeholder PR against k/website (branch dev-1.19) are needed by Friday, June 12.

[liggitt]

https://kubernetes.io/docs/tasks/tls/certificate-rotation/ would need updating to note the GA status and non-experimental signing duration flag. Placeholder open at https://github.com/kubernetes/website/pull/21108

[annajung]

Thank you @liggitt , I will update the tracking sheet accordingly

[kikisdeliveryservice]

Hi @liggitt

I see that https://github.com/kubernetes/kubernetes/pull/91116 has merged already, if you have any other PRs, please link them to this issue so that we can track them. As a reminder Code Freeze is June 25th :)

Thanks!!

[kikisdeliveryservice]

Hi @liggitt !

To follow-up on the email sent to k-dev today, I wanted to let you know that Code Freeze has been extended to Thursday, July 9th. You can see the revised schedule here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.19

We expect all PRs to be merged by that time. Please let me know if you have any questions. 😄

Best,
Kirsten

[annajung]

Hi @liggitt, a friendly reminder of the next deadline coming up.
Please remember to populate your placeholder doc PR and get it ready for review by Monday, July 6th.

[kikisdeliveryservice]

Hi @liggitt ,

Is this enhancement now code complete? As a reminder Code Freeze is Thursday July 9th.

Thanks!

[liggitt]

Yes

[annajung]

Hi @liggitt, just a quick reminder to get your doc PR ready for review (Remove WIP/rebased/all ready to go) by EOD. Thank you!

[liggitt]

Doc PR is ready for review

[kikisdeliveryservice]

Hi @liggitt !

Since this KEP is GA in 1.19 can you please update the status to implemented so that we can close this issue?

Thank you!
Kirsten

[liggitt]

KEP update in https://github.com/kubernetes/enhancements/pull/1984

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

KEP update in https://github.com/kubernetes/enhancements/pull/1984

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.