Enhancements: Kubelet TLS Bootstrap

[philips]

Feature Description

• One-line feature description (can be used as a release note): kubelet generates a private key and a CSR for submission to a cluster-level certificate signing process.
• Primary contact (assignee): @mikedanese
• Responsible SIGs: sig-auth
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/kubelet-tls-bootstrap.md
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @liggitt
• Approver (likely from SIG/area to which feature belongs): @liggitt
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (1.4)
  
  
  • Beta release target (1.6)
  
  
  • Stable release target (1.12)
  
  

[philips]

cc @kubernetes/sig-node FYI about this feature for Kubelet TLS bootstrap

[mikedanese]

I think we should consider this pre-alpha until we complete:

• kubectl integration
• kubelet integration
• e2e testing

Is anyone currently working on this?

[philips]

@mikedanese @gtank should be working on all of those things to get the feature done done.

[gtank]

Current status of what needs to be done:

1. kubelet needs to use the Certificates API. This will involve one new flag, provisionally --bootstrap-auth-token which will take a long random string. The kubelet uses this token to authenticate its requests to the Certificates API. Cluster-level access control should ensure that it can only be used for certificates requests. When this token exists, if the kubelet can't find TLS assets locally it will generate a fresh keypair and a Certificate Signing Request (CSR) to submit to the API. It will then watch the CSR object for the appearance of an issued certificate before proceeding with registration. It will use the certificate for client cert auth in subsequent API requests.
2. kubectl needs to fully support CSR objects. This is currently blocked on a problem with the swagger generation and base64-encoded []byte fields. We also need to document the best way of handling the manual approval flow.
3. Still need to write tests and update all the docs.

[philips]

Status update:

PR for kubelet TLS bootstrap is up: https://github.com/kubernetes/kubernetes/pull/30094

[mikedanese]

Kubectl support tracked in kubernetes/kubernetes#30163 with some basic support prs out

[gtank]

Status update for TLS work:

In flight:

• Implement TLS bootstrap for kubelet using --bootstrap-auth-token https://github.com/kubernetes/kubernetes/pull/30094
• Add an option to controller-manager to auto approve all CSRs https://github.com/kubernetes/kubernetes/pull/30153
• Add a certificate signing request resource printer in kubectl https://github.com/kubernetes/kubernetes/pull/30166
• Implement kubectl procelain csr commands https://github.com/kubernetes/kubernetes/pull/30237
• csr: add approval to the typed client https://github.com/kubernetes/kubernetes/pull/30236
• add shortname for certificate signing request in kubectl https://github.com/kubernetes/kubernetes/pull/30165
• swagger: report golang type []byte as json type string https://github.com/emicklei/go-restful/pull/311

Issues:

• certificatesigningrequests should overwrite spec.user info in createstrategy https://github.com/kubernetes/kubernetes/issues/30239
• kubectl should support CertificateSigningRequest https://github.com/kubernetes/kubernetes/issues/30163

Merged:

• add delete strategy to certificates registry https://github.com/kubernetes/kubernetes/pull/30181

[liggitt]

It will use the certificate for client cert auth in subsequent API requests.

The scope of https://github.com/kubernetes/kubernetes/pull/20439/ was around obtaining TLS serving certs. The kubelet's use of this API should probably be limited to obtaining serving certs initially. Since the kubelet already needs API credentials to submit the initial CSR, obtaining client credentials this way doesn't buy us a whole lot from a bootstrapping perspective.

[mikedanese]

It allows us to bootstrap individual kubelet identities from a single cluster wide shared secret (bearer token for a kubelet-bootstrap user authorized to submit CSRs).

[liggitt]

If the goal is to identify kubelets individually so we can partition node permissions (which I am in favor of), multiple identities obtained from a single shared credential aren't meaningful from an auth perspective.

If we just want to identify requests from different nodes for debugging purposes, we can do that with user-agent strings (the same way controllers set up their own client user-agent string)

[mikedanese]

multiple identities obtained from a single shared credential

I'm not convinced of this. In fact I think this would work equally well if the CSR API was open (didn't require auth[n/z]). The shared secret is a mechanism that prevents a specific dos attack where a user could spam the CSR API with CSRs that would not be approved. @gtank can you weigh in on this?

[bgrant0607]

cc @kubernetes/sig-cluster-lifecycle

[philips]

@mikedanese @liggitt can we please move the technical discussion about the design of the system to the https://groups.google.com/forum/#!forum/kubernetes-sig-node mailing list.

[gtank]

@liggitt @mikedanese The nodes derive individual identities from the shared token. The token only exists to 1) filter requests meant for other clusters and 2) allow us restrict the nodes (via groups or ABAC) from accessing the rest of the API before they have client certs.

[pwittrock]

@philips Is this on target for the 1.4 feature freeze this Friday (Aug 19)?

[mike-saparov]

@pwittrock: based on https://github.com/kubernetes/kubernetes/pull/30094 this feature suddenly started being redesigned in flight after 5 months of work... Any suggestions on how to avoid the major redesign three days before feature freeze are welcome

[smarterclayton]

Occasionally we will realize gaps in designed features as they are implemented. We definitely need a process that handles that - rushing features to delivery without all the proper technical due diligence being done is going to cause more issues than occasionally features missing a release. I think everyone involved is trying to find the best and most secure option for the platform here.

[mike-saparov]

@smarterclayton based on PR discussions it would be great if @liggitt and @deads2k could help with parts of the feature, wdyt? any coding / review help is appreciated to address raised concerns.

[smarterclayton]

I believe both of them had been representing sig-auth on the security aspects here - I had not yet seen an update on the remaining issues though. That's probably appropriate on the linked issue.

[gtank]

Update: kubelet support merged in https://github.com/kubernetes/kubernetes/pull/30922

[yifan-gu]

To be more clear, kubernetes/kubernetes#30922 only requests the kubelet client cert from API server. We will need follow up work on getting the kubelet serving cert.

[errordeveloper]

Could you create another issue to clarify that?

On Wed, 24 Aug 2016, 00:18 Yifan Gu, [email protected] wrote:

To be more clear, kubernetes/kubernetes#30922
https://github.com/kubernetes/kubernetes/pull/30922 only requests the
kubelet client cert from API server. We will need follow up work on getting
the kubelet serving cert.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/43#issuecomment-241910465,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWS-UriOVAXjKfqJAoP-mRahm9LOxcks5qi3_TgaJpZM4JS7ez
.

[yifan-gu]

@errordeveloper Basically we need to think about:

• How this continues to combine with the work of dynamic kubelet settings https://github.com/kubernetes/kubernetes/issues/27980
• How we get the serving cert for kubelet
• How to do the cert rotation

Probably need to sync with @kubernetes/sig-auth to get a better idea before creating the issue.

[janetkuo]

@gtank @yifan-gu Are the docs ready? Please update the docs in https://github.com/kubernetes/kubernetes.github.io, and then add PR numbers and check the docs box in the issue description

[calebamiles]

@janetkuo, I'll be tracking our progress on creating the docs. Is there anything besides high level "this is how you turn it on" + known issues that you're expecting for the PR? Also sorry for the delayed response!

[jaredbhatti]

@calebamiles That should be fine, and if you could add a link to the PR in this issue that would help me track it. Thanks!

[jaredbhatti]

@gtank @yifan-gu @calebamiles Another ping on docs. Any PRs you can point me to?

[aaronlevy]

I believe we also still need to figure out next steps for including kubelet server certificates as part of the TLS bootstrap. Based on the kubelet authN proposal - this would greatly simplify the deployment of secure clusters.

Initially discussed in: https://github.com/kubernetes/kubernetes/pull/30094 it sounded like something along these lines might be reasonable?

1. kubelet uses bootstrap-kubeconfig to post client-cert CSR
2. kubelet retrieves signed certificate and is now a "full-client" of api
3. kubelet uses client-cert to post a server-cert CSR
4. kubelet starts api

/cc @gtank @yifan-gu @philips @liggitt

[yifan-gu]

@gtank @yifan-gu @calebamiles Another ping on docs. Any PRs you can point me to?

I can take the item, but what's expected in the doc?
What I can think about is:

• Introduce this new flag --experimental-bootstrap-kubeconfig, why we need it.
• Explain what it does step-by-step
• Give a successful step-by-step guide on how to set up?

Others?

@gtank @aaronlevy

[aaronlevy]

• Was an auto-approver in the controller-manager merged? If so, probably help to document that functionality also.
• Also maybe "adding your own approver section"?

[errordeveloper]

Yeah, the auto-approver is in. FYI, this repo is meant to be low on
traffic. Let's migrate the discussions elsewhere.

On Tue, 13 Sep 2016, 00:54 Aaron Levy, [email protected] wrote:

• Was an auto-approver in the controller-manager merged? If so,
  probably help to document that functionality also.
• Also maybe "adding your own approver section"?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/43#issuecomment-246531802,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPWSzol3PTrv47n-xvFZx3JBQjRvMtYks5qpeYggaJpZM4JS7ez
.

[calebamiles]

@jaredbhatti @janetkuo, we will hopefully have a docs PR ready today

[devin-donnelly]

@calebamiles , Any word on that Docs PR? Please reference if it's up.

[calebamiles]

@devin-donnelly, https://github.com/kubernetes/kubernetes.github.io/pull/1246

[devin-donnelly]

Great, thanks!

[gtank]

As part of the documentation effort, here's a flow chart for how this works as of the 1.4 release.

[mikedanese]

We are aiming to bring the certificates API to beta in v1.6.

[idvoretskyi]

@gtank @mikedanese @yifan-gu please, provide us with the release notes and documentation PR or link at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

Also, please, select the valid checkpoints at the Progress Tracker.

[yifan-gu]

@idvoretskyi Added Introduces an API for the Kubelet to request TLS certificates from the API server.
What is the valid checkpoints?

[mikedanese]

So, certificates API will be beta in 1.6, I'm working on a docs PR for that but also trying to unblock upgrade testing first. Will post in the spreadsheet once it's ready.

[idvoretskyi]

@yifan-gu I'm speaking about the "Progress Tracker" checkpoints at the heading message.

[mikedanese]

PR documenting certificates API is here https://github.com/kubernetes/kubernetes.github.io/pull/2939, the kubelet flags are still experimental in 1.6.

[luxas]

@kubernetes/sig-auth-feature-requests @kubernetes/sig-node-feature-requests
Is this going fully to GA in v1.8 or just a subset of it?
What I can see there are different sub-features in this feature:

• the csrapprover and csrsigner in the controller-manager
• the certificates API group
• the CSR post functionality in the kubelet (the --bootstrap-kubeconfig flag)

Which of these features are being worked on in the v1.8 timeframe?

[mikedanese]

@luxas this depends on the certificates API being GA which I don't see happening in 1.8. There's a lot in early design phase that will either depend on the certificates API in some form (e.g. pod identity, spiffy, istio) or redo the API. I see greater value in getting feedback on the API as these PKI efforts mature than moving to GA in 1.8.

[idvoretskyi]

@christianhuening this is a feature development progress tracking issue, not intended for solving user-facing questions.

Please, open an issue at https://github.com/kubernetes/kubernetes/issues (or check if one has been opened before).

Thanks

[luxas]

@mikedanese @kubernetes/sig-auth-feature-requests are you planning to graduate this to GA soon?

[idvoretskyi]

@roberthbailey still beta for 1.10?

[mikedanese]

This is going to GA next.

[idvoretskyi]

@mikedanese GA in 1.10, right?

[idvoretskyi]

@mikedanese thanks.

[Bradamant3]

@mikedanese it looks as though the docs could use some updating for GA:

• built: https://kubernetes.io/docs/admin/kubelet-tls-bootstrapping/
• repo: https://github.com/kubernetes/website/blob/master/docs/admin/kubelet-tls-bootstrapping.md

Could you also please update the spreadsheet for 1.10? Thanks!

[mikedanese]

This didn't make the cut.

[luxas]

This didn't make the cut.

cc @nickchase

[idvoretskyi]

@mikedanese can you please update the OP with the new issue template? It will simplify the feature tracking a lot. Thanks

@kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-node-feature-requests

[justaugustus]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-node-feature-requests

Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[idvoretskyi]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-node-feature-requests

Any plans for this in 1.11?

cc @justaugustus

[mdlinville]

@mikedanese please fill out the appropriate line item of the
1.11 feature tracking spreadsheet
and open a placeholder docs PR against the
release-1.11 branch
by 5/25/2018 (tomorrow as I write this) if new docs or docs changes are
needed and a relevant PR has not yet been opened.

[justaugustus]

@mikedanese -- We're doing one more sweep of the 1.11 Features tracking spreadsheet.
Would you mind filling in any incomplete / blank fields for this feature's line item?

[mikedanese]

No updates. Bumped to next milestone.

[justaugustus]

Thanks for the update, @mikedanese!

[justaugustus]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-cluster-lifecycle-feature-requests @kubernetes/sig-node-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y)
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

Set the following:

• Description
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[luxas]

@kubernetes/sig-auth-feature-requests can we graduate this to stable in v1.12?

[justaugustus]

@mikedanese @kubernetes/sig-auth-feature-requests @kubernetes/sig-auth-misc --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

[mikedanese]

I'm ok with this going to GA as long as the certificates API remains beta.

[liggitt]

Can a GA feature depend on a beta API? What does that mean, support-wise?

[justaugustus]

I've added this to the 1.12 sheet as stable, but let me know if we need to walk that back.
cc: @kacole2 @wadadli @robertsandoval @rajendar38

[luxas]

I'm ok with this going to GA as long as the certificates API remains beta.

Why not graduate the as-is API to GA and do a v2 in case we need additional features or whatever?

[liggitt]

We discussed this in the sig-auth meeting on 8/8, and agreed the bootstrap feature can be promoted to stable, but not the CSR API yet. That means the externally-facing portions of the bootstrap mechanism (bootstrap kubeconfig, etc) will continue to be supported.

[zparnold]

Hey there! @philips I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[philips]

Zach- I filed the feature initially but other folks took over the work. So,
I won't be able to help with the docs. Sorry.

On Mon, Aug 20, 2018 at 2:25 PM Zach Arnold [e-mail]
wrote:

Hey there! @philips https://github.com/philips I'm the wrangler for the
Docs this release. Is there any chance I could have you open up a docs PR
against the release-1.12 branch as a placeholder? That gives us more
confidence in the feature shipping in this release and gives me something
to work with when we start doing reviews/edits. Thanks! If this feature
does not require docs, could you please update the features tracking
spreadsheet to reflect it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/43#issuecomment-414468340,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACDCOhjX6sz-ErOXML9lrqSGnd4eHTUks5uSykwgaJpZM4JS7ez
.

[zparnold]

Ok, thanks for letting me know. Whom shall I ask for this data then in your
opinion?

On Mon, Aug 20, 2018 at 2:26 PM Brandon Philips [e-mail]
wrote:

Zach- I filed the feature initially but other folks took over the work. So,
I won't be able to help with the docs. Sorry.

On Mon, Aug 20, 2018 at 2:25 PM Zach Arnold [e-mail]
wrote:

Hey there! @philips https://github.com/philips I'm the wrangler for
the
Docs this release. Is there any chance I could have you open up a docs PR
against the release-1.12 branch as a placeholder? That gives us more
confidence in the feature shipping in this release and gives me something
to work with when we start doing reviews/edits. Thanks! If this feature
does not require docs, could you please update the features tracking
spreadsheet to reflect it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/kubernetes/features/issues/43#issuecomment-414468340
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AACDCOhjX6sz-ErOXML9lrqSGnd4eHTUks5uSykwgaJpZM4JS7ez

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/43#issuecomment-414468895,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE81SF57z1mjfKYZ9QPBQVJHzoItTVRYks5uSymggaJpZM4JS7ez
.

[mikedanese]

@zparnold please use the assignee of the feature issue rather than the original creator (which is immutable) of the issue for this kind of stuff. I can do this.

[zparnold]

Ok, thanks! Learning the ropes around here. :)

On Mon, Aug 20, 2018 at 2:37 PM Mike Danese [e-mail]
wrote:

@zparnold https://github.com/zparnold please use the assignee of the
feature issue rather than the original creator (which is immutable) of the
issue for this kind of stuff. I can do this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/43#issuecomment-414471622,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE81SHodJlKFUOZaBgWmnDEiCYq5HUr6ks5uSywMgaJpZM4JS7ez
.

[zparnold]

@mikedanese Will you be able to handle docs?

[justaugustus]

@mikedanese --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

[mikedanese]

GA docs PR here:

https://github.com/kubernetes/website/pull/10232

[kacole2]

It looks like this has graduated to GA so I'm going to clean up and close the issue. nice job everyone!

/close

[k8s-ci-robot]

@kacole2: Closing this issue.

In response to this:

It looks like this has graduated to GA so I'm going to clean up and close the issue. nice job everyone!

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.