Enhancements: Integrate Cluster Bootstrap/Discovery with Kubenetes Core

[jbeda]

Feature Description

• One-line feature description (can be used as a release note): Add a new authentication and verification mechanism called Bootstrap Tokens to use to easily add new members to a cluster
• Primary contact (assignee): @luxas @jbeda
• Responsible SIGs: sig/cluster-lifecycle, sig/auth
• Design proposal link (community repo): https://github.com/kubernetes/community/pull/628
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @mikedanese @liggitt @ericchiang
• Approver (likely from SIG/area to which feature belongs): @mikedanese @liggitt @ericchiang
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y): v1.6
  
  
  • Beta release target (x.y): v1.8
  
  
  • Stable release target (x.y): TODO
  
  

[idvoretskyi]

@jbeda I noticed that the feature has been opened after the agreed submission deadline (Oct 10). Is there any reason why this feature has to be added to 1.5 milestone?

[jbeda]

@idvoretskyi I screwed up and I apologize. We were tracking all of the work in the the lifecycle SIG and met on Tuesday morning. Missed it by one day. Hopefully this isn't a blocker?

[mikedanese]

meta: I don't understand the 1.5 milestone deadline. If a feature get's dreamed up after the deadline, alpha implemented and merged before we cut 1.5 is it still a feature? I see the purpose of this repo as providing visibility to progress on large features. I don't see any purpose to a submission deadline. There's also no documentation of a deadline in this repo (when or why).

[idvoretskyi]

@mikedanese the purpose of the repo - is providing visibility and tracking on the product features, you are right. At the same time, during the release development process, we'd like to see the actual picture of the features that are going to be added to the product.
Feature submission deadline - is not the deadline for developing the feature; it's a deadline for declaring your desire to develop some functionality that you'd like to see in the released product. We are expecting from all the contributors to follow the release roadmap carefully to avoid unnecessary planning chaos.
PS. Thank you for mentioning the reflecting the deadline in the documentation - I've submitted a PR to add it to the release roadmap https://github.com/kubernetes/features/pull/133.

[idvoretskyi]

@jbeda as we have already agreed in the mailing thread [0], this is not a blocker.

1. https://groups.google.com/d/msg/kubernetes-sig-cluster-lifecycle/AyNEsX2lJnU/Hv0SYJFJEgAJ

[jbeda]

cc @kubernetes/sig-cluster-lifecycle

[luxas]

Hmm, this didn't make v1.5, did it?

[idvoretskyi]

@jbeda @luxas any final agreement on this?

[jbeda]

Pushing to 1.6

[idvoretskyi]

@jbeda can you confirm that this feature targets alpha for 1.6?

[apsinha]

Should we add the alpha-in -1.6 label to this?

[idvoretskyi]

@apsinha I think so, updated.

[luxas]

I'm not sure whether we'll call it alpha or beta, it might be beta as well, if we count the kube-discovery method currently used in kubeadm as the alpha implementation

cc @jbeda @mikedanese

[idvoretskyi]

@luxas marked as alpha following the previous discussions. Let's discuss the appropriate feature stage if there are other suggestions.

[roberthbailey]

I believe @mikedanese labeled the api objects as beta for 1.6.

[luxas]

@jbeda Are we planning to automatically enable the BootstrapSigner and the TokenCleaner in v1.6?

[jbeda]

@luxas I don't think that we can as this is alpha. I'm happy turning this on by default for kubeadm clusters with the appropriate command line flags. After we show it works well with kubeadm we can turn them on for everyone.

[luxas]

@jbeda I'm ok with that, and that's what I supposed as well. Anyway, didn't we consider the kube-discovery version of this work the alpha part and this version beta?

[idvoretskyi]

@jbeda please, provide us with the release notes and documentation PR (or links) at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

[jbeda]

@lukemarsden Is pulling together release notes. I'm working on doc PR now.

[luxas]

We want to graduate Bootstrap Tokens to beta in v1.7.

Our TODO list for v1.7 is:

• Create full-blown e2e tests for the BootstrapSigner and TokenCleaner controllers (assignee: @jbeda)
• Add the option for a token to authenticate as a certain group with the system:bootstrappers: prefix
• Lock-down the kubeadm CSR autoapproving to only auto-approve CSRs with the system:bootstrappers:tls-bootstrap (exact name TBD) group. The kubeadm init token will have this group attached to it, but users that create bootstrap tokens for other purposes must explicitely set the token in that group in order to be able to use auto-approving.
• Move the generic code for dealing with Bootstrap Tokens to client-go: https://github.com/kubernetes/client-go/issues/114
• Enable the BootstrapSigner and TokenCleaner controllers by default

cc @jbeda @mikedanese @liggitt @deads2k ^

[jbeda]

Based on some conversations with some security folks, it looks like we might want to enhance the way we do tokens to be asymmetric. Right now if a token leaks, it allows an attacker to not just join the cluster but also to MITM the bootstrap and MITM the API server. Scenario-wise, this is more of a problem as the tokens last a longer time.

Some context: https://kubernetes.slack.com/archives/C0EN96KUY/p1497967620498941

This will require a new design doc for the new scheme along with implementation in 1.8. It'll require updates to the bootstrap signer, cleaner and authorizer along with updates to kubeadm. Corresponding docs will need to be updated too.

[jbeda]

/sig auth

[luxas]

This is targeted at v1.8, @jbeda, @mattmoyer, myself and some people from sig-auth is gonna work on getting this to beta.

[luxas]

In flight for v1.8 to get this feature to beta:

• [x] Make it possible to authenticate a Bootstrap Token in different groups: https://github.com/kubernetes/kubernetes/issues/49306 (assignee: @mattmoyer)
• [x] Adding one more verification method for kubeadm join to make the flow asymmetric: https://github.com/kubernetes/kubernetes/pull/49520 (related gdoc proposal) (assignee: @mattmoyer)
• [x] Setting a 24h TTL on the default BT from kubeadm init https://github.com/kubernetes/kubeadm/issues/343 (assignee: @mattmoyer)
• [x] Write e2e tests for the BootstrapSigner, TokenCleaner and BootstrapTokenAuthenticator https://github.com/kubernetes/kubernetes/issues/46634 (assignee: @wanghaoran1988)
• [x] Rename the API server flag --experimental-bootstrap-token-auth to --enable-bootstrap-token-auth https://github.com/kubernetes/kubernetes/issues/50613 (assignee: @mattmoyer)
• [x] Making kubeadm use the new --enable-bootstrap-token-auth flag where supported https://github.com/kubernetes/kubeadm/issues/414 (assignee: @luxas)

@jbeda @ericchiang @mattmoyer Please fill in here if there was something I forgot.

[liggitt]

Enable the BootstrapSigner, TokenCleaner and BootstrapTokenAuthenticator by default

I would not expect the BootstrapTokenAuthenticator enabled by default... new authn/authz modes are opt-in

[luxas]

@liggitt I'm ok with that, just thought that the procedure was to enable authn/authz modules by default as well as the controllers.
Reflected the statement above, thanks!

[idvoretskyi]

@jbeda @luxas please, update the feature description with the new template https://github.com/kubernetes/features/blob/master/ISSUE_TEMPLATE.md

[luxas]

@idvoretskyi Updated with the new release note template
@jbeda PTAL and see if that looks good to you as well

[luxas]

All the checkboxes above are checked :sunglasses:, ready for beta!
Thanks @mattmoyer @wanghaoran1988!

[mattmoyer]

What do we need to get bootstrap tokens to stable/GA in 1.9?

[jbeda]

I'm ready to declare victory now. Are there any outstanding issues?

On Thu, Oct 12, 2017 at 8:53 AM Matt Moyer [e-mail] wrote:

What do we need to get bootstrap tokens to stable/GA in 1.9?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/130#issuecomment-336181447,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACRvvxgaC1o13WB2Qmxywo2CLyQXr-Aks5srjYNgaJpZM4KT7wK
.

[luxas]

@jbeda Rate-limiting the public configmap :smile:

[jbeda]

Things we can do to further lock this down:

1. Rate limit anonymous access (at all to k8s) per calling IP address
2. Limit anonymous access to RFC1918 networks by default.
3. Limit usage of bootstrap token to RFC1918 networks by default.

I thought I saw someplace that there was a global rate limit for anonymous access but I can't find it now.

2 and 3 sound good but often times folks will use a LB that will rewrite the source that'll make this less than ideal.

[idvoretskyi]

@luxas @jbeda stable for 1.9, right?

[luxas]

The official label is beta in v1.9 (but it's kinda stable :smile:)

[zacharysarah]

@jbeda :wave: Please indicate in the 1.9 feature tracking board
whether this feature needs documentation. If yes, please open a PR and add a link to the tracking spreadsheet. Thanks in advance!

[luxas]

@jbeda @mattmoyer I guess we could remove the v1.9 milestone here as nothing changed between v1.8 and v1.9 for this feature. We'll target GA in a future release.

@zacharysarah no new docs needed

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[justaugustus]

@luxas @jbeda
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[justaugustus]

This feature current has no milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y)
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

Set the following:

• Description
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that Features Freeze is tomorrow, July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

P.S. This was sent via automation

[kacole2]

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

• Docs (open placeholder PRs): 11/8
• Code Slush: 11/9
• Code Freeze Begins: 11/15
• Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[roberthbailey]

/remove-lifecycle stale
/lifecycle stale

[roberthbailey]

/remove-lifecycle stale
/lifecycle frozen

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[claurence]

@luxas Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[kacole2]

Hello @luxas, I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP to be included. If this feature has been abandoned please let us know.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[kacole2]

Hi @luxas @roberthbailey , I'm the 1.16 Enhancement Lead/Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[jeremyrickard]

Hey there @roberthbailey @luxas -- 1.17 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll also need to convert the design proposal into a KEP. To be accepted in the release, all enhancements MUST have a KEP, the KEP MUST be merged, in an implementable state, and have both graduation criteria/test plan.

Thanks!

[roberthbailey]

Please check with @timothysc / @justinsb.

[jeremyrickard]

Thanks @roberthbailey! @timothysc @justinsb ping on this? Enhancement freeze will be Tuesday, October 15, EOD PST. Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/close

after the discussion in the kubeadm office hours with @timothysc we decided to close this ticket as out of date.

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

/close

after the discussion in the kubeadm office hours with @timothysc we decided to close this ticket as out of date.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.