Kubeadm: "kubeadm token create"d tokens don't belong to necessary groups to allow joining existing cluster.

Created on 4 Oct 2017 · 11Comments · Source: kubernetes/kubeadm

What keywords did you search in kubeadm issues before filing this one?

token create

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
debian stretch on KVM on debian stretch (baremetal)
OS (e.g. from /etc/os-release):
Debian GNU/Linux 9 (stretch)
Kernel (e.g. uname -a):
Linux serval-k8s0 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux
Others:

What happened?

After initial token obtained at end of kubeadm init expired, I wanted to add another node to the cluster.
I tried using token obtained from kubeadm token create.

The token I created doesn't belong to group "system:bootstrappers:kubeadm:default-node-token", thus,
trying to add node via kubeadm join using the token fails.
Specifically, the kubeadm join command completes w/ the following msg:

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

, while kubelet service refuses to run from following message:

error: failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:534cce" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope

What you expected to happen?

I expected either of the following:

kubeadm token create to join extra group "system:bootstrappers:kubeadm:default-node-token" by default.
Some documentation, maybe in kubeadm token --help should have noted that tokens that this command generate isn't enough to join the cluster by itself and that you need to specify kubeadm token create --groups 'system:bootstrappers:kubeadm:default-node-token' explicitly.

How to reproduce it (as minimally and precisely as possible)?

Create a new cluster using kubeadm init
Issue a new token using kubeadm token create
Try to join a new node using the token

Anything else we need to know?

google ldap: kouhei@

prioritimportant-soon

Source

nyaxt

👍3

Most helpful comment

Workaround is to pass --groups system:bootstrappers:kubeadm:default-node-token to token create.

I agree with @fabriziopandini that it is not a great experience. The implication is that kubeadm token create will, by default, give you something equivalent to the token in kubeadm init.

Options:

Update the docs for kubeadm create both on k8s.io and in the help text.
Default to the a group. It looks like this might depend on the version you are targeting but we can probably be liberal in the groups we list here.

I'm going to suggest we default to including the group. If the user doesn't want to use these for auth then they'll get an error and need to reset it. That'll probably be a rare case.

jbeda on 5 Oct 2017

👍7

All 11 comments

Seeing the same issue, using same version of kubeadm 1.8 with same git commit and k8s version

Only difference is the OS is ubuntu 16.04.2 LTS

FYI I also tried with and without the --discovery-token-ca-cert-hash flag that kubeadm init echoed on the master, made no difference (Did kubeadm reset between the 2 calls so all was clean)

pmcgrath on 4 Oct 2017

This looks to be high priority.

jbeda on 5 Oct 2017

This is "by design". You need to set the group if you want to get that behavior.

What should be fixed here though (IMO):

Documentation in the help text about this group you might wanna use
Better kubeadm join UX?

luxas on 5 Oct 2017

cc @mattmoyer can you own this if needed please? I wouldn't set this to be urgent, but we can provide a better UX

luxas on 5 Oct 2017

@jbeda @mattmoyer @luxas why not make the default behavior of kubeadm create token equal to the behaviour kubeadm init?

In fact this is already implemented for "usages" (in this case kubeadm create token assign by defaults the same usages assigned by kubeadm inits, that is "signing", "authentication"). IF there are no issue from a security prospective, IMO we can do the same for "groups" as well.

PS. This change will be one line of code change, and this will be automatically documented by the --help

fabriziopandini on 5 Oct 2017

Workaround is to pass --groups system:bootstrappers:kubeadm:default-node-token to token create.

I agree with @fabriziopandini that it is not a great experience. The implication is that kubeadm token create will, by default, give you something equivalent to the token in kubeadm init.

Options:

Update the docs for kubeadm create both on k8s.io and in the help text.
Default to the a group. It looks like this might depend on the version you are targeting but we can probably be liberal in the groups we list here.

I'm going to suggest we default to including the group. If the user doesn't want to use these for auth then they'll get an error and need to reset it. That'll probably be a rare case.

jbeda on 5 Oct 2017

👍7

@jbeda Just followed your suggestion and I can confirm my node has joined the cluster

Thanks

pmcgrath on 6 Oct 2017

👍3

Hooray! It really works after adding group into token creation.
Finally.
Somebody needs to update documentation.

vglisin on 17 Oct 2017

hiy , i am getting this after signing in with that token:

configmaps is forbidden: User "system:bootstrap:e769a8" cannot list configmaps in the namespace "default"
close
warning
persistentvolumeclaims is forbidden: User "system:bootstrap:e769a8" cannot list persistentvolumeclaims in the namespace "default"
close
warning
secrets is forbidden: User "system:bootstrap:e769a8" cannot list secrets in the namespace "default"
close
warning
services is forbidden: User "system:bootstrap:e769a8" cannot list services in the namespace "default"
close
warning
ingresses.extensions is forbidden: User "system:bootstrap:e769a8" cannot list ingresses.extensions in the namespace "default"
close
warning
daemonsets.extensions is forbidden: User "system:bootstrap:e769a8" cannot list daemonsets.extensions in the namespace "default"
close
warning
pods is forbidden: User "system:bootstrap:e769a8" cannot list pods in the namespace "default"
close
warning
events is forbidden: User "system:bootstrap:e769a8" cannot list events in the namespace "default"
close
warning
deployments.extensions is forbidden: User "system:bootstrap:e769a8" cannot list deployments.extensions in the namespace "default"
close
warning
replicasets.extensions is forbidden: User "system:bootstrap:e769a8" cannot list replicasets.extensions in the namespace "default"
close
warning
jobs.batch is forbidden: User "system:bootstrap:e769a8" cannot list jobs.batch in the namespace "default"
close
warning
replicationcontrollers is forbidden: User "system:bootstrap:e769a8" cannot list replicationcontrollers in the namespace "default"
close
warning
statefulsets.apps is forbidden: User "system:bootstrap:e769a8" cannot list statefulsets.apps in the namespace "default"
close