Kubeadm: Slave node fails to join cluster with the message "failed to run Kubelet: cannot create certificate signing request: Unauthorized"

[prateek1192]

What keywords did you search in kubeadm issues before filing this one?

Unauthorized, Kubeadm join, rejoining a node

BUG REPORT

Versions

kubeadm version (use kubeadm version):

Master: kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Slave: kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Master: kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Slave: kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

Ubuntu baremetal nodes

• OS (e.g. from /etc/os-release):

cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

uname -a
Linux deepc04.acis.ufl.edu 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

I had a two node cluster in which one was master and another slave. It was running from the last 26 days. Today i tried to remove a node using kubeadm reset and add it again and kubelet was not able to start

cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d
The binary conntrack is not installed, this can cause failures in network connection cleanup.
server.go:376] Version: v1.10.2
feature_gate.go:226] feature gates: &{{} map[]}
plugins.go:89] No cloud provider specified.
28752 server.go:233] failed to run Kubelet: cannot create certificate signing request: Unauthorized

while the join command is successful

[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "aaaaa:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://aaaaa:6443"
[discovery] Requesting info from "https:/aaaaaa:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server
[discovery] Successfully established connection with API Server "aaaa:6443"

This node has joined the cluster:

• Certificate signing request was sent to master and a response
  was received.
• The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

What you expected to happen?

I expected it to rejoin the node in the cluster

How to reproduce it (as minimally and precisely as possible)?

If we can figure out why this log came, we can know "failed to run Kubelet: cannot create certificate signing request: Unauthorized"

Add a node in the cluster

[timothysc]

/assign @detiber

[neolit123]

@prateek1192

hi,

Master: kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", 
Slave: kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", 

<snip>

Master: kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", 

<snip>

Slave: kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", 

it seems that the kubectl and kubeadm versions are swapped on the master and slave.
while this probably should work regardless and given your comment that it was running for a while:

I had a two node cluster in which one was master and another slave. It was running from the last 26 days. Today i tried to remove a node using kubeadm reset and add it again and kubelet was not able to start

but did you try matching the versions - e.g. having 1.10.0 or v1.10.2 everywhere?
just to eliminate a possible edge case of sorts.

[prateek1192]

Hello
Thanks for looking into this.
As you said, it was a version mismatch problem. I did a kubeadm reset on master and added the slave again after installing the newer versions on master and it all works now. Thanks for the help !

Regards
Prateek Arora

[prateek1192]

Hey neolit123, I hit the issue again, this time all of my nodes had the same version. I did a kubeadm reset on all the nodes and did an init again and it works again. Is there a timeout parameter after which a kubeadm join token expires because I see the join command working for some time (I dont know the exact period but it failed after a day the last time) ?

[Kailashcj]

kubeadm init creates an initial token with a 24-hour TTL. This value can be configured. please refer # https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-token/