Kubeadm: node-name different than `hostname` not working

[jeffmhastings]

Is this a request for help?

Yes. I posted a question on slack but received no response.

What keywords did you search in kubeadm issues before filing this one?

"node-name", "Unable to register node"

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

When I run kubeadm init using the nodeName configuration option or --node-name command line flag I end up in a state where kubeadm init fails to complete.

hostname:

kubeadm command:

kubeadm init --node-name ip-x-x-x-x.ec2.internal

kubeadm output:

[root@ip-x-x-x-x ~]# kubeadm init --node-name ip-x-x-x-x.ec2.internal
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.8.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [ip-x-x-x-x.ec2.internal kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.x.x.x]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] This often takes around a minute; or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 29.501696 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node ip-x-x-x-x.ec2.internal as master by adding a label and a taint
timed out waiting for the condition

relevant kubelet logs:

Dec 01 15:07:06 ip-10-x-x-x kubelet[16229]: I1201 15:07:06.858910   16229 kubelet_node_status.go:83] Attempting to register node ip-10-x-x-x
Dec 01 15:07:06 ip-10-x-x-x kubelet[16229]: E1201 15:07:06.860760   16229 kubelet_node_status.go:107] Unable to register node "ip-x-x-x-x" with API server: nodes "ip-x-x-x-x" is forbidden: node "ip-x-x-x-x.ec2.internal" cannot modify node "ip-x-x-x-x"

Versions

kubeadm version (use kubeadm version): kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version): 1.8.4
• Cloud provider or hardware configuration: ec2 instance - not specifying cloud provider
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
• Kernel (e.g. uname -a): Linux ip-x-x-x-x 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Others:

What happened?

Kubeadm init times out. Errors in kubelet logs about not being able to register node <hostname> even though I'm setting <nodeName>

What you expected to happen?

Kubeadm init completes using the nodeName I set.

How to reproduce it (as minimally and precisely as possible)?

Spin up ec2 instance in a VPC that has a private DNS zone (i.e. not ec2.internal). hostname should be ip-x-x-x-x. Install kubeadm 1.8.4 and run kubeadm init setting node-name to the fqdn ip-x-x-x-x.ec2.internal.

Anything else we need to know?

[jeffmhastings]

I figured this one out. It seems that in addition to the --node-name option for kubeadm you also need to set --hostname-override for kubelet. I had assumed kubeadm would handle that.

At the very least this is a deficiency in the documentation.

[Matty9191]

+1 to updating the docs. Hit this today.

[samcorzine]

Could you clarify a bit about the Kubelet command? I'm facing the same issue and but it seems if I run the kubelet --hostname-override command in advance that fails as well. I feel like I'm missing something simple here.

[timothysc]

/assign @liztio

[guangxuli]

@liztio @timstoop i have the same problem, any progress?

[drpaneas]

Same issue for me. It fails at [markmaster] Will mark node ultron.suse.de as master by adding a label and a taint.

[luxas]

Implementing https://github.com/kubernetes/kubeadm/issues/846 will fix this issue. We're targeting automatic --hostname-override configuration from the --node-name flag in v1.11. Thanks all!

[liztio]

Gonna validate @luxas's solution

[luxas]

This is fixed in v1.11 thanks to https://github.com/kubernetes/kubernetes/pull/64706.
Now information in --node-name flows down to --hostname-override for the kubelet, which is respected in all cases but using an in-tree cloud provider.

[danbeaulieu]

@luxas is this fix only for the cli parameter --node-name in v.1.11? I am using v1.11.1 but using a config file and setting the nodeName value in the config file and kubelet is still using hostname

[dannymk]

@luxas is this fix only for the cli parameter --node-name in v.1.11? I am using v1.11.1 but using a config file and setting the nodeName value in the config file and kubelet is still using hostname

Check: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha3
specifically the field: "apiServerCertSANs:"

[mludvig]

Now information in --node-name flows down to --hostname-override for the kubelet, which is respected in all cases _but using an in-tree cloud provider_.

So how do I do that on AWS?

I'm getting heps of these in journalctl -xeu kubelet:

kubelet[25235]: E0624 16:42:31.041372   25235 kubelet.go:2244] node "ip-192-168-144-98.ap-southeast-2.compute.internal" not found

However it can be resolved, not sure why kubelet says it can't:

[root@kube2 ~]# host ip-192-168-144-98.ap-southeast-2.compute.internal
ip-192-168-144-98.ap-southeast-2.compute.internal has address 192.168.144.98

I'm using kubeadm 1.14.3

[neolit123]

you can try asking in the #sig-aws or #sig-cloud-provider channel on the k8s slack.
sig-aws will soon fold in sig-cloud-provider.