Kubeadm: Node names change after reboot from short hostname to fqdn

[sl4dy]

When node is deployed the common hostname is used:

#kubeadm init
#kubectl get nodes
NAME        STATUS         AGE
msl-kub03   Ready,master   30s

After reboot the node starts with FQDN:

#kubectl get nodes
NAME                          STATUS            AGE
msl-kub03                     NotReady,master   9m
msl-kub03.int.na.int.com   Ready             8m

I can workaround this by using _--api-external-dns-names_ flag for the master:

#kubeadm init --api-external-dns-names msl-kub03.int.na.int.com
#kubectl get nodes
NAME                          STATUS         AGE
msl-kub03.int.na.int.com   Ready,master   27m

But there is no way to do this for the minions.

[bjornl]

I suspect this is due to your master and/or minion changed it's hostname up on reboot.

Check that hostname set to the correct non fqdn variant.
hostnamectl | grep hostname
uname -n

You may permanently set it on a systemd managed host with "hostnamectl set-hostname msl-kub03" or add it to the kernel system parameter file with "echo kernel.hostname=msl-kub03 >> /etc/sysctl.conf" and reboot.

One should also make sure that /etc/hosts entry is correctly setup. These commands should return correct info:
hostname -s (short name)
hostname -d (domain name)
hostname -f (fqdn name)
hostname -i (ip address)

[sl4dy]

Unfortunately that's not the case. I have rebooted the node multiple times prior to Kubernetes installation via kubeadm. The hostname metadata is persistent:

17/02/06 14:17:30 rack-na/fake (devel-minimal)
[root@msl-kub01:/home/msl] hostnamectl | grep hostname
   Static hostname: msl-kub01.int.na.int.com
Transient hostname: msl-kub01

17/02/06 14:18:54 rack-na/fake (devel-minimal)
[root@msl-kub01:/home/msl] hostname -s
msl-kub01

17/02/06 14:19:19 rack-na/fake (devel-minimal)
[root@msl-kub01:/home/msl] hostname -d
int.na.int.com

17/02/06 14:19:19 rack-na/fake (devel-minimal)
[root@msl-kub01:/home/msl] hostname -f
msl-kub01.int.na.int.com

17/02/06 14:19:19 rack-na/fake (devel-minimal)
[root@msl-kub01:/home/msl] hostname -i 
172.30.40.117

[bjornl]

Your static hostname is a FQDN, which it should not be if you want to use short names.

On my master I have:

hostnamectl | grep Static
Static hostname: kube1

uname -n
kube1

[sl4dy]

Well static hostname should be FQDN regarding the official RedHat documentation.

However the nodes deployed by kubeadm behave like this:

Scenario 1:

• static hostname is FQDN
• node is deployed and displayed with short hostname (why - if it looks for static hostname)
• after reboot node is displayed with FQDN

Scenario 2:

• static hostname is short hostname
• node is deployed and displayed with short hostname
• after reboot node is displayed with short hostname
• however this goes against RedHat recommendation (and we depend on FQDN in static hostname internally)

[bjornl]

Traditionally UNIX have always used short names in lower case as a machines hostname. UNIX existed before DNS was invented. I would say RedHat's recommendation is not correct, a UNIX server may not have a domain or be connected to DNS resolve at all. In the end it boils down to a matter of personal taste if one want to use short names or FQDN as the hostname for their servers. This is not a deficiency or bug in kubeadm.

[sl4dy]

I am not disputing that. However it is hard requirement for us to have Kubernetes nodes available & deployed with FQDNs. The same hostnames will be used in different clusters / domains.

So as suitable workaround I add --hostname-override <hostname_or_ip> to kubelet systemd /etc/systemd/system/kubelet.service.d/10-kubeadm.conf configuration:

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS --hostname-override=FQDN

[luxas]

This is not a kubeadm issue, it's the (correct in my understanding) behaviour of the kubelet.
Closing, reopen if you disagree...