Okd: OKD 4.4 Baremetal UPI Install: FCOS does not persist its hostname after reboot.

[cgruver]

OKD version: 4.4.0-0.okd-2020-04-07-175212
OS Version: Fedora CoreOS 31.20200406.20.0

Cluster installed as Baremetal UPI

FCOS installed with iPXE and fixed IP using kernel params:

ip=10.11.11.71::10.11.11.1:255.255.255.0:okd4-worker-1.my.domain.org:eth0:none

The cluster installed and ran just fine.

I shutdown the cluster for transport of the lab machines:

1. Cordon the workers:
   for i in 0 1 2 ; do oc adm cordon okd4-worker-${I}.my.domain.org ; done
for i in 0 1 2 ; do oc adm drain okd4-worker-${I}.my.domain.org --ignore-daemonsets --force --grace-period=60 --delete-local-data; done
2. Shutdown the workers
3. Shutdown the masters

When the master nodes were restarted. They all came up with their hostname set to localhost. They were obviously unable to join the cluster since etcd did not know them.

I was able to recover the cluster by correcting the hostname on each master and worker node:

for i in 0 1 2 ; do ssh core@okd4-master-${I}.my.domain.org "sudo hostnamectl set-hostname okd4-master-${I}.my.domain.org && sudo shutdown -r now"; done
for i in 0 1 2 ; do ssh core@okd4-worker-${I}.my.domain.org "sudo hostnamectl set-hostname okd4-worker-${I}.my.domain.org && sudo shutdown -r now"; done

[claudio-walser]

I think this is the default behavior, either you modify your ignition files before installation to set a static hostname or you set hostnames by dhcp (which is the recommended method i think)

It is described pretty good in this blog post which applies for okd as well
https://www.openshift.com/blog/openshift-4-bare-metal-install-quickstart

[cgruver]

FCOS is supposed to support kernel params for configuration as well, but it is still evolving.

ip=10.11.11.71::10.11.11.1:255.255.255.0:okd4-master-0.my.domain.org:eth0:none Should work. The IP and other network information is persisted correctly.

[cgruver]

Update from fcos team:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/419

[cgwalters]

either you modify your ignition files before installation to set a static hostname

Yes, we recommend using Ignition to write /etc/hostname. That's the clearest and most reliable way to ensure the host has the name you want. But, we also added https://github.com/coreos/ignition-dracut/pull/156 to RHCOS to make this more convenient, and the above-linked NM issue is about fixing it for FCOS too.

[cgruver]

@cgwalters Understood about the Ignition config. However, that means a custom ignition file for every host. As much as possible, I'd like the keep the ignition files clean as produced by the openshift-install command. The kernel arg method would allow me to apply the host network config much earlier in a very lightweight iPXE script.

I really appreciate the work you guys are doing. Great stuff! I'm looking forward to the day that our stodgy old OCP 3.11 clusters get replaced with OCP 4.X.

[cgruver]

The fix for this is about to drop to the testing stream for FCOS.

[vrutkovs]

Lets keep this open - this requires for the fix to:
a) land in stable stream
b) have installer updated so that fresh installs would pick this up

[cgruver]

The fix for this is now in the stable stream 31.20200505.3.0

[disposab1e]

It's not fixed in stable 31.20200505.3.0. Still have this issue here. UBI with mentioned revision.

[cgruver]

Note, that this issue was specific to setting IP parameters and hostname with kargs passed in as part of PXE boot process.

i.e. ip=10.11.11.71::10.11.11.1:255.255.255.0:okd4-master-0.my.domain.org:eth0:none

Are you seeing systems setting their hostname to localhost after rebooting from a successful OKD install?

[disposab1e]

I have this problem with setting hostname via PXE and dhcp during the first provisioning of the masters. Cause they reboot 3 times it will get lost there. So this problem does not only depend to your situation.

[dustymabe]

I have this problem with setting hostname via PXE and dhcp during the first provisioning of the masters. Cause they reboot 3 times it will get lost there. So this problem does not only depend to your situation.

sounds like you need to open a new bug with a specific reproducer.

[vrutkovs]

Latest 4.5 nightlies should use F32-based images, which have a fix for persistent IP - could you give those a try?

[vrutkovs]

This should be resolved in Beta 6, please reopen if it still occurs on latest beta/nightly

[rajinator]

I have a similar thing happening @vrutkovs

Release - 4.6.0-0.okd-2020-11-27-200126

FCOS image - 32.20201104.3.0 stable

I have a dhcp server setup to give out hostnames. Bootstrap as well as the 3 node-cluster nodes lose hostname after reboot. Bootkube service goes into restart loop on bootstrap after a while, workers kubelet error message says node "fedora" not found.

[openshift-ci-robot]

@rajinator: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rajinator]

Update, can also confirm that this issue does not happen when using the 4.5.0-0.okd-2020-10-15-235428 version installer