Kubeadm: Use localhost for API server health checks?

[jamiehannaford]

When kubeadm waits for the master, it calls the /healthz endpoint to ensure it's running:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/apiclient/wait.go#L77

However this is problematic for HA setups where the advertiseAddress is the Load Balancer IPv4 and other masters in the LB pool might not having running apiservers listening on 6443 yet (imagine we're running kubeadm init for the first time on master0). This will cause the LB to distribute traffic to dead endpoints and kubeadm will hang indefinitely.

Is it possible to use localhost instead of the given advertise address?

/cc @luxas @timothysc

[luxas]

Let's enqueue this for the next kubeadm meeting?

[timothysc]

TODO: figure out if the downward API change fixes this issue.

[kris-nova]

I'm happy to take this one too

Here is my thoughts on approaching it if somebody wants to sanity check me:

Validate this is still a concern

Looks like we need to see if this is still even applicable, if it is we can code in a change

Try localhost first

Can we have the program try localhost first to find the API server, and if that fails then fall back on advertiseAddress? I think having failover logic here would make sense. Also not sure if localhost first is the right approach. Thoughts everyone?

[jamiehannaford]

Not sure this is relevant anymore because the HA guide now uses private IPv4s rather than the LB IPv4 for the advertise address :)

[seh]

In the the "Set up master Load Balancer" section of the "Creating HA clusters with kubeadm" document, we find the following advice:

You will need to ensure that the load balancer routes to just master0 on port 6443. This is because kubeadm will perform health checks using the load balancer IP. Since master0 is set up individually first, the other masters will not have running apiservers, which will result in kubeadm hanging indefinitely.

Is that still true, in light of kubernetes/kubernetes#59371 having merged and this issue being closed? It sounds like _kubeadm_ will no longer consult the load balancer during _kubeadm init_ on master0.

[seh]

Reading kubernetes/kubernetes#59371 again today, I'm less confident that it addresses this need to treat the first master machine specially in the load balancer target configuration.

@andrewrynhard or @jamiehannaford, do you know if the special configuration is still necessary? Is this question worth filing a fresh documentation-focused issue?

[andrewrynhard]

@seh https://github.com/kubernetes/kubernetes/pull/59371 only addresses the case when SelfHosting is set to true. In that case, and before https://github.com/kubernetes/kubernetes/pull/59371 was merged, the automatically generated DaemonSet for the API server had the master0 host's IP set for the readiness and liveness probes. This would mean that if you had master1, and master2, and then master0 went down, then it would effectively take out master1, and master2 because the IPs on the probes for them would point to master0. It sounds like you are not using the SelfHosting feature so https://github.com/kubernetes/kubernetes/pull/59371 will have no effect.

[seh]

Another question about this statement:

You will need to ensure that the load balancer routes to just master0 on port 6443. This is because kubeadm will perform health checks using the load balancer IP.

When setting up the masters, the only place I can see where we're instructed to tell _kubeadm_ about the load balancer is in the _config.yaml_ file passed to _kubeadm init_—specifically, in the apiServerCertSANs sequence.

Where does _kubeadm_ use that address to issue health checks? It's not clear to me where any of the master components would be trying to talk to the load balancer at this point.