K3os: Installer: Easy Multi-HA join config

[PhilsLab]

Is your feature request related to a problem? Please describe.
When installing, i miss the option to join as a master node. Joining to an existing k3os cluster as another master node would speed up the installation a lot, since I don't want to add manual cloud-config for that.

Describe the solution you'd like
Add an installer question something like:

Join existing cluster in HA mode server [y/N]: y

Also, please add docs on how to manually join as a HA member. I've tried it with the suggested steps from the k3s docs, and it does not seem to work. kubectl reports a "no connection" error locally.

Describe alternatives you've considered
Manually adding the required parameters to cloud-config seems a bit tedious when building up a cluster, though it is one of the core settings for my use case when installing k3os.

[PhilippMolitor]

Any news on this? It seems to get ignored.

[joalmjoalm]

Any advice how to configure 3 node k3os cluster w three servers w embedded etcd? Tried adding server to config.yaml and succeeded getting two servers up and running (1x k3s + 1x k3os) but not the third one (k3os) even if the all three registered to the first cluster-init master (k3s).

[msw10100]

Bump

[cdeadspine]

This needs more documentation please.

I had one master running ok.

Is it ok to start up deployments already? Or should all 3 servers be "clustered" together before anything is deployed?
Is the original "--cluster-init" good enough forever? 
Or does this have to be completed again later just before servers 2 and 3 added?

I am attempting to add 2 more servers in this configuration.

Server 2 installation started with the install asking me for the "token" only and not asking for the other master.

When installing server 2 of 3, should you enter the token? 
Or not bother until later? 
Such as when the k3os versions are the same / when you can enter the other master url at the same time?

After it booted it was an older version, so I used SSH to log in to the node and do necessary labels for automatic updating

kubectl -n k3os-system patch plan k3os-latest -p '{"spec":{"version":"v0.19.5-rc.1"}}' --type=merge
kubectl label node --overwrite --all "plan.upgrade.cattle.io/k3os-latest"="enabled"
kubectl label nodes --overwrite --all "k3os.io/upgrade"="enabled"            
kubectl get events --watch

Ok now they are the same version, so from the instructions at
https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/
I log into the second server again, to attempt to join the cluster.

sudo K3S_TOKEN=token k3s server --server https://192.168.10.43:6443
"cluster bootstrap already complete" 
had to use ip address instead of hostname (mDNS not working maybe? or is icmp/ping just not a default thing?)

"Managed etcd cluster not yet initialized"
Is this normal? Does this mean it doesn't initialize etcd until the third server is added? Or is this related to the "address already in use" below?

"Cluster bootstrap already complete" 
This is probably because during install I put in a token already.... Do I need to use --cluster-reset somehow?

"starting kubernetes: preparing server: init cluster datastore and https: listen tcp :6443: bind: address already in use"
This seems like a real problem.. Like perhaps I need to kill off k3s before it can re-initialize / re-join the cluster?
The internet says lots about "k3s-killall.sh" but I can't find this anywhere in a k3os box

So what now? Do I go ahead and try the same thing on server 3, or is there a way to verify that server 2 is part of the cluster?

Where does the "there must be 3+ and odd servers" come from? Is this etcd requiring this? What do proper intermediary steps look like? What exactly happens when one of those 3 servers fails. If this is etcd, then I think the docs should link to what further reading is required.

[cdeadspine]

Ok so for some reason in my brain "--cluster-init" seemed like a one time thing I had to run, but it appears it should be in the config.yaml for the first server only

k3os:
  k3s_args:
  - server
  - "--disable=servicelb"
  - "--cluster-init"
  token: token
ssh_authorized_keys:
- ssh-ed25519 token

This is apparently smart enough to not clobber things on multiple runs, and I had no trouble just adding this to an already running (not yet clustered) computer. (was at one point an older version of k3os, automatically updated via kubernetes:labels)
This was originally installed with only ssh_authorized_keys, then reconfiguring on the running system worked ok

ssh into device
cp /k3os/system/config.yaml /var/lib/rancher/k3os/config.yaml
make changes as shown above with --cluster-init

At some point this testing did remove all deployment resources, I assume the first time --cluster-init ran?

The second computer however, didn't seem to work being an older version that was updated... or something. I had to re-create an entirely different installation iso for the second+ cluster members

k3s_args:
- server
- "--disable=servicelb"              
token: token
server_url: https://192.168.10.43:6443
ssh_authorized_keys:
- ssh-ed25519 token

This is slightly disturbing because something I did resulted in "x509: certificate signed by unknown authority by etcd" and I didnt know how to clear existing etcd certificates without reinstallation

The second disturbing thing is that now I have to hardcode this ip into an iso image. The first server auto-generated some hostname as k3os- and my router gave it a dhcp address of 192.168.10.43

I don't think mDNS is a default part of k3os? so I cant use the hostname, and I guess I can tell the router to keep this ip address as static... but this doesn't feel very "cluster like"

[jabbas]

@cdeadspine worked flawlessly for me. First node with --cluster-init, the rest without on v0.19.5-rc.1