Rancher: Feature Request - Kubernetes HA through etcd-cluster

[thomaspeitz]

Rancher Version:
1.0.1
Docker Version:
1.10.3
OS:
ubuntu
Steps to Reproduce:
Have three Nodes in a kubernetes ENV.
Terminate the Instance on which the etcd container is running.

Results:
Kubernetes is down + All Secrets/Pods/RC Configs are lost after the cluster
repairs itself. Which works like a charm. Thank you :)

Expected:
etcd is HA, has data on all nodes, cluster repairs it self and does not lose all the data

Temporarily Workaround Idea:
If we would allow to configure the etcd address in the kubernetes stack I could install . This would allow me to have etcd HA on different nodes. And every kubernetes Node could go offline without impacting the cluster.

I ask myself if i can click on the kubernetes stack and configure external etcd cluster through the upgrade button, and if the settings i configured will be exchanged after next rancher upgrade?

Implementation Idea:
Start etcd container on every server and spin up a cluster. Kubernetes Container connects to an Array of etcd (like in coreos implementation - https://coreos.com/kubernetes/docs/latest/deploy-master.html - See configuration parameter --> - --etcd-servers=${ETCD_ENDPOINTS}). I would try to implement it through manually changes in the stack by myself but how can i "save" it that it won't change after next upgrade?

Just found the catalog ha etcd from rancher. Will try to implement it into the kubernetes stack.

Greetings Thomas 👍

[thomaspeitz]

After a lot of research. Here are some Ideas to implement it.
First one:

• Create an etcd cluster through docker container on every rancher host. This is really painful because etcd is not created to be managed in dynamic environments. As soon as a Node goes down or a new one is added you need manual interaction (source: https://coreos.com/etcd/docs/latest/runtime-reconf-design.html) - This could be managed through a Container which is running on one Node of the Cluster. This Container would run together with the main kubernetes container (kube-apiserver). This container could query the rancher metadata service and collect information about the nodes. On every Node an etcd container would be started with some kind of api interface which allows to the start etcd service with the right parameters (source: https://coreos.com/etcd/docs/latest/clustering.html - These parameters must include the node ip addresses of the old servers). Furthermore the new server has to be added through etcdctl (etcdctl member add...) first before even started through the api interface. I hope this shows how stressful it is to maintain it and keep track of edge cases (like lost quorum or anything like that)

Second one:

• Create the etcd container on one node and ensure that if the node goes down the data volume will be mounted on a different host and the etcd is started again (together with the kube-apiserver). This would force to create some glusterfs between all nodes. I think this will be painful as well.

Third one:

• Backup the etcd every 5 minutes and transfer the data to the other nodes. If the etcd host goes down, the new container would ensure that it only creates a new etcd cluster if there is no backup. This would allow to easy implement it from my view but is not really "HA" as it loses data.

Third one - From my poin of view the "best" one until etcd is easy to deploy on dynamic environments:

• Allow to configure the etcd node addresses which kube-apiserver accesses.
• Publish a good guide how to install a three Node etcd cluster and monitor it.

Trying the third one (modifying etcd address in kube api server docker container)

24.4.2016 19:26:57F0424 17:26:57.688936 1 server.go:211] Cloud provider could not be initialized: could not init cloud provider "rancher": Could not create rancher client: &url.Error{Op:"Get", URL:"", Err:(*http.badStringError)(0xc2081c5740)}


Seems that this was cause through a loss of labels (vimdiff left side old stack, right stack after upgrading two times. One time with etcd on a seperate host, and then a second upgrade to roll it back)

Referenced Bug: https://github.com/rancher/rancher/issues/4476

After setting the labels manually on upgrade (beside changing the etcd addresses to a selfmanaged etcd cluster) the cluster is working again :) Tomorrow i will test what happens if the node running the kubernetes api container goes down. From my point of view it should be able to reconnect to the etcd cluster and no services go down afterwards.

[LLParse]

Pull request

[soumyalj]

Verified on master with three hosts. Powering off one of the hosts brings up etcd on another host successfully. Kubernetes pods, rc, services can be created successfully too.