Kubespray: How do I grow my cluster from 1 etcd node to 3 etcd nodes?

[juliohm1978]

Kubespray version: v.2.8.4
Ansible: 2.7.9
Host OS: Ubuntu 18.04 LTS

Inventory

[all]
k8s-master01-lab etcd_member_name=etcd1
k8s-master01-lab-20190425 etcd_member_name=etcd2
k8s-etcd01-lab-20190425 etcd_member_name=etcd01-20190425
k8s-node01-lab
k8s-node02-lab

[kube-master]
k8s-master01-lab
k8s-master01-lab-20190425

[etcd]
k8s-master01-lab
k8s-master01-lab-20190425
k8s-etcd01-lab-20190425

[kube-node]
k8s-node01-lab
k8s-node02-lab

[k8s-cluster:children]
kube-master
kube-node

I'm trying to increase the number of etcd nodes in my cluster from 1 to 3, but Kuebspray fails while adding the second node, mid execution:

TASK [etcd : Join Member | Add member to etcd cluster] *************************
Tuesday 30 April 2019  14:01:18 -0300 (0:00:00.371)       0:03:46.049 ********* 
FAILED - RETRYING: Join Member | Add member to etcd cluster (4 retries left).
FAILED - RETRYING: Join Member | Add member to etcd cluster (3 retries left).
FAILED - RETRYING: Join Member | Add member to etcd cluster (2 retries left).
FAILED - RETRYING: Join Member | Add member to etcd cluster (1 retries left).
fatal: [k8s-etcd01-lab-20190425]: FAILED! => {"attempts": 4, "changed": true, "cmd": "/usr/local/bin/etcdctl --endpoints=https://172.25.134.100:2379,https://172.25.134.127:2379,https://172.25.134.94:2379 member add etcd01-20190425 https://172.25.134.94:2380", "delta": "0:00:00.027881", "end": "2019-04-30 14:01:35.151129", "msg": "non-zero return code", "rc": 1, "start": "2019-04-30 14:01:35.123248", "stderr": "client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member https://172.25.134.100:2379 has no leader", "stderr_lines": ["client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member https://172.25.134.100:2379 has no leader"], "stdout": "", "stdout_lines": []}

The cluster was broken after this execution and I had to reset.yml to get it back up again.

On the same note, Kubespray also forbids me from running cluster.yml with only two nodes.

TASK [kubernetes/preinstall : Stop if even number of etcd hosts] ***************
Tuesday 30 April 2019  14:26:38 -0300 (0:00:00.229)       0:00:16.511 ********* 
fatal: [k8s-master01-lab]: FAILED! => {
    "assertion": "groups.etcd|length is not divisibleby 2", 
    "changed": false, 
    "evaluated_to": false, 
    "msg": "Assertion failed"
}
fatal: [k8s-master01-lab-20190425]: FAILED! => {
    "assertion": "groups.etcd|length is not divisibleby 2", 
    "changed": false, 
    "evaluated_to": false, 
    "msg": "Assertion failed"
}
fatal: [k8s-node01-lab]: FAILED! => {
    "assertion": "groups.etcd|length is not divisibleby 2", 
    "changed": false, 
    "evaluated_to": false, 
    "msg": "Assertion failed"
}
fatal: [k8s-node02-lab]: FAILED! => {
    "assertion": "groups.etcd|length is not divisibleby 2", 
    "changed": false, 
    "evaluated_to": false, 
    "msg": "Assertion failed"
}
fatal: [k8s-etcd01-lab-20190425]: FAILED! => {
    "assertion": "groups.etcd|length is not divisibleby 2", 
    "changed": false, 
    "evaluated_to": false, 
    "msg": "Assertion failed"
}

So, my question is... how exactly am I supposed to grow the etcd cluster using Kubespray?

[fritchie]

I haven't tried it yet but this may work:

https://github.com/kubernetes-sigs/kubespray/blob/master/docs/recover-control-plane.md

[juliohm1978]

I was able to include one extra etcd node at a time by running cluster.yml with extra args.

-e ignore_assert_errors=yes

The number of tasks affected by ignore_assert_errors is quite small. If anyone wants to do it this way, it's worth revisiting its occurrences to make sure nothing important would be missing.

This allows Kubespray to bypass the even number verification (0020-verify-settings.yml).