Installer: etcd member does not recover after killing a master

[alexouzounis]

Version

$ openshift-install version
openshift-install-darwin-amd64 v0.14.0

Platform (aws|libvirt|openstack):

aws

What happened?

Created cluster, everything worked as expected.
We then tested its resilience by zapping ec2 instance master nodes, just one. The etcd member that was scheduled on that new master node failed to run -

15:29 $ KUBECONFIG=auth/kubeconfig kubectl logs etcd-member-ip-10-0-137-234.eu-west-1.compute.internal -c discovery  --namespace kube-system
I0320 15:33:51.280793       1 run.go:47] Version: 4.0.15-1-dirty
I0320 15:33:51.283212       1 run.go:57] ip addr is 10.0.137.234
I0320 15:33:51.286821       1 run.go:137] checking against etcd-2.poc.openshift.zeus.builtwith.solar.
I0320 15:33:51.292744       1 run.go:137] checking against etcd-1.poc.openshift.zeus.builtwith.solar.
I0320 15:33:51.297963       1 run.go:137] checking against etcd-0.poc.openshift.zeus.builtwith.solar.
E0320 15:33:51.305083       1 run.go:63] error looking up self: could not find self
I0320 15:34:51.317691       1 run.go:137] checking against etcd-1.poc.openshift.zeus.builtwith.solar.
I0320 15:34:51.319700       1 run.go:137] checking against etcd-2.poc.openshift.zeus.builtwith.solar.
I0320 15:34:51.389706       1 run.go:137] checking against etcd-0.poc.openshift.zeus.builtwith.solar.
E0320 15:34:51.393983       1 run.go:63] error looking up self: could not find self
I0320 15:35:51.307119       1 run.go:137] checking against etcd-0.poc.openshift.zeus.builtwith.solar.
I0320 15:35:51.308793       1 run.go:137] checking against etcd-1.poc.openshift.zeus.builtwith.solar.
I0320 15:35:51.310656       1 run.go:137] checking against etcd-2.poc.openshift.zeus.builtwith.solar.
E0320 15:35:51.312222       1 run.go:63] error looking up self: could not find self

We realised that Route53 which should have been updated with the new IP of the master node was still the old one so we poked it manually to the new IP. We then noticed the second init container was also having issues with certificate signing

15:54 $ KUBECONFIG=auth/kubeconfig kubectl logs etcd-member-ip-10-0-137-234.eu-west-1.compute.internal -c certs  --namespace kube-system
Error: error requesting certificate: error sending CSR to signer: certificatesigningrequests.certificates.k8s.io "system:etcd-server:etcd-0.poc.openshift.zeus.builtwith.solar" already exists
Usage:
  kube-client-agent request --FLAGS [flags]

Flags:
      --assetsdir string    Directory location for the agent where it stores signed certs
      --commonname string   Common name for the certificate being requested
      --dnsnames string     Comma separated DNS names of the node to be provided for the X509 certificate
  -h, --help                help for request
      --ipaddrs string      Comma separated IP addresses of the node to be provided for the X509 certificate
      --kubeconfig string   Path to the kubeconfig file to connect to apiserver. If "", InClusterConfig is used which uses the service account kubernetes gives to pods.
      --orgname string      CA private key file for signer

ERROR: logging before flag.Parse: F0320 15:51:01.399070       8 main.go:18] Error executing kube-client-agent: error requesting certificate: error sending CSR to signer: certificatesigningrequests.certificates.k8s.io "system:etcd-server:etcd-0.poc.openshift.zeus.builtwith.solar" already exists

we had a look and noticed the certificate was on a pending state. We then issued an oc certificate approve hoping that would move us forward. Unfortunately that was the end of it.

What you expected to happen?

New master node to join the cluster and etcd to stat successfully, going back to a 3 node setup.

How to reproduce it (as minimally and precisely as possible)?

• Create new cluster
• Delete a master, either from EC2 console or by deleting the machine CRD

[wking]

We then tested its resilience by zapping ec2 instance master nodes, just one.

This does not work now (as you found out), and is unlikely to work until we have an in-cluster operator capable of attaching machine-API-created masters to the existing etcd cluster. We want to do that eventually, but I'm not sure what the timeline is.

[DanyC97]

@wking you reckon the timeline will be past 4.0 ?

[alexouzounis]

How is OpenShift truly HA if we cannot lose a single master ?

[wking]

How is OpenShift truly HA if we cannot lose a single master ?

You can lose a master, you just currently need manual intervention to recover cpletely once the replacement comes up. In the meantime, the cluster is degraded, but still alive and functional.

And I doubt this gets fixed in 4.0. But it's really not an installer issue, so I'm not the person to ask for timelines. I'm fine with this issue as a tracker, though.

[alexouzounis]

Thanks @wking for your help - I really appreciate it. Can you point me to the person I should talk to such that we can get a list of actions we need to take to put the new master in full operation and possibly submit a feature request to automate those afterwards?

[maci0]

Also tried adding 2 additional masters as an exercise. adding the machines with proper profile etc works and they get etcd static pods trying to boot but never work because the discovery wasn't successful.
then tried adding them using etcdctl member add but that didn't help much either.

[maci0]

ok, made some progress.. when adding new master nodes and they're supposed to run etcd, the DNS entries for etcd discovery need to be updated in route53.

the next hurdle is then that the etcd csrs dont get approved automatically. did a workaround to get them approved before the cert init container gives up.
so the csr is signed, the etcd pod goes up. can not however add it to the cluster as the other members dont trust his certificate. my guess is that the initial etcd certs were issued by the boostrap k8s with a different ca

[maci0]

error looking up self: could not find self specifically is because of the missing/wrong DNS A record.
If adding more nodes the SRV record needs some love too

[wking]

I'm fine with this issue as a tracker, though.

Ah, rhbz#1679215 is the master-team tracker. That's a better fit (they can actually fix this), so closing this issue in favor of that one.

[DanyC97]

shame your Jira is behind the _closed doors_ ..in this way nobody outside RH can see when is going to be tackled (in the old days you had Trello and that was useful when it was kept up to date)

[thoraxe]

We are in the process of figuring out how to open JIRA. That's a temporary issue.

[DanyC97]

We are in the process of figuring out how to open JIRA. That's a temporary issue.

thanks a bunch @thoraxe , great news ! 💯