Charts: Single node etcd cluster cannot recovery from restart

[canhnt]

Reproduction step:

• Deploy an etcd cluster with single node

helm install --name foo bitnami/etcd

• Go to pod shell, write some key via etcdctl (v3)
• Delete the foo-etcd-0 pod.
• New foo-etcd-0 pod is created but cannot start successfully. After several restarting time, the pod goes to CrashLoopBackOff state.
• pod crashing log:

==> The ID of the host is 0
==> Creating data dir...
==> There is no data at all. Creating new cluster
==> Configuring RBAC authentication!
stty: 'standard input': Inappropriate ioctl for device
stty: 'standard input': Inappropriate ioctl for device
New password:
User root created
auth: already enabled

[tompizmor]

Thanks for reporting the issue. I could reproduce this issue as well. We will work on a fix and update this issue once it is fixed.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[canhnt]

/remove

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[canhnt]

/remove

[qiuyuzhou]

I have same issue. Is there a workaround for fix it?

[juan131]

Hi everyone,

I just merged a PR from @tompizmor which addresses this issue. Could you please confirm the issue disappeared using the latest version of the chart?

[canhnt]

I cannot install etcd chart v1.4.1. After installation, the etcd-0 pod failed to start:

==> The ID of the host is 0
==> Data exists. Re-joining etcd member
cat: /bitnami/etcd/data/member_id: No such file or directory

If I scale to 3 replicas, etcd-1 and etcd-2 pods start successfully. However they do not join the single cluster but cause the split-brain (i.e. 2 single-node clusters).

[tompizmor]

mmm, I think this was fixed at https://github.com/bitnami/charts/pull/1010 and broke again with https://github.com/bitnami/charts/pull/1006. I will work on a fix!

[tompizmor]

@canhnt Just merged https://github.com/bitnami/charts/pull/1019 that should address this issue

[canhnt]

Tested the chart v1.4.2, etcd-0 node starts successfully and can recover from failure (i.e. restart/deletion).

However split-brain issue https://github.com/bitnami/charts/issues/901 still exists, even when I choose 3 replicas from the beginning.

[tompizmor]

Correct! The fix I did does not fix that issue.

[canhnt]

I suspect the v1.4.2 cannot create a multi-node etcd cluster anymore. I try to deploy from the beginning with 3 nodes but it create 3 clusters, each has 1 node. If it is the case, then it breaks a feature.

[tompizmor]

That should not happen. I just deployed the etcd chart with the following options and only a three nodes cluster was created. Also, I even kill a pod and once it is re-scheduled it joins the previous cluster so I am not seeing the split-brain issue.

helm install --name etcd --set statefulset.replicaCount=3,auth.rbac.enabled=false

After enter in the first pod, I see the cluster is properly configured.

I have no name!@etcd-etcd-0:/opt/bitnami/etcd$ etcdctl cluster-health
member 673147ddb335dbb8 is healthy: got healthy result from http://etcd-etcd-0.etcd-etcd-headless.default.svc.cluster.local:2379
member 6e7d8c8e4d8e5b2d is healthy: got healthy result from http://etcd-etcd-2.etcd-etcd-headless.default.svc.cluster.local:2379
member 8b0f5221f70a3e3a is healthy: got healthy result from http://etcd-etcd-1.etcd-etcd-headless.default.svc.cluster.local:2379
cluster is healthy

I have no name!@etcd-etcd-0:/opt/bitnami/etcd$ etcdctl member list
673147ddb335dbb8: name=etcd-etcd-0 peerURLs=http://etcd-etcd-0.etcd-etcd-headless.default.svc.cluster.local:2380 clientURLs=http://etcd-etcd-0.etcd-etcd-headless.default.svc.cluster.local:2379 isLeader=true
6e7d8c8e4d8e5b2d: name=etcd-etcd-2 peerURLs=http://etcd-etcd-2.etcd-etcd-headless.default.svc.cluster.local:2380 clientURLs=http://etcd-etcd-2.etcd-etcd-headless.default.svc.cluster.local:2379 isLeader=false
8b0f5221f70a3e3a: name=etcd-etcd-1 peerURLs=http://etcd-etcd-1.etcd-etcd-headless.default.svc.cluster.local:2380 clientURLs=http://etcd-etcd-1.etcd-etcd-headless.default.svc.cluster.local:2379 isLeader=false

Also you can see here that I am using the latest version:

helm list
NAME    REVISION    UPDATED                     STATUS      CHART       APP VERSION NAMESPACE
etcd    1           Fri Jan 11 16:13:16 2019    DEPLOYED    etcd-1.4.2  3.3.10      default

However, I do have seen an issue when auth.rbac.enabled=true (which is the default value). I need to further check it.

[canhnt]

@tompizmor : you are correct, with auth.rbac.enabled=false, the 3 nodes cluster is up properly. I think my previous test failed because I use auth.rbac.enabled with default true value.

However the 3-nodes cluster cannot recover from failure. If I delete etcd-0 pod:

NAME         READY   STATUS             RESTARTS   AGE
foo-etcd-0   0/1     CrashLoopBackOff   1          16s
foo-etcd-1   1/1     Running            0          96s
foo-etcd-2   1/1     Running            0          88s

Cluster members:

I have no name!@foo-etcd-2:/opt/bitnami/etcd$ etcdctl member list
58db32c7f99d6612, started, foo-etcd-2, http://foo-etcd-2.foo-etcd-headless.canh.svc.cluster.local:2380, http://foo-etcd-2.foo-etcd-headless.canh.svc.cluster.local:2379
5e0e29aa0836e1de, started, foo-etcd-1, http://foo-etcd-1.foo-etcd-headless.canh.svc.cluster.local:2380, http://foo-etcd-1.foo-etcd-headless.canh.svc.cluster.local:2379

Logs in etcd-0:

==> The ID of the host is 0
==> Data exists. Re-joining etcd member
2019-01-12 04:59:01.905012 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://foo-etcd-0.foo-etcd-headless.canh.svc.cluster.local:2379
2019-01-12 04:59:01.905148 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
2019-01-12 04:59:01.905178 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://foo-etcd-0.foo-etcd-headless.canh.svc.cluster.local:2380
2019-01-12 04:59:01.905191 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=foo-etcd-0=http://foo-etcd-0.foo-etcd-headless.canh.svc.cluster.local:2380,foo-etcd-1=http://foo-etcd-1.foo-etcd-headless.canh.svc.cluster.local:2380,foo-etcd-2=http://foo-etcd-2.foo-etcd-headless.canh.svc.cluster.local:2380,
2019-01-12 04:59:01.905198 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2019-01-12 04:59:01.905204 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
2019-01-12 04:59:01.905210 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
2019-01-12 04:59:01.905222 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
2019-01-12 04:59:01.905232 I | pkg/flags: recognized and used environment variable ETCD_NAME=foo-etcd-0
2019-01-12 04:59:01.905274 I | etcdmain: etcd Version: 3.3.10
2019-01-12 04:59:01.905277 I | etcdmain: Git SHA: 27fc7e2
2019-01-12 04:59:01.905282 I | etcdmain: Go Version: go1.10.4
2019-01-12 04:59:01.905285 I | etcdmain: Go OS/Arch: linux/amd64
2019-01-12 04:59:01.905288 I | etcdmain: setting maximum number of CPUs to 10, total number of available CPUs is 10
2019-01-12 04:59:01.915601 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
2019-01-12 04:59:01.915648 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2019-01-12 04:59:01.915799 I | embed: listening for peers on http://0.0.0.0:2380
2019-01-12 04:59:01.915840 I | embed: listening for client requests on 0.0.0.0:2379
2019-01-12 04:59:02.142591 I | etcdserver: name = foo-etcd-0
2019-01-12 04:59:02.142617 I | etcdserver: data dir = /bitnami/etcd/data
2019-01-12 04:59:02.142624 I | etcdserver: member dir = /bitnami/etcd/data/member
2019-01-12 04:59:02.142627 I | etcdserver: heartbeat = 100ms
2019-01-12 04:59:02.142630 I | etcdserver: election = 1000ms
2019-01-12 04:59:02.142633 I | etcdserver: snapshot count = 100000
2019-01-12 04:59:02.142646 I | etcdserver: advertise client URLs = http://foo-etcd-0.foo-etcd-headless.canh.svc.cluster.local:2379
2019-01-12 04:59:02.189527 I | etcdserver: restarting member 518879bcf1df407b in cluster 383abb2999b63f27 at commit index 10
2019-01-12 04:59:02.189587 I | raft: 518879bcf1df407b became follower at term 8
2019-01-12 04:59:02.189601 I | raft: newRaft 518879bcf1df407b [peers: [], term: 8, commit: 10, applied: 0, lastindex: 10, lastterm: 8]
2019-01-12 04:59:02.240311 W | auth: simple token is not cryptographically signed
2019-01-12 04:59:02.256619 I | etcdserver: starting server... [version: 3.3.10, cluster version: to_be_decided]
2019-01-12 04:59:02.257409 I | etcdserver/membership: added member 518879bcf1df407b [http://foo-etcd-0.foo-etcd-headless.canh.svc.cluster.local:2380] to cluster 383abb2999b63f27
2019-01-12 04:59:02.257497 I | etcdserver/membership: added member 58db32c7f99d6612 [http://foo-etcd-2.foo-etcd-headless.canh.svc.cluster.local:2380] to cluster 383abb2999b63f27
2019-01-12 04:59:02.257520 I | rafthttp: starting peer 58db32c7f99d6612...
2019-01-12 04:59:02.257575 I | rafthttp: started HTTP pipelining with peer 58db32c7f99d6612
2019-01-12 04:59:02.259083 I | rafthttp: started streaming with peer 58db32c7f99d6612 (writer)
2019-01-12 04:59:02.259104 I | rafthttp: started streaming with peer 58db32c7f99d6612 (writer)
2019-01-12 04:59:02.259539 I | rafthttp: started peer 58db32c7f99d6612
2019-01-12 04:59:02.259650 I | rafthttp: added peer 58db32c7f99d6612
2019-01-12 04:59:02.259969 I | rafthttp: started streaming with peer 58db32c7f99d6612 (stream MsgApp v2 reader)
2019-01-12 04:59:02.260976 I | rafthttp: started streaming with peer 58db32c7f99d6612 (stream Message reader)
2019-01-12 04:59:02.262055 I | etcdserver/membership: added member 5e0e29aa0836e1de [http://foo-etcd-1.foo-etcd-headless.canh.svc.cluster.local:2380] to cluster 383abb2999b63f27
2019-01-12 04:59:02.262077 I | rafthttp: starting peer 5e0e29aa0836e1de...
2019-01-12 04:59:02.262087 I | rafthttp: started HTTP pipelining with peer 5e0e29aa0836e1de
2019-01-12 04:59:02.262618 I | rafthttp: started streaming with peer 5e0e29aa0836e1de (writer)
2019-01-12 04:59:02.263183 I | rafthttp: started streaming with peer 5e0e29aa0836e1de (writer)
2019-01-12 04:59:02.263874 I | rafthttp: started peer 5e0e29aa0836e1de
2019-01-12 04:59:02.263896 I | rafthttp: added peer 5e0e29aa0836e1de
2019-01-12 04:59:02.264052 N | etcdserver/membership: set the initial cluster version to 3.0
2019-01-12 04:59:02.264088 I | etcdserver/api: enabled capabilities for version 3.0
2019-01-12 04:59:02.264103 N | etcdserver/membership: updated the cluster version from 3.0 to 3.3
2019-01-12 04:59:02.264129 I | etcdserver/api: enabled capabilities for version 3.3
2019-01-12 04:59:02.264302 I | etcdserver/membership: removed member 518879bcf1df407b from cluster 383abb2999b63f27
2019-01-12 04:59:02.265290 I | rafthttp: started streaming with peer 5e0e29aa0836e1de (stream Message reader)
2019-01-12 04:59:02.265563 I | rafthttp: started streaming with peer 5e0e29aa0836e1de (stream MsgApp v2 reader)
2019-01-12 04:59:02.268426 E | etcdserver: the member has been permanently removed from the cluster
2019-01-12 04:59:02.268441 I | etcdserver: the data-dir used by this member must be removed.
2019-01-12 04:59:02.268521 I | etcdserver: aborting publish because server is stopped
2019-01-12 04:59:02.268542 I | rafthttp: stopping peer 5e0e29aa0836e1de...
2019-01-12 04:59:02.268554 I | rafthttp: stopped streaming with peer 5e0e29aa0836e1de (writer)
2019-01-12 04:59:02.268561 I | rafthttp: stopped streaming with peer 5e0e29aa0836e1de (writer)
2019-01-12 04:59:02.268572 I | rafthttp: stopped HTTP pipelining with peer 5e0e29aa0836e1de
2019-01-12 04:59:02.268585 I | rafthttp: stopped streaming with peer 5e0e29aa0836e1de (stream MsgApp v2 reader)
2019-01-12 04:59:02.268593 I | rafthttp: stopped streaming with peer 5e0e29aa0836e1de (stream Message reader)
2019-01-12 04:59:02.268597 I | rafthttp: stopped peer 5e0e29aa0836e1de
2019-01-12 04:59:02.268600 I | rafthttp: stopping peer 58db32c7f99d6612...
2019-01-12 04:59:02.268605 I | rafthttp: stopped streaming with peer 58db32c7f99d6612 (writer)
2019-01-12 04:59:02.268611 I | rafthttp: stopped streaming with peer 58db32c7f99d6612 (writer)
2019-01-12 04:59:02.268660 I | rafthttp: stopped HTTP pipelining with peer 58db32c7f99d6612
2019-01-12 04:59:02.268668 I | rafthttp: stopped streaming with peer 58db32c7f99d6612 (stream MsgApp v2 reader)
2019-01-12 04:59:02.268674 I | rafthttp: stopped streaming with peer 58db32c7f99d6612 (stream Message reader)
2019-01-12 04:59:02.268684 I | rafthttp: stopped peer 58db32c7f99d6612

Actually this is another bug that we may create/track in another issue. For this issue, I confirm that it's fixed and can be closed.

[tompizmor]

@canhnt You are right! That is a different issue.
Also, I realized it only happens if you delete the first pod , foo-etcd-0 in your case, but it works fine if you delete any other pod.

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[nrxr]

It seems like this is fixed with the PR #1039.

[carrodher]

PR #1039 has been merged, can you reproduce the issue using the new version?

[stale[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[raoofm]

@canhnt is your issue fixed?

[canhnt]

With bitnami/etcd chart v1.5.5 (app 3.3.12):

• Deploy with helm install --name etcd bitnami/etcd --set statefulset.replicaCount=3,auth.rbac.enabled=false
• Wait until 3 pods are stable
• Delete pod-0.
• Pod-0 cannot recover from failure, goes into CrashLoopBackOff
• Error logs of pod-0:

==> The ID of the host is 0
==> Data exists. Re-joining etcd member
cat: /bitnami/etcd/data/member_id: No such file or directory

[tompizmor]

@canhnt My last PR should address that issue!