Etcd: 1 node etcd cluster is not working

[suresh-chaudhari]

Error:
{"errorCode":300,"message":"Raft Internal Error","cause":"etcdserver: request timed out","index":0}

To work etcd client , I find that it requires minimum 2 etcd node up to store any data in etcd server.

Issue:
I have created the cluster of 3 etcd node and down 2 etcd node , If we down 2 etcd node it is not able to call the etcd server.

Please resolve this issue ASAP for etcd cluster. etcd cluster must be work with single node.

Find the all details below:
etcd-server version: 3.1.5
org.mousio etcd4j client version: 2.13.0

To reproduce the issue:
I created the cluster of 3 nodes on local machine using following commands:

etcd0 node1 script:
/home/vf-root/installed/etcd/etcd --name er-etcd0
--data-dir /var/lib/etcd/er-ms-node0
--listen-client-urls http://0.0.0.0:2379
--advertise-client-urls http://127.0.0.1:2379
--listen-peer-urls http://127.0.0.1:2380
--initial-advertise-peer-urls http://127.0.0.1:2380
--initial-cluster er-etcd0=http://127.0.0.1:2380,er-etcd1=http://127.0.0.1:2381,er-etcd2=http://127.0.0.1:2382
--initial-cluster-token er-etcd-cluster
--initial-cluster-state new

etcd1 node2 script:
/home/vf-root/installed/etcd/etcd --name er-etcd1
--data-dir /var/lib/etcd/er-ms-node1
--listen-client-urls http://0.0.0.0:2378
--advertise-client-urls http://127.0.0.1:2378
--listen-peer-urls http://127.0.0.1:2381
--initial-advertise-peer-urls http://127.0.0.1:2381
--initial-cluster er-etcd0=http://127.0.0.1:2380,er-etcd1=http://127.0.0.1:2381,er-etcd2=http://127.0.0.1:2382
--initial-cluster-token er-etcd-cluster
--initial-cluster-state new

etcd2 node3 script:
/home/vf-root/installed/etcd/etcd --name er-etcd2
--data-dir /var/lib/etcd/er-ms-node2
--listen-client-urls http://0.0.0.0:2377
--advertise-client-urls http://127.0.0.1:2377
--listen-peer-urls http://127.0.0.1:2382
--initial-advertise-peer-urls http://127.0.0.1:2382
--initial-cluster er-etcd0=http://127.0.0.1:2380,er-etcd1=http://127.0.0.1:2381,er-etcd2=http://127.0.0.1:2382
--initial-cluster-token er-etcd-cluster
--initial-cluster-state new

While you run above 3 scripts it would create the 3 node etcd-cluster, It is working with 3 node and 2 node etcd cluster:

Issue can be reproduce while doing this step:
1.Required to down 2 etcd node from running 3 node etcd-cluster.
2.while running curl command for single node running cluster using below command it gives the error
curl -L -X PUT http://127.0.0.1:2379/v2/keys/foo -d value="bar"

Error:
{"errorCode":300,"message":"Raft Internal Error","cause":"etcdserver: request timed out","index":0}

Let me know if you required more things to do.

[pmoust]

Check what is failure tolerance

[xiang90]

@suresh-chaudhari

Read what @pmoust posted. It will help you understand what etcd can do.

[heyitsanthony]

If those two nodes are permanently lost, the cluster won't be able to make forward progress. The etcd cluster needs disaster recovery to start a new cluster with the old cluster data.