Etcd: rpc error: code = DeadlineExceeded desc = context deadline exceeded

[OneCodeMonkey]

starting etcd cluster with 3 nodes failed. The first two node connected, but the third reports:

rpc error: code = DeadlineExceeded desc = context deadline exceeded

[spzala]

Please provide more info if you are still having this issue.

[smartvolshell]

version 3.4
execute "etcdctl endpoint health"
"error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded",
"msg": "retrying of unary invoker failed",

[tsvetko-droid]

I think I am hitting a similar issue:

Kubernetes version: 1.18.8

Etcd cluster with 3 members

etcdctl version: 3.4.13
API version: 3.4

Etcd logs show the following errors:

{"level":"debug","ts":"2020-11-05T20:09:11.191Z","caller":"v3rpc/watch.go:193","msg":"failed to receive watch request from gRPC stream","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"debug","ts":"2020-11-05T20:17:12.193Z","caller":"v3rpc/watch.go:193","msg":"failed to receive watch request from gRPC stream","error":"rpc error: code = Canceled desc = body closed by handler"}

etcdctl reports all endpoints are healthy:

etcdctl --endpoints=https://:2379,https://:2379,https://:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key endpoint health

https://:2379 is healthy: successfully committed proposal: took = 22.183879ms
https://:2379 is healthy: successfully committed proposal: took = 22.319905ms
https://:2379 is healthy: successfully committed proposal: took = 24.610958ms

However, when I run etcdctl endpoint health on each cluster member I get:

etcdctl endpoint health
{"level":"warn","ts":"2020-11-05T20:45:06.862Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-c8b0da71-f0fe-49c9-8ac5-e4d33077926c/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
command terminated with exit code 1

Running etcdctl alarm list on each member also shows the same:

etcdctl alarm list
{"level":"warn","ts":"2020-11-05T20:47:50.885Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-ef24a712-6b02-47ac-8b81-dc6136142d2e/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection closed"}
Error: context deadline exceeded
command terminated with exit code 1

[sumit-anantwar]

Facing the same issue.
I am trying to setup ETCD cluster on 2 nodes.
The services on both the nodes are properly running, but etcdctl fails with Error: context deadline exceeded
Checking endpoint health also reports the same error.

vagrant@master-1:~$ etcdctl endpoint health                                                                           
{"level":"warn","ts":"2020-11-26T21:49:03.204Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-08349238-638e-434d-aff9-37777482a226/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}                                                  
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded                                     
Error: unhealthy cluster                                                                                              

This is my systemd unit file for etcd.service

[Unit]                                                                                              
Description=etcd                                                                                    
Documentation=https://github.com/etcd-io/etcd                                                         

[Service]                                                                                           
ExecStart=/usr/local/bin/etcd \                                                                     
  --name master-1 \                                                                                 
  --cert-file=/etc/etcd/etcd-server.crt \                                                           
  --key-file=/etc/etcd/etcd-server.key \                                                            
  --peer-cert-file=/etc/etcd/etcd-server.crt \                                                      
  --peer-key-file=/etc/etcd/etcd-server.key \                                                       
  --trusted-ca-file=/etc/etcd/ca.crt \                                                              
  --peer-trusted-ca-file=/etc/etcd/ca.crt \                                                         
  --peer-client-cert-auth \                                                                         
  --client-cert-auth \                                                                              
  --initial-advertise-peer-urls https://192.168.5.11:2380 \                                         
  --listen-peer-urls https://192.168.5.11:2380 \                                                    
  --listen-client-urls https://192.168.5.11:2379,https://127.0.0.1:2379 \                           
  --advertise-client-urls https://192.168.5.11:2379 \                                               
  --initial-cluster-token etcd-cluster-0 \                                                          
  --initial-cluster master-1=https://192.168.5.11:2380,master-2=https://192.168.5.12:2380 \         
  --initial-cluster-state new \                                                                     
  --data-dir=/var/lib/etcd                                                                          
Restart=on-failure                                                                                  
RestartSec=5  

[Install]
WantedBy=multi-user.target                                                                                      

[pnoker]

I think I am hitting a similar issue:

Kubernetes version: 1.18.8

Etcd cluster with 3 members

etcdctl version: 3.4.13
API version: 3.4

Etcd logs show the following errors:

{"level":"warn","ts":"2020-12-10T13:24:39.343+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-9bccea73-19e9-47b2-b0b5-a2080c2ee773/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded