Etcd: v3.3.0-rc.0 member list, endpoint status with auth cannot connect to one member

[OPSTime]

[root@node1 etcd]# ETCDCTL_API=3 /usr/local/etcd/etcdctl --user 'root:111111' member list -w table
+------------------+---------+-------+------------------------+------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+------------------------+------------------------+
| 736948b5f5c09617 | started | node1 | http://192.168.0.81:2380 | http://10.20.6.81:2379 |
| 8cd04f584be13fc0 | started | node2 | http://192.168.0.82:2380 | http://10.20.6.82:2379 |
| c4cf1eeedaaed948 | started | node3 | http://192.168.0.83:2380 | http://10.20.6.83:2379 |
+------------------+---------+-------+------------------------+------------------------+
[root@node1 etcd]# ETCDCTL_API=3 /usr/local/etcd/etcdctl --user 'root:111111' endpoint status --cluster -w table
Failed to get the status of endpoint http://192.168.0.83:2379 (context deadline exceeded)
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://192.168.0.81:2379 | 736948b5f5c09617 | 3.3.0 | 701 MB | true | 81 | 288007 |
| http://192.168.0.82:2379 | 8cd04f584be13fc0 | 3.3.0 | 701 MB | false | 81 | 288008 |
+------------------------+------------------+---------+---------+-----------+-----------+------------+

[OPSTime]

$ /usr/local/etcd/etcd --version
etcd Version: 3.3.0
Git SHA: f7a395f
Go Version: go1.9.2
Go OS/Arch: linux/amd64

$ ETCDCTL_API=3 /usr/local/etcd/etcdctl version
etcdctl version: 3.3.0
API version: 3.3

[xiang90]

@spzala would you like to take a look?

[spzala]

@xiang90 sure, I am in mid of another issue but I will be soon digging into it.

[spzala]

@lyddragon meanwhile, would you please follow the steps here on compact/defrag, if that's something can be done in your env and see if that helps? It's something that has helped me in the recent past with similar error - https://github.com/coreos/etcd/blob/master/Documentation/op-guide/maintenance.md#space-quota steps under "Removing excessive keyspace data and defragmenting the backend database will put the cluster back within the quota limits:"

I couldn't recreate issue on a quick try on my master branch on a development/single node machine but I will dig more.

$ ./bin/etcdctl --user 'root:111111' endpoint status --cluster -w table
+-----------------------+------------------+-----------+---------+-----------+-----------+------------+
|       ENDPOINT        |        ID        |  VERSION  | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-----------------------+------------------+-----------+---------+-----------+-----------+------------+
| http://localhost:2379 | 8e9e05c52164694d | 3.3.0+git |   20 kB |      true |        25 |         52 |
+-----------------------+------------------+-----------+---------+-----------+-----------+------------+

[OPSTime]

endpoint http://192.168.0.83:2379 is stoped, status have not chage in member list! Is still started!

[spzala]

Thanks @lyddragon for verifying .. I will investigate more but I need to set up multi-node cluster first.

[spzala]

@lyddragon hi, I hope you can wait on it some more. I couldn't reproduce it on single node and I am still in process of of setting up 3 node cluster. Thanks!

[spzala]

I could not recreate the error with the 3.3.0, might be it's something with the user environment. /cc @xiang90

$ ./etcdctl --user 'root:111111' endpoint status --cluster -w table
+-------------------------+------------------+---------+---------+-----------+-----------+------------+
|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+-------------------------+------------------+---------+---------+-----------+-----------+------------+
| http://xxx:2379 | 6c4483e5661b6662 |   3.3.0 |   25 kB |     false |         6 |         15 |
| http://xxx:2379 | 768a83b2c90eb88d |   3.3.0 |   25 kB |     false |         6 |         15 |
| http://xxx:2379 | de78facad5da439b |   3.3.0 |   25 kB |      true |         6 |         15 |
+-------------------------+------------------+---------+---------+-----------+-----------+------------+

[xiang90]

@spzala i cannot reproduce this either.

@lyddragon can you please provide more information on how to reproduce it? A step by step guide would be preferred.

[gyuho]

I couldn't reproduce either with auth enabled, in master branch.

[spzala]

@xiang90 @gyuho thanks for taking a look at it!

[xiang90]

i am closing this out. please reopen if a reproduce can be provided.