Etcd: client is unaware of unhealthy server when getting response

[yichengq]

Because client pins to one endpoint if possible, it will always make requests to one server. If the target server is isolated from etcd cluster and becomes unhealthy for a long time, the client will be outdated too for that long time. We don't provide way to handle it or warn it today.

[xiang90]

/cc @mx2323 You might want to follow on this.

[gyuho]

Is this still an issue?

Raft says:

The leader handles all client requests (if a client contacts a follower, the follower redirects it to the leader).

Can we use the same mechanism to notify client at least the target server is not reachable? When used with proxy, I see the error message of proxy endpoint is not available when the proxy is wrongly configured. Does etcd handle differently in this case?

Thanks,

[yichengq]

@gyuho For watch and read, it gets the data from local store. It is allowed/normal for clients to get data from local store. We just don't provide way to handle or warn about unhealthy server today.

[gyuho]

I see. Thanks,

[mitake]

@yichengq @gyuho if a client turns on client.GetOption.Quorum, reading from leader's store is enforced so the client can be aware of unhealthy server.

For example,

1. create a cluster with 3 nodes
2. create a key
3. kill 2 nodes
4. get the key with etcdctl get with the option --quorum (which turns on client.GetOption.Quorum)
   in such a sequence, the get request in 4th step will fail because of losing quorum. So I think current etcd already has a solution for this issue.

[mx2323]

The issue isn't on reads, it's when a watch is registered on an etcd server, but the server on which the watch is sent to is partitioned from the rest of the etcd cluster. In these situations, the client will sit with a watch untriggered when it could have gone to another server and gotten the new update.

[mitake]

@mx2323 thanks for your pointing, I couldn't consider the case of watch.

[xiang90]

closed by https://github.com/coreos/etcd/pull/5332

[immesys]

The readme still says reads can be satisfied by unhealthy member, and references this issue. Is that still the case? If not, perhaps amend the readme?

[bweston92]

Readme still seems to be out of date.