Etcd: ETCD rechability and health check

[sanamsarath]

I have ETCD running on a remote cluster. I need to keep track of the readability/liveliness of the ETCD cluster from the application side cluster. I have a proxy monitoring app between the ETCD and applications that rely on ETCD.

Following are the 2 possibilities I could think of:

1. start a dummy watch request, and if the connection fails I will update the applications that the server is not reachable.
2. Keep sending the dummy get requests with some time interval and if 3 get responses fail continuously , could assume the ETCD is not reachable.

What would be the best practice in general to keep track of the ETCD liveliness ??

Thanks.

[hexfusion]

If all you want is simple health check of members what about endpoint health?

ETCDCTL_API=3 etcdctl --endpoints $endpoints endpoint health
127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.969572ms

https://github.com/etcd-io/etcd/blob/fa521f4e00fedfb6d98449d92a6408d0b3b0d922/etcdctl/ctlv3/command/ep_command.go#L87

[hexfusion]

@sanamsarath does this work for you?

[sanamsarath]

@hexfusion sorry for the late response.

I can consider using the health check option, but I need to try checking if it works with HTTP json interface instead of default etcdctl as I am implementing my client in c++.

Also, I want to make sure that health check request will return the health of entire cluster and not the single endpoint/member alone.

Thanks for the suggestion.

[hexfusion]

I can consider using the health check option, but I need to try checking if it works with HTTP json interface instead of default etcdctl as I am implementing my client in c++.

For REST you can do a get against "/health"

Also, I want to make sure that health check request will return the health of entire cluster and not the single endpoint/member alone.

It returns the health of each node so "health" is cumulative etcdctl will return "Error: unhealthy cluster" is one member returns unhealthy for endpoint health. So in a 3 node cluster one unhealthy node still leaves a quorate cluster but "unhealthy". Once quorum is lost then you have bigger problems:). But in that case all members will return false or simply timeout. So you would need to manage that logic within your application.

I hope that helps.

[sanamsarath]

Thank you.