Consul: How do we tell when a new member is synced?

[fraenkel]

We are automating the updates across multiple members. How do we detect when its safe to move onto the next member?

So far all we have been able to determine is to wait until we see the magical log line, "Synced service 'consul'". We have yet to find any API which will tell us this fact. We could look at the index but that isn't a guarantee.

[DanyC97]

+1

[armon]

@fraenkel Do you just mean when a client has sync'd all the services/checks or do you mean when a server has sync'd the Raft transactions? I don't think we expose the client sync state, but you could always query the agent info with /v1/agent/services and then cross-check with /v1/catalog/node/ to ensure they match. For the Raft transactions, using consul info to check the last_log_index is the easiest way.

[fraenkel]

@armon When we roll an upgrade, we are trying to determine when it's safe to move to the next server. We have noticed that if we roll too early, we either lose quorum (very bad), or we maintain quorum but lose data (bad).
Currently we are watching the consul service on a given node and when we see the Tags go from nil to [], we know the Synced service 'consul' message has appeared and it seems to be safe to move on.

If we can compare the last_log_index with the commit_index, then I would go with that. Just trying to find a safe and sane way to automate install and upgrades.

[armon]

@fraenkel Oh yeah, I would use the last_log_index and commit_index between the existing leader and the new nodes to check.

[gtmtech]

Is there any reason there isnt an HTTP API call which tells you the last_log_indexes of all the servers? (Is that too hard/impossible ?)

[slackpad]

This was added in 0.8.0 - https://www.consul.io/api/operator/autopilot.html#read-health. You can get all the critical raw info, as well as an over "healthy" boolean. See also https://www.consul.io/docs/guides/autopilot.html which has some interesting features related to server upgrades.