Sensu-go: Investigate the etcd alarms API

[echlebek]

Etcd provides an alarms API. It is basically undocumented. Find out what it does, and figure out if we can use it to provide some information to the sensu backend.

[mercul3s]

Etcd alarms seem to be reserved internally for incidents involving cluster maintenance or instability. I dug into this a bit and thought of a couple things we can do with them:

• Add them to the health api
  
  
  
  • We should include any alarms in the health api response. I did a quick and dirty proof of concept here. Note that this branch is incomplete and will break the existing cluster health command as the updated return value is not yet accounted for.
  
  
• Add a watcher for a PUT on alarms keys (alarms have their own store) and fire a handler on an event. This might be similar to a keepalive handler, but for sensu-backend, and can act as a dead man's switch in case of any etcd alarms (indicating trouble with an etcd cluster).

[mercul3s]

@cyphus I summarized a couple bullet points from our earlier conversation and the research I did - feel free to add comments if you've got 'em.

[cyphus]

I think that the presence of any alarms is valuable info in the sensuctl health response, and that dead man's switch would make a great catch-all for any unexpected error we encounter. @mercul3s if you're happy with the level you've investigated etcd alarms, I'd suggest closing this and making issues for what you want to see implemented.

[mercul3s]

@cyphus I've added #1978 and #1979 for these enhancements.