Habitat: Instrument the Supervisor with metrics

[christophermaier]

The Supervisor is currently a bit of an operational black box, which complicates both the operation of any non-trivial Habitat installation, as well as our efforts to improve the Supervisor.

Chef engineering appears to be moving toward Prometheus as a metrics platform-of-choice. We even had rudimentary support for Prometheus in the Supervisor in the past, but it was removed because the implementation wasn't fully thought out.

Here is an initial list of metrics that could be useful to expose:

• memory usage (broken down by subsystem if possible)
  The jemalloc_ctl crate could be useful here.
  Exposing the size of important data structures (such as "rumor heat") would be nice, but potentially complicated.
• Number of rumors, by type
• CPU utilization (possibly by thread/subsystem)
• bytes sent
• bytes received
• SWIM messages sent (by type)
• SWIM messages received (by type)
• SWIM ping/ack/pingreq round-trip time
• gossip messages sent (by type)
• gossip messages received (by type)
• gossip round trip time
• supervisor "tick" time
• service "tick" time
• hook run times
• number of alive peers
• number of suspect peers
• number of confirmed peers
• number of departed peers
• RPC call counts / timing
  load, unload, start, stop, etc.
• timing for how long metric collection itself takes
• HTTP gateway requests, timings

Figuring out how to surface basic memory/CPU/RPC/etc info from the Launcher would be interesting, as well, but represents the additional wrinkle of it being a separate process from the Supervisor.

Some interesting resources that could help:

• Rust Prometheus crate
  https://github.com/pingcap/rust-prometheus
• Prometheus Best Practices
  https://prometheus.io/docs/practices/
• Consul Telemetry Documentation
  https://www.consul.io/docs/agent/telemetry.html
  (Mainly listed as a source of inspiration)

[jeremymv2]

It might be nice to keep tabs on file descriptors via something like getdtablesize so we can ensure we don't introduce leaks for long running Supervisors.

[irvingpop]

Another good source of metrics inspiration is etcd's: https://coreos.com/etcd/docs/latest/metrics.html

We've found that the etcd_disk_ and etcd_network_ metrics have been priceless in diagnosing dodgy customer computing environments for chef-backend.

[christophermaier]

Another potentially interesting thing would be to record how long elections take from initiation to resolution.

[christophermaier]

Another possible metric idea: number of ignored rumors (by type). This could give an idea how much time and network traffic we devote to spreading old rumors.

[christophermaier]

More metric ideas!

• current incarnation of each Supervisor
• current election term for various elections