Habitat: RFC: CLI to give better visibility into ring member state

[bixu]

I propose the following additions to the Habitat CLI:

hab svc members and hab svc config edit

Let me explain why.

First, although there is Supervisor output that lets us observe some events in the ring, it’s very difficult to get an overview of overall ring _state_ this way. When debugging issues with gossiped configuration, leader election and member departure it would be enormously helpful to have a way of viewing the most important parts of ring state - hab svc members <svc.group> could provide this.

Second, the full power of broadcasting configuration changes automatically is a bit lost when users are forced to spend lots of time managing .toml files. Instead, hab svc config edit <svc.group>, which could function much like knife node edit would give users fast access to configuration control (there should probably be a sibling hab sac config show command that would allow a read-only view of service config state in the ring). Picture using this command to examine and then modify the a setting for a config key like log_level, for example.

Here’s how the interfaces for the commands might look:
```

$ hab svc members --json=true postgresql.dev

{
"member_id": "2c9f3a74935a43a29ddb95647e0e2c15",
"service_group": "postgresql.dev",
"pkg_ident": "core/postgresql/9.6.9/20181018102331",
"leader": false,
"update_leader": false,
"ip": "172.31.13.170",
"alive": true,
"departed": false,
"cfg": {
"port": 5433
}
}


$ hab svc members postgresql.dev

service:
member_id: 2c9f3a74935a43a29ddb95647e0e2c15
service_group: postgresql.dev
pkg_ident: core/postgresql/9.6.9/20181018102331
leader: false
update_leader: false
ip: 172.31.13.170
alive: true
departed: false

service cfg:
port: 5433

$ hab svc config edit postgresql.dev # opens TOML in local EDITOR

log_level = 'ERROR'
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,client=%h %r (%x:%e)'
logging_collector = 'on'
max_connections = 100
max_locks_per_transaction = 64
port = 5432

[replication]
enable = false
lag_health_threshold = 1048576
name = 'replication'
password = 'replication'

[superuser]
name = 'admin'
password = 'admin'

edits can be made, then the buffer is rendered to disk and

uploaded into the ring as the latest config

[HT154]

How would hab svc config edit/show ... handle the different layers of config? Would they only include gossiped config? Or would it include a combination of gossiped+user.toml+default.toml? How would this work in situations where different group members have different user.toml (is this possible??)? How would this work if packages in a service group are heterogenous and have different default.toml (pretty sure this is possible with packages of different versions/origins)? If only gossiped config is included, I can see this being a possible source of confusion for users where the effective config is not visible via the CLI

[y-me-y]

I am totally in support of both of the options above but would add some caveats to them

• In terms of editing the config option with the command line option - I'd like to make sure maybe we could sanitize secret config by embedding it or hiding certain values unless we were in a debug mode.
• In hab svc members postgresql.dev I think it would be useful to prevent this from happening with certain packages - like a parameter in the plan.sh that prevents this return. Or adding functionality like above where we can sanitize certain sections of the config.

[bixu]

@HT154, I think the only config that matters is the current config in the ring (gossiped) - it's the one that services actually use/render out.

@y-me-y, I totally get your concerns there. I think it's also true that if an operator has gotten to this point they have full control anyway, and if we don't trust them we are already p0wned, if you understand me?

[y-me-y]

@bixu - I think it's more the controlled access I am cool with you having access to this server - but not all the information on the ring. I support getting the information but I see cases where putting some controls in place especially for large orgs would be useful and beneficial. - Like http redact feature that was introduced.

[bixu]

@y-me-y, Agreed. I’d be willing to use a minimal, less secure version of the feature for now if I could get it sooner, however ;)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[bixu]

I still think visibility in to the ring is super important for debugging.