Consul: Gossip encryption key is not validated before made persistent in local.keyring

[markfra]

When you setup an agent from scratch like below, a faulty encrypt key is not validated before it's being written to local.keyring. As a consequence modifying the encrypt via the config-file has no effect. Removing the keyring mitigates the issue, but in my opinion the key should be validated earlier:

Consul v0.5.2
$ cat encrypt.json
{ "encrypt": "asdfasdfasdfasdfasdfasdf" }
$ consul agent -data-dir=/tmp/consul -config-file=encrypt.json
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
==> Error starting agent: Failed to configure keyring: key size must be 16 bytes

$ cat encrypt.json
{ "encrypt": "Y3bFNO1Fi4JK+BpTzxKvCA==" }
$ consul agent -data-dir=/tmp/consul -config-file=encrypt.json
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
==> Error starting agent: Failed to configure keyring: key size must be 16 bytes

$ cat /tmp/consul/serf/local.keyring
["asdfasdfasdfasdfasdfasdfasdf"]
$ consul agent -data-dir=/tmp/consul -config-file=encrypt.json
==> WARNING: It is highly recommended to set GOMAXPROCS higher than 1
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
Node name: 'consul.localdomain'
Datacenter: 'dc1'
Server: false (bootstrap: false)
Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
Cluster Addr: 10.0.2.15 (LAN: 8301, WAN: 8302)
Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
Atlas:

==> Log data will now stream in as it occurs:

2015/10/13 21:30:50 [INFO] serf: EventMemberJoin: consul.localdomain 10.0.2.15
2015/10/13 21:30:50 [ERR] agent: failed to sync remote state: No known Consul servers

[jhmartin]

Any progress on this?

[brahama]

Still happening in 0.6.4

[rboyer]

I have a simple proposed PR that fixes this issue.

[ross]

Would like to see this one fixed as well. We just ran into this problem via a slightly different route. We build system images in our CI pipeline during which things run in the "test" environment. Consul is spun up there and sets the encrypt value into serf/local.keyring. When those images are spun up in staging/production the correct key is placed into the consul config file, but ignored and the system is unable to join the cluster.

[slackpad]

Fixed via #2270.

[slackpad]

@ross I think you are hitting the current behavior - the configured key is designed to be an initial one only - https://www.consul.io/docs/agent/options.html#_encrypt:

Specifies the secret key to use for encryption of Consul network traffic... The provided key is automatically persisted to the data directory and loaded automatically whenever the agent is restarted. This means that to encrypt Consul's gossip protocol, this option only needs to be provided once on each agent's initial startup sequence. If it is provided after Consul has been initialized with an encryption key, then the provided key is ignored and a warning will be displayed.

Consul does this because there are commands to update the keys later and users won't want to revert to the initial key when restarting. You probably want your CI pipeline to blow away that state before baking an image.

[ross]

Consul does this because there are commands to update the keys later and users won't want to revert to the initial key when restarting. You probably want your CI pipeline to blow away that state before baking an image.

Realized that after digging a bit further. I think we can actually use the stuff you just landed to work around that by having an invalid key during tests/image building so that it'll never get persisted.