Vault: Allow vault to reload its config without sealing the vault

[BRMatt]

Quite often CF systems are configured to restart/reload services when their configuration changes. In the case of Vault this would seal all servers, resulting in downtime, and would require operator intervention to fix. There are situations where the Vault configuration may change intentionally (e.g. the statsd address changes), or unintentionally (someone refactoring chef cookbooks accidentally removes trailing whitespace from the config file).

Admittedly it could be possible to reconfigure and unseal each Vault server individually, but this requires a lot of manual work and increases the risk of downtime. In an ideal world, Vault would be able to safely reload its configuration from disk, whilst keeping the vault unsealed.

One thing to bear in mind is that this could potentially allow an attacker to reconfigure Vault in a harmful way (e.g. changing the storage backends), but on the other hand, a properly configured server should prevent unauthorised users from editing Vault's configuration. Given that the security model excludes memory analysis, excluding Vault's configuration from being edited doesn't seem unreasonable, especially as an attacker could always change the config and wait for an operator to restart the service.

Thoughts?

[hashbrowncipher]

I'd be interested in a fix for this.

I am considering using short-lived certificates for my Vault server. To achieve this, Vault would have to be restarted and unsealed every time the certificate rotates.

[armon]

Marked as an enhancement. Doing an online change of the TLS keys will be particularly challenging.

[theonewolf]

:+1:

[BSick7]

:+1: We run vault on coreos nodes in high-availability mode. It feels clunky that a restart after security updates would seal the vault.

[doublerebel]

:+1:

[jefferai]

As an FYI, this is on the radar, but only in a limited fashion. Reloading configuration will require going through the API, and the only item that is planned to be supported, at least initially, is the Vault server certificate/private key. Storage backends could eventually implement some reloading logic of their own for their parameters.

[roboll]

Per the docs:

Starting with 0.5.2, limited configuration options can be changed on-the-fly by sending a SIGHUP to the server process. These are denoted below.

Assuming those docs are in fact correct?

[vishalnayak]

@roboll

Yes. the docs are correct!

The config file that is used to launch Vault can be updated to contain new tls_cert_file and tls_key_file. A SIGHUP signal to Vault process will then reload the certificate and its private key.

[theonewolf]

I've also had good mileage using NGINX as a front-end in forwarding connections to Vault. So then refreshing certs becomes an NGINX issue, and I don't have to worry about Vault becoming unreachable / having issues due to expired certificates.

I bind Vault to localhost and don't forward such connections over a physical network.

Does anyone have any concerns about running such a configuration: HTTPS --> NGINX --> HTTP (localhost) --> Vault.

[jefferai]

Doing so breaks the trust chain between client and application, and allows for the possibility of traffic sniffing between the reverse proxy and app. This may not be of concern to you, but if you have to be compatible with certain industry standards, like PCI 3.0, this is not allowed.

[theonewolf]

Great point @jefferai. Glad Vault has this functionality now, so I can stop using NGINX in the near future.

[jefferai]

I'm closing this for the moment as TLS parameter reloading is implemented. For other specific reloading requests, open up new issues and we can address each one on a case-by-case basis depending on feasibility.