Vault: Docs recommend running vault agent as init container

[tustvold]

Apologies if this isn't the correct place to post this, but the docs for integrating vault with kubernetes currently recommend running the vault agent as an init container that exits after authenticating.

This, however, is problematic as if for whatever reason the generated token expires, e.g. a temporary network outage, the init container will never be re-run and the pod will remain stuck indefinitely in an error state. Kubernetes currently doesn't have a way to restart the entire pod on error, see here, it will just continually restart the container that exited, had the failing health check, etc...

If using consul-template running vault agent as a sidecar and using vault_agent_token_file is possibly a safer recommendation. Once the outage resolves itself the vault-agent will generate a new token and pass this through to the application container(s). Even if it takes the application pods restarting to load in new dynamic credentials, which kubernetes will do automatically, this is surely preferable to remaining indefinitely broken?

Even if using envconsul it is possible to workaround the fact it currently doesn't support vault_agent_token_file by mounting the generated token at ~/.vault-token, but I presume this will be resolved properly once envconsul is updated to use go modules.

[tustvold]

It is possibly also worth noting that it is guaranteed that the generated token will expire after the max_ttl, so for installations following the instructions in the documentation with a default vault configuration - all their workloads will get stuck with no credentials after 32 days.

[robinpercy]

Agreed, the docs don't call out the TTL consideration clearly enough.

FWIW, I've tested 2 workarounds, based on using vault-agent with consul-template, as per: https://learn.hashicorp.com/vault/identity-access-management/vault-agent-k8s

1. Make vault-init a sidecar instead of init container, and set exit_after_auth=false in its config, that will keep the agent alive and able to renew the token for you. Or,
2. Leave the init container alone, but set renew_token = true for consul-template.

[tomerleib]

faced the same issues described here.
Eventually, @robinpercy first solution helped me.