Consul-template: Log the source of the vault token to help in diagnosing renewal behavior

[angrycub]

The Vault token can be read from one of 3 different sources. Some of these are set in a backward compatible way that hides the source and makes determining the token's source difficult. Adding a log entry for these could go a long way in helping here.

As a reminder the 3 ways are:

• config file
• token file (defaults to ~/.vault-token for backward compatibility)
• environment variable VAULT_RENEW_TOKEN

Consul Template version

consul-template v0.25.1 (b11fa800)

Configuration

None supplied, using flags and defaults.

Command

consul-template -exec 'bash -c "while true; do echo $(date); sleep 300; done"'

Debug output

https://gist.github.com/angrycub/d5f9693bb3fcb5fc5b6f4e01355bb001

Expected behavior

As no Vault token was provided by any means, I would not expect a Vault token renewer to start since it would be impossible to renew. In the absence of a configured token (via environment or config), I would expect the behavior provided by setting -vault-renew-token=false

Actual behavior

A renew starts and attempts to use the default Vault address of https://127.0.0.1:8200 resulting in

2020/08/26 19:14:42.213972 [WARN] vault.token: failed to renew: Put https://127.0.0.1:8200/v1/auth/token/renew-self: dial tcp 127.0.0.1:8200: connect: connection refused

this will retry until max attempts and then terminate the consul-template process and the child process.

Steps to reproduce

1. Run consul template with the following command, providing no other configuration:

consul-template --log-level=trace -exec 'bash -c "while true; do echo $(date); sleep 300; done"'

[eikenb]

Vault Token renewal should be disabled if not set. But it looks for the token in places other than the config file by default. Do you happen to have the environment variable VAULT_RENEW_TOKEN set?

For more see: #1297 and https://github.com/hashicorp/consul-template/commit/7a4d06a4b821ac64a851e4f1d0289a2d07ed3e1c

[eikenb]

Hey @angrycub, thanks for filing this issue.

First, see above. I forgot to address you. Also when checking for VAULT_RENEW_TOKEN, also check for the ~/.vault-token file. If that exists and contains a token, it will trigger token renewal.

I am not able to reproduce this issue with 0.25.1, so my current theory is you have one of the above set.

[eikenb]

If that exists and contains a token, it will trigger token renewal.

It actually doesn't matter what that file contains. It is assumed if it exists it contains a token and the contents are read and assigned to the token.

[angrycub]

That was it! A ~/.vault_token file was the culprit. Perhaps we could add logging to explain where it found its vault credential? It was unexpected because it was running in a systemd unit, but there was a .vault_token in root's home directory.

[eikenb]

Good idea. If you don't mind I'll convert this ticket into a request to have the token source logged.