Consul-template: Add "token" function to retrieve (derived) Vault tokens

[mbeiter]

This is a feature request to add a new function that allows retrieving a derived Vault token and writing that derived Vault token into a configuration file for another service to use.

Use case: A VM (e.g. in EC2) where more than one service is running on. Consul template to retrieve a token that is derived from Consul Template's primary Vault auth token, using the ault /auth/token/create endpoint, and then making such a derived token available to a service running on the VM. If several services are running on the VM, each with different unix users and privileges, it is then possible to assign to each of them a derived Vault token with a service specific minimal subset of policies associated with the Vault token used by Consul Template to connect to Vault.

The /auth/token/create endpoint returns a response wrapped into an auth object, which is not parsable by the current secret function: The existing secret function requires a response of a Vault secrets engine, which is wrapped in a "Data" object.

The API call against /auth/token/create is currently possible and made, the problem is then the subsequent parsing of the result.

Consul Template version

$ consul-template -version
consul-template v0.20.0 (b709612c)

Configuration

Assume that the Vault token used by Consul Template has the policies policy_1, policy_2, and policy_3 associated, it is currently possible to configure this for getting a derived token with policy_1 and policy_2:

token={{with secret "/auth/token/create" "policy=policy_1" "policy=policy_2"}}{{ .auth.client_token }}{{ end }}

Which results in the following trace:

2019/06/25 21:11:48.923399 [TRACE] (view) vault.write(auth/token/create -> fe4759db) starting fetch
2019/06/25 21:11:48.923466 [TRACE] vault.write(auth/token/create -> fe4759db): PUT /v1/auth/token/create?stale=true&wait=1m0s
2019/06/25 21:11:49.445762 [TRACE] (view) vault.write(auth/token/create -> fe4759db) marking successful data response
2019/06/25 21:11:49.445901 [TRACE] (view) vault.write(auth/token/create -> fe4759db) successful contact, resetting retries
2019/06/25 21:11:49.445966 [TRACE] (view) vault.write(auth/token/create -> fe4759db) received data
2019/06/25 21:11:49.446084 [TRACE] (view) vault.write(auth/token/create -> fe4759db) starting fetch
2019/06/25 21:11:49.446149 [TRACE] vault.write(auth/token/create -> fe4759db): starting renewer
2019/06/25 21:11:49.446396 [DEBUG] (runner) receiving dependency vault.write(auth/token/create -> fe4759db)
2019/06/25 21:11:49.446461 [DEBUG] (runner) initiating run
2019/06/25 21:11:49.446502 [DEBUG] (runner) checking template e09b9966b548472790930921d73d98ec
2019/06/25 21:11:49.446698 [DEBUG] (runner) missing data for 1 dependencies
2019/06/25 21:11:49.446948 [ERR] (cli) /REDACTED/template.ctmpl: execute: template: :1:101: executing "" at <.auth.client_token>: can't evaluate field auth in type *dependency.Secret

No surprise, because type *dependency.Secret obviously requires return of a Data object in the Vault response.

Note: Accessing the data object instead obviously results in an empty data object:

token={{with secret "/auth/token/create" "policy=policy_1" "policy=policy_2"}}{{ .Data}}{{ end }}

Expected behavior

A new function like this should be available:

token={{with vaultauth "/auth/token/create" "policy=policy_1" "policy=policy_2"}}{{ .auth.client_token }}{{ end }}

[pySilver]

@eikenb so there is no workaround and its not supported yet?

[eikenb]

@pySilver I haven't had time to work look to closely at this but after a quick skim I'd say that no, it isn't supported by consul-template yet.

I'm not aware of a workaround. You might want to look at vault agent as it might allow for something, but I'm not that familiar with it so can't say for sure.

[Is0MerLNtaLt]

@eikenb Vault Agent has this issue too.

[luisfurnas]

This actually poses a problem for me, since I really needed to send a Vault token with a restrictive policy to a Nomad job, and it seems I can't do it through Consul Template due to this issue.
Is anyone working on this?

[eikenb]

The team has been distracted with a new project for som time but we are about done with the initial version of it and will have time for these issues soon. Please remember to :+1: vote for this issue. Thanks.

[davidrosson]

I'm using consul-template v0.24.1 (running as a container in a K8s Pod) and was able to use the secret function in the following configuration to retrieve a Vault token and write it to a file:

{{with secret "/auth/token/create" "policy=policy_1" }}{{.Auth.ClientToken}}{{ end }}

The difference is using .Auth.ClientToken instead of .auth.client_token to get the token data. Not sure which version of consul-template implemented this capability.

Here's the trace:

2021/04/22 13:23:09.549734 [INFO] consul-template v0.24.1 (58aa6c60)
2021/04/22 13:23:09.549751 [INFO] (runner) creating new runner (dry: false, once: false)
...
2021/04/22 13:23:09.552023 [TRACE] (view) vault.write(auth/token/create -> 2ded95de) starting fetch
2021/04/22 13:23:09.552040 [TRACE] vault.write(auth/token/create -> 2ded95de): PUT /v1/auth/token/create?stale=true&wait=1m0s
2021/04/22 13:23:09.748942 [TRACE] vault.token: successfully renewed
2021/04/22 13:23:09.779680 [TRACE] (view) vault.write(auth/token/create -> 2ded95de) marking successful data response
2021/04/22 13:23:09.779705 [TRACE] (view) vault.write(auth/token/create -> 2ded95de) successful contact, resetting retries
2021/04/22 13:23:09.779709 [TRACE] (view) vault.write(auth/token/create -> 2ded95de) received data
2021/04/22 13:23:09.779729 [TRACE] (view) vault.write(auth/token/create -> 2ded95de) starting fetch
2021/04/22 13:23:09.779733 [TRACE] vault.write(auth/token/create -> 2ded95de): starting renewer
2021/04/22 13:23:09.779917 [DEBUG] (runner) receiving dependency vault.write(auth/token/create -> 2ded95de)
2021/04/22 13:23:09.779933 [DEBUG] (runner) initiating run
2021/04/22 13:23:09.779936 [DEBUG] (runner) checking template e4e7973b3ae717cf947a96e1944f88ee
2021/04/22 13:23:09.780213 [DEBUG] (runner) rendering "(dynamic)" => "/vault/secrets/vault-token"
2021/04/22 13:23:09.780337 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/vault-token"
...
2021/04/22 13:23:09.788148 [DEBUG] (runner) watching 2 dependencies
2021/04/22 13:23:09.788163 [DEBUG] (runner) all templates rendered
2021/04/22 13:23:09.788170 [DEBUG] (runner) enabling template-specific quiescence for "e4e7973b3ae717cf947a96e1944f88ee"
2021/04/22 13:23:09.788185 [DEBUG] (cli) receiving signal "child exited"
2021/04/22 13:23:09.907646 [TRACE] vault.write(auth/token/create -> 2ded95de): successfully renewed

This approach originally came from a ConfigMap spec from BanzaiCloud: https://banzaicloud.com/docs/bank-vaults/mutating-webhook/howto-ct-prom/#configmap

Note that some parts of the linked ConfigMap spec are actually outdated, and don't work with consul-template 0.24.1, specifically the use of grace = "5m"; in any case, their use of .Auth.ClientToken was the critical change to get this working.

Hopefully this helps someone out!

[jsanant]

@davidrosson - Thank you so much! Was waiting for this feature.

A small change I noticed when creating the token, the policy parameter should be policies.

{{with secret "/auth/token/create" "policies=policy_1" "no_default_policy=true"}}{{.Auth.ClientToken}}{{ end }}

When I used policy it would attach the root policy to the token.

[eikenb]

Thanks for all the updates and for figuring out secret works @davidrosson!

I apologize for not catching the meaning of this on my initial reading, I was trying to sort of keep up while doing other work and didn't spend the appropriate time reviewing.

So as this is supported, I'm thinking that just adding an example to the documentation for secret might suffice. Thoughts?

[eikenb]

Hey @davidrosson... @jsanant reported that the policy parameter needed to be policies=.. not policy=.. and the documentation I checked agreed. I wanted to double check that this was a typo and not that I'm missing something. Thanks.

[eikenb]

I'm going to go ahead and update the docs and close this. Please let me know if my assumptions about the policy/policies pick was wrong. Thanks.