Consul-template: 'secrets' query does not renders out.txt

[Dentrax]

Consul Template version

consul-template v0.25.2 (870905d)
Vault v1.7.0 ('4e222b85c40a810b74400ee3c54449479e32bb9f+CHANGES')

Configuration

in.tpl

{{ range secrets "secret/" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/" }}
{{ with secret (printf "secret/%s" .) }}{{ range $k, $v := .Data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

Command

$ vault server -dev -dev-root-token-id=root -log-level=trace

P.S: I can not see any requests logs here after run consul-template CLI.

Debug output

https://paste.ubuntu.com/p/SqTPQGJm2W/

Expected behavior

It should render:

bar
foo
zip

Actual behavior

$ cat out.txt => EMPTY

Steps to reproduce

1. $ vault server -dev -dev-root-token-id=root -log-level=trace
2. $ vault kv put secret/foo bar=baz
3. $ vault kv list secret/

Keys
----
foo
qux

1. $ consul-template -template "in.tpl:out.txt" -vault-renew-token=false -vault-token='root' -once => OK
2. $ cat out.txt => EMPTY

Trace logs:

[INFO] consul-template v0.25.2 (870905d)
[INFO] (runner) creating new runner (dry: false, once: true)
[DEBUG] (runner) final config: {"Consul":{"Address":"","Namespace":"","Auth":{"Enabled":false,"Username":"","Password":""},"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":false,"Key":"","ServerName":"","Verify":true},"Token":"","Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000}},"Dedup":{"Enabled":false,"MaxStale":2000000000,"Prefix":"consul-template/dedup/","TTL":15000000000,"BlockQueryWaitTime":60000000000},"DefaultDelims":{"Left":null,"Right":null},"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":0},"KillSignal":2,"LogLevel":"debug","MaxStale":2000000000,"PidFile":"","ReloadSignal":1,"Syslog":{"Enabled":false,"Facility":"LOCAL0","Name":"consul-template"},"Templates":[{"Backup":false,"Command":"","CommandTimeout":30000000000,"Contents":"","CreateDestDirs":true,"Destination":"out.txt","ErrMissingKey":false,"Exec":{"Command":"","Enabled":false,"Env":{"Denylist":[],"Custom":[],"Pristine":false,"Allowlist":[]},"KillSignal":2,"KillTimeout":30000000000,"ReloadSignal":null,"Splay":0,"Timeout":30000000000},"Perms":0,"Source":"in2.tpl","Wait":{"Enabled":false,"Min":0,"Max":0},"LeftDelim":"","RightDelim":"","FunctionDenylist":[],"SandboxPath":""}],"Vault":{"Address":"http://127.0.0.1:8200","Enabled":true,"Namespace":"","RenewToken":false,"Retry":{"Attempts":12,"Backoff":250000000,"MaxBackoff":60000000000,"Enabled":true},"SSL":{"CaCert":"","CaPath":"","Cert":"","Enabled":true,"Key":"","ServerName":"","Verify":true},"Transport":{"DialKeepAlive":30000000000,"DialTimeout":30000000000,"DisableKeepAlives":false,"IdleConnTimeout":90000000000,"MaxIdleConns":100,"MaxIdleConnsPerHost":13,"TLSHandshakeTimeout":10000000000},"UnwrapToken":false},"Wait":{"Enabled":false,"Min":null,"Max":null},"Once":true,"BlockQueryWaitTime":60000000000}
[INFO] (runner) creating watcher
[INFO] (runner) starting
[DEBUG] (runner) running initial templates
[DEBUG] (runner) initiating run
[DEBUG] (runner) checking template c23f0a4c34465b4756b6e34817ef0d6b
[DEBUG] (runner) missing data for 1 dependencies
[DEBUG] (runner) missing dependency: vault.list(secret)
[DEBUG] (runner) add used dependency vault.list(secret) to missing since isLeader but do not have a watcher
[DEBUG] (runner) was not watching 1 dependencies
[DEBUG] (watcher) adding vault.list(secret)
[DEBUG] (runner) diffing and updating dependencies
[DEBUG] (runner) watching 1 dependencies
[DEBUG] (runner) receiving dependency vault.list(secret)
[DEBUG] (runner) initiating run
[DEBUG] (runner) checking template c23f0a4c34465b4756b6e34817ef0d6b
[DEBUG] (cli) receiving signal "urgent I/O condition"
[DEBUG] (runner) rendering "in2.tpl" => "out.txt"
[DEBUG] (runner) diffing and updating dependencies
[DEBUG] (runner) vault.list(secret) is still needed
[DEBUG] (runner) watching 1 dependencies
[DEBUG] (runner) all templates rendered
[INFO] (runner) once mode and all templates rendered
[INFO] (runner) stopping
[DEBUG] (runner) stopping watcher
[DEBUG] (watcher) stopping all views

References

• https://github.com/hashicorp/consul-template/blob/master/docs/templating-language.md#secrets

_cc:_ @developer-guy

[yilmazo]

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

[eikenb]

1468 is merged and fixes this.

[rishianand06]

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

I think this is related to KV-v2 secrets engine. adding 'metadata' to secret path worked for me.

in.tpl

{{ range secrets "secret/metadata" }}
{{ . }}{{ end }}

in2.tpl

{{ range secrets "secret/metadata" }}
{{ with secret (printf "secret/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

My Environment -
consul-template v0.25.2 (c8372a0a)
Vault v1.7.0-rc1 (9af08a1c5f0f855984a1fa56d236675d167f578e)

Put secrets in vault -
vault kv put secret/tools/tool1/token/acc1 key1='vaule1'
vault kv put secret/tools/tool1/token/acc1 key2='vaule1'

vault kv put secret/tools/tool1/token/acc2 key3='vaule2'

Vault cli command output -
vault kv list secret/tools/tool1/token

Keys

acc1
acc2

vault kv get secret/tools/tool1/token/acc1
============= Data =============
Key Value
--- -----
key1 value1
key2 value1

vault kv get secret/tools/tool1/token/acc2
============= Data =============
Key Value
--- -----
key3 value2

Consul templates -
Now I need to write consul templates to read K/V pairs for each account (acc1, acc2,.....).
Tried to do the same way, as you told.

in.tpl
{{ range secrets "secret/tools/tool1/token/metadata" }}
{{ with secret (printf "secret/tools/tool1/token/data/%s" .) }}{{ range $k, $v := .Data.data }}
{{ $k }}: {{ $v }}
{{ end }}{{ end }}{{ end }}

Run
consul-template -template "in.tpl:out.txt" -once

Actual behavior
$ cat out.txt => EMPTY

Expected behavior
It should render:

key1 value1
key2 value1
key3 value2

Please let me know, if I am missing something.

Thanks

[eikenb]

Hey @rishianand06,

Have you tried with a build from the master branch yet, this should be fixed there. It (the fix) will also be included in version 0.26.0, which I'll be releasing sometime in the next week or two.

If this was tried on the master branch and it didn't fix it for you please let me know. Thanks.

[rishianand06]

@eikenb No I haven't tried that yet. Will try and let you know in case of any issue/s.
Thanks :)