consul-template incompatible with KV Secrets Engine Version 2; has misleading and harmful failure message.

[apeschel]

consul-template does not appear to work with the Hashicorp Vault KV Secrets Engine version 2. While not necessarily a significant problem, there are several related problems that cause this to be extremely troublesome:

1. Starting a vault locally via -dev automatically puts the KV Secrets Engine into version 2.
2. The error message generated by consul-template is extremely misleading, and in no way indicates a version incompatibility.
3. The trace and debug logs give no insight whatsoever as to a version incompatibility.
4. There does not appear to be any documentation whatsoever declaring that consul-template is not compatible with KV Secrets Engine Version 2.

As a new user of Vault and consul-template, these problems have caused me to waste a considerable amount of time and have lead to a very poor initial impression with both tools. Simply having some relevant failure messages would have greatly improved my experience.

Consul Template version

consul-template v0.19.5 (f8c8205)

Installed via Homebrew

Configuration

No custom consul-template configuration.

Command

consul-template -vault-renew-token=false -template "in.tpl:out.txt" -once -dry

Debug output

https://gist.github.com/apeschel/8ea15fb79836a3a428cd3ec84f55dce7

Expected behavior

consul-template should connect to vault running KV Secrets Engine version 2, or at least generate a relevant version incompatibility error message.

Actual behavior

consul-template generates a misleading "key not found" message.

Steps to reproduce

Start a vault with the given config:

backend "inmem" { }

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

disable_mlock = true

vault server -config vault.conf

Create a vault, login, and add a secret:

vault operator init -key-shares=5 -key-threshold=2
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=XXX
vault unseal
vault unseal
vault kv put secret/my-application [email protected]

Create a simple template:

{{ with secret "secret/my-application" -}}
email={{ .Data.email }}
{{ end -}}

Output generated as expected:

consul-template -vault-renew-token=false -template "in.tpl:out.txt" -once -dry
> out.txt
[email protected]

Upgrade to KV Secrets Engine version 2:

vault kv enable-versioning secret/

consul-template is no longer able to connect to the vault:

consul-template -vault-renew-token=false -template "in.tpl:out.txt" -once -dry
2018/10/15 18:28:25.982033 [WARN] (view) vault.read(secret/my-application): no secret exists at secret/my-application (retry attempt 1 after "250ms")
2018/10/15 18:28:26.237407 [WARN] (view) vault.read(secret/my-application): no secret exists at secret/my-application (retry attempt 2 after "500ms")                                                                                                                                                                                                                       2018/10/15 18:28:26.743015 [WARN] (view) vault.read(secret/my-application): no secret exists at secret/my-application (retry attempt 3 after "1s")                                                                                                                                                                                                                          2018/10/15 18:28:27.744238 [WARN] (view) vault.read(secret/my-application): no secret exists at secret/my-application (retry attempt 4 after "2s")

[gmr]

Given a KV v2 secret backend of team/foo, a path of applications/bar, and key of password the following worked for me:

{{ with secret "team/foo/data/applications/bar" }}
Password: {{ .Data.data.password }}
{{ end }}

[apeschel]

@gmr Can you provide a full walkthrough on that example, please? I am not very familiar enough with this to fill in the gaps, and can't successfully recreate it. Thanks.

[gmr]

So if you call your kv v2 secret backend top-secret, add a kv pair at the path of credentials, with the key password you'd use the syntax:

{{ with secret "top-secret/data/credentials" }}
Password: {{ .Data.data.password }}
{{ end }}

Note the addition of data which consul-template (and vault in return) is expecting in the path to get it to work, and the use of .Data.data.password to get at the password value. Make sense?

[jleechp-occm]

To expand slightly on the above. If you're trying to enumerate secrets you have to use:

{{ range secrets "top-secret/metadata/credentials }}
...
{{ end }}

[AlKapkone]

@gmr Hello, Gavin. I tried to use your recommendations, but still doesn't working..

[AlKapkone]

Does anyone has any updates on this ?

[kjanshair]

@gmr I tried this and it works on KV v2 but doesn't work with KV v1.

[gmr]

@kjanshair I would not expect it to. The paths are different. That usage is documented in the consul-template README IIRC.

[kjanshair]

@gmr I see:

{{ with secret "secret/passwords" }}
{{ .Data.wifi }}{{ end }}

For KV v1, but does not updating.

[kjanshair]

Complete issue description. #1199