Consul-template: Unable to write to KVv2

[kasunt-nixdev]

Using consul-template v0.21.0 (0d999b32)

I have the following consul-template configuration following the doc.

vault {
  ssl {
    ca_cert = "tls/ca.pem"
  }
  retry {
    backoff = "1s"
  }
}
template {
  {{ secret "secret/data/test/admin" "password=test" }}
}

However, it keeps throwing the error as if there is no request body.

[WARN] (view) vault.write(secret/data/test/admin -> 48e87fe5): vault.write(secret/data/test/admin -> 48e87fe5): Error making API request.

URL: PUT http://<ip>/v1/secret/data/test/admin
Code: 400. Errors:

* no data provided (retry attempt 1 after "250ms")

I thought the format is {{ secret "<PATH>" "<DATA>" }} but is it not the case ?
Just as a suggestion it would be good to have an example in the doc too.

[eikenb]

Reproduced this with just a template containing...

{{ secret "secret/data/test/admin" "password=test" }}

And ...

$ consul-template -template=template.tmpl
2019/08/09 23:48:39.199330 [WARN] (view) vault.write(secret/data/test/admin -> 48e87fe5): vault.write(secret/data/test/admin -> 48e87fe5): Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/data/test/admin
Code: 400. Errors:

* no data provided (retry attempt 1 after "250ms")
2019/08/09 23:48:39.451967 [WARN] (view) vault.write(secret/data/test/admin -> 48e87fe5): vault.write(secret/data/test/admin -> 48e87fe5): Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/data/test/admin
Code: 400. Errors:

* no data provided (retry attempt 2 after "500ms")
[...]

[kasunt-nixdev]

@eikenb Thanks for looking into it. I debugged it a little further.

It seems though as the Vault api expects the format,

{
    "data": { 
        "password":"abcd"
    }
}

However, what consul-template attempts to push is {"password":"abcd"} when the following template is used,
{{ secret "secret/data/test/admin" "password=abcd" }}

[kasunt-nixdev]

@eikenb if it helps was able to patch it locally on my end with,

dataNew := make(map[string]interface{})
dataNew["data"] = d.data
vaultSecret, err := clients.Vault().Logical().Write(d.path, dataNew)

[eikenb]

Thanks for reporting it and helping out. Your change fixes it. I just want to write up a test for this then I'll get a PR pushed. Though feel free to speak up if you'd like to do it.

[kasunt-nixdev]

@eikenb nope go ahead. Im happy to test your changes once you cut a release.

[eikenb]

There was a twist. KVv1 vs KVv2... the old syntax works for KVv1 where KVv2 needs the update. Still OK, just means it will take a bit more time to make sure it works for both.

[kasunt-nixdev]

Ahh. I had not tested KVv1. Totally forgot that only v2 requires to be wrapped around data key. That's good to know.

[kasunt-nixdev]

@eikenb Also will need to consider secrets backend plugins that may depend on this.

[eikenb]

I think we found a corner of the app that doesn't see a lot of use. This secrets writing functionality has been broken for a while. For KVv1 it panics due to a nil pointer deref and KVv2 does nothing except that error. The write tests that use/check it write against the function and call with a data structure already setup for KVv2 when needed, where the template secret write call doesn't do this.

Anyways... so I have the code fixed and those tests fixed, but there is no test yet for the template itself (which I want, to make sure the template's use of the back-end is tested).

I don't think this would impact plugins. This only changes the behavior of the one internal function to send the data to vault in the correct format for that version.

The only external ramification I can see is that writes to KVv1 stores don't reply with the data as KVv2 ones do, so using with secret path key=value wouldn't have any values set from the with. But it used to panic, so I don't think that's breaking backwards compatibility.

[rckjacobs1]

Is there an expected time frame for the fix to be available. Thanks for all your efforts.

[eikenb]

@rckjacobs1 I'm hoping to have it ready in the next couple days. Having it in a release will take a while longer.

[rckjacobs1]

Downloaded the 3 updated files vault_read_test.go, vault_write.go, vault_write_test.go
Executed the following template
}
template {
{{ secret "secret/data/testuser" "password=test" }}
}
ERROR: * no data provided

Question: is there any other action required besides downloading and deloyng the 3 files?

[eikenb]

Hey @rckjacobs1.

Theoretically yes... the changes were in those 3 files.

Why did you download just the 3 files? Are you trying to merge the change into an older version? If so, which version? Seems like it must be something with the version in use as,
for me, using current master it works fine with that template.

[rckjacobs1]

Used the current master as you recommended and it works fine. Thanks for your quick response.