Consul-template: Support for vault PKI?

[issacg]

I'd like to use consul-template together with Vault's PKI backend.

I'm not quite sure how to go about it for 2 reasons:
1) Firstly, /pki/issue is a POST operation, so I think consul-template would need to use vault.Logical().Write(), right? (actually even Write() I'm not sure about; Write() does PUT, and we need POST, right?)
2) Even if I did this, I'd be unable to do this in one run - the call to /pki/issue is the only time I'd get the private key, so it (key) would need to be stored immediately to its own entire file; I could then get the public key with another to /pki/cert/ - except that I don't have the serial since the call to /pki/issue would (presumably) only save the private key

It doesn't seem like this is so far-fetched, and I could make my own script to POST to vault and get the keys, but wondering if it makes sense for consul-template to be able to do it, rather than re-inventing the wheel (at least parially :))

[sethvargo]

Hi @issacg

I don't know if Consul Template is the right tool for working with the Vault PKI backend. What advantage are you getting by using Consul Template instead of just using the Vault CLI directly?

[issacg]

Given the lack of lease renewal (https://github.com/hashicorp/vault/issues/877) I'm no longer sure there is. I was going to say the fact that consul-template will keep the keypair up-to-date and restart related services. I still need to do so, but if Vault isn't giving me a renewable lease, that's something else that I'd need to hack around anyway, so maybe it's just smarter to write my own tool...

[sethvargo]

@issacg sounds like a good new open source tool :smile:

[issacg]

Here's a start: https://github.com/issacg/vault-pki-client

[jefferai]

@sethvargo Reopening this after a customer request. The PKI backend returns a standard Response object so there's no real reason you couldn't make a call into it and get a cert out, and in fact they would very much like to do exactly this. It seems like there are currently two impediments:

1. consul-template needs to allow path parameters to be passed in. It's possible it already does this; I'm just looking at the README with this example:

{{with vault "secret/passwords"}}{{.Data.password}}{{end}}

You'd need to be able to do something like

> {{with vault "pki/issue common_name=host.example.com format=pem_bundle"}}{{.Data.pem_bundle}}{{end}}

Is this supported, and if not, can it be?

1. The PKI backend in Vault needs to be able to return a private key, cert, and issuing CA as a PEM bundle. This is very easy to add in as an option, as it already supports a format parameter that is pem or der. So it could be a format of pem_bundle and the concatenated bundle could be in a response parameter.

I could get (2) into 0.5 if we think (1) is doable.

Thoughts?

[sethvargo]

Hey @jefferai

That's certainly _possible_, but I don't know how to pass in those extra args to the API: https://github.com/hashicorp/vault/blob/2d13b41fef4839511d5dd24d5ec597899af08d91/api/logical.go#L13-L26. Any ideas?

[enkaskal]

@sethvargo since you said _any_ ideas, and given that I recently ran into a similar situation, I thought I'd throw in my $0.02 :)

First, the pki secret backend doesn't issue via Read, but rather Write

I took a stab at adding support based on @jefferai's previous comment which includes format=pem_bundle although it should be noted that one should use {{ .Data.certificate }} not {{ .Data.pem_bundle }} based on the /pki/issue api; please see: https://github.com/enkaskal/consul-template/commit/8e166e9a040252d6afb30839939fe0bec8a53fd9

N.B. Be warned this is _literally_ the first Go code I've ever written so please consider it PoC at best and am not responsible if your eye's fall out or explode ;)

Further, I believe that consul-template handles the renewals (reissue in this case).

However, I run into issues when I want to chain the public certs and have a separate private key. Some examples of this are NGINX, Apache HTTPd, and even Vault itself (tls_cert_file/tls_key_file) when run in server mode. I've solved this via a plugin for consul-template to get me started, but it's kinda hackish and I'm finding myself wanting meta-consul-templates. Which begs the question, perhaps I'm missing a more elegant solution?

I welcome any questions, comments, or thoughts.

[willusher]

I built consul-template with @enkaskal 's patch, and it worked well. I can't comment on the code quality (having never written any Go), but the functionality is there.

This is an important feature for us. @sethvargo / @jefferai - is there anything I can do to help get this feature merged?

[enkaskal]

@willusher glad you found it useful :)

@sethvargo / @jefferai I'm happy to do any cleanup or add any tests required for a successful PR.

[automaticgiant]

i'm not sure this feature is finished. as stated in https://groups.google.com/d/msg/vault-tool/U2ZPpSgBQno/CNJATMvZAwAJ , there is problem when you need to have a cert issued but you need to place the private key and the cert in different files. i thought maybe we could get around it with scratch and/or rendertemplate, but neither facilitate splitting up the pki secret payload between multiple files. so i really like that consul-template supports vault pki, but i think that in a significant portion of use cases, it doesn't really.
i'm eyeballing a consul-template plugin with side-effects right now. if you have better ideas, let me know! not really sure what the best way to handle this in c-t is. maybe an inverse of the file function to write?
scratch sounded good, except for the whole not being shared between invocations/templates bit.

maybe i do have a solution, at least to the common problem of needing to place pairs of cert/privkey. i guess teh simplest way to solve it is to have a separate template for the issue endpoint with a command that takes the values as stdin or something.... nope - can't use retrieved values as arguments, so i guess i would want to place the big bundle file, then exec a script to unpack it and delete the bundle. meh. ok.

[sirlatrom]

@automaticgiant Just supply something along the lines of -template "/tmp/bundle.ctmpl:/tmp/bundle.pem:openssl x509 -pubkey -noout -in bundle.pem > pubkey.pem" with bundle.ctmpl something like this:

{{- with secret "pki/issue/my-domain-dot-com" "ttl=72h" "common_name=foo.example.com" "format=pem_bundle" }}
{{- .Data.certificate }}
{{- end }}

Or perform whatever openssl or other command you want whenever the cert bundle is issued.

[automaticgiant]

what i ended up with was:

template {
  contents = <<EOF
#!/bin/sh
echo $0 placing tls stuff in $PWD
{{ with secret "pki/platform/issue/component-server" "common_name=server.sdp" }}
cat > ca.cert <<EOS
{{ .Data.issuing_ca }}
EOS

cat > server.cert <<EOS
{{ .Data.certificate }}

cat > server.key <<EOS
{{ .Data.private_key }}
{{ end }}
EOS

rm -f $0
EOF

  destination = "place_bundle.sh"

  exec {
    command = "sh place_bundle.sh"
  }
}

so that does what i need it to do, i guess. i do think that there is still a bit of a gap for using the vault pki. we still have to do the placement by calling something outside, so :( .

i guess the worst part is the documentation. i don't want to start this pattern(there's probably something better), but there has to be a way we can document the vault support better to show how one might use it with the pki backend, right? as a c-t noob, i've spent like an hourish figuring this out. and it's still not going to help me do the same thing with the template functionality in nomad(which i am assuming is just using all/most the c-t code, so adding to support here should transfer...).

@sirlatrom - why the - in {{- blah ?

[leprechau]

@automaticgiant That's whitespace control within golangs text/template language.

https://golang.org/pkg/text/template/#hdr-Text_and_spaces

[issacg]

@automaticgiant if you're not against not using hashicorp's tooling, the original alternative tool I wrote (https://github.com/issacg/vault-pki-client) doesn't bundle the cert and key together

[automaticgiant]

not against non-hashicorp, but the node dep kindof takes the wind out of my sails. loves me some static golang binaries, hashicorp or not. there is a node app i might use it with, but not sure yet. happy to have the choice though @issacg

[calvn]

@issacg @automaticgiant I haven't used it myself, but you could give zeit/pkg a try to turn the node app into a static binary for distribution/use.

[issacg]

Actually, it's already packaged for some platforms as a static binary at https://github.com/issacg/vault-pki-client/releases and it's pretty trivial to roll your own (I use https://github.com/nexe/nexe)