Consul: Asymmetric base64 encoding seems silly

[mattwilliamson]

What is the use-case where someone would want automatically base64 encoded values, but not automatically decode? If I wanted base64 to support binary blobs or whatever, I would just encode when putting. Maybe there could be a header passed to bypass this or a url parameter?

[highlyunavailable]

Are you talking about the KV store?

I can't speak to use cases, but the reason this is the case AFAIK is that you can pass in arbitrary bytes to the KV store to PUT, but since there's a bunch of data coming back as JSON (e.g. the LastIndex) as well as the data, the arbitrary bytes must be encoded before being returned since they're coming back inside a JSON object rather than just being fetched directly, even though they were put directly.

[mattwilliamson]

Yes I am. Is there a way to get the plaintext instead of JSON back? I'm trying consul out because etcd clustering is way too brittle to be used by a docker orchestration service, like rancher or coreos.

[highlyunavailable]

Unfortunately, no. https://www.consul.io/docs/agent/http/kv.html has more details on what the KV endpoint can do, but there isn't a way to just get plain text. Maybe the Hashicorp guys have plans to do a plain bytes version in v2 of the API but right now you're going to have to do some base64 decoding if you want to use the endpoint.

[mattwilliamson]

Alright. Consul sounds pretty useless for normal bash/devops script use cases. Thanks.

[highlyunavailable]

Not really? I have used Consul pretty extensively in scripting for "devops" tasks, it just needs to either be used via a language that has JSON support (e.g. Python) or the output needs to go through jq and in this case base64 -d before you use it in a bash script. A lot of the stuff you'd do in etcd via key/values is a first class citizen in Consul too, don't discount that - e.g. service discovery doesn't need JSON built/maintained, you can just register services with the local Consul agent.

[mattwilliamson]

Sorry just sounds super convoluted when all I care about is k/v

[slackpad]

Hi @mattwilliamson, @highlyunavailable is spot on in his comments. There are lots of folks using Consul KV from scripts as he described. Here's a working example:

echo $(curl -s http://demo.consul.io/v1/kv/global/time | jq -r '.[0] .Value' | base64 -D)

Consul does this so there's no way to break the JSON formatting for the arbitrary values. Tools like consul-template and consul-cli also are pretty common in extracting keys and pave over this for you. His comment about lots of first class support for common KV use cases is also very apt. Instead of needing to make KV queries to look for services, for example, many applications can just use Consul's DNS interface and be configured simply with foo.service.consul to get a healthy instance.

We will take a look at a raw value (non-JSON) interface for the V2 API as an additional convenience.

Hope that helps!

[slackpad]

Thanks to @kikitux for pointing out that there is an existing raw interface for reading keys that I totally didn't know was there :-)

If the "?raw" query parameter is used with a non-recursive GET, the response is just the raw value of the key, without any encoding.

(from here)

So the example would just be:

echo $(curl -s http://demo.consul.io/v1/kv/global/time?raw)

Sorry about that!

[mattwilliamson]

Whoa ok. This can definitely help. Thanks!