Consul: KV behavior inconsistent between keys/recurse/separator

[benfleis]

I have a keyspace of /foo/bar, with bar having a value baz. Like an ls command, I want to know the children of foo, but not necessarily the children's children. A straight up GET doesn't work, while using ?keys does. This is behaviourally inconsistent, as the docs for keys ambiguously state:

It is possible to also only list keys without their values by using the "?keys" query parameter along with a GET request.

Which suggests to me (as would also be my intuition), that calling _without_ keys should return the "direct" key subset, and with keys is as explicitly promised.

I understand that I can also achieve this via ?recurse&separator=/, but it feels strange to ask for, and then expressly limit, recursion, when a natural operation, with or without &keys feels right.

A simple curl log is pasted below to demonstrate. (And edited to add visual whitespace.) To be 100% clear, I would expect the 2nd query (/kv/foo/, without args) to return the same thing as the last.

core@dev /opt/sbin $ curl -XPUT --url "http://localhost:8500/v1/kv/foo/bar" -d "baz"
true
core@dev /opt/sbin $ curl -XGET --url "http://localhost:8500/v1/kv/foo/bar"
[{"CreateIndex":53,"ModifyIndex":53,"LockIndex":0,"Key":"foo/bar","Flags":0,"Value":"YmF6"}]
core@dev /opt/sbin $ curl -XGET --url "http://localhost:8500/v1/kv/foo/"
core@dev /opt/sbin $ curl -XGET --url "http://localhost:8500/v1/kv/foo/?keys"
["foo/bar"]
core@dev /opt/sbin $ curl -XGET --url "http://localhost:8500/v1/kv/foo/?recurse"
[{"CreateIndex":53,"ModifyIndex":53,"LockIndex":0,"Key":"foo/bar","Flags":0,"Value":"YmF6"}]
core@dev /opt/sbin $ curl -XGET --url "http://localhost:8500/v1/kv/foo/?recurse&separator=/"
[{"CreateIndex":53,"ModifyIndex":53,"LockIndex":0,"Key":"foo/bar","Flags":0,"Value":"YmF6"}]

[armon]

I'm not sure I'm clear what the issue is. A straight GET does a direct K/V lookup. ?recurse instead gets every key under a prefix. ?keys acts just like recurse, but returns the keys only. ?seperator is used to limit the depth of results.

[benfleis]

It is not clear to me from reading the docs that ?keys should act like recurse. My reading of it says that 'keys' should be an orthogonal JSON selector, not modify query behavior. The name doesn't give that impression either (as it would if called recurse-keys).

I also just realized that separator only applies to the keys command, not to the recurse. This is perhaps implied by the fact that they're in the same paragraph and discussed together, but wasn't particularly clear. It's another thing that feels orthogonal -- like a pre- or post- filter.

So I suppose the issue is clarity of documentation, combined with (what for me is) a counterintuitive feel for the names/behaviors. Make sense?

[armon]

Got it. I'll leave this ticket open until we update the docs for clarity. Thanks!

[benfleis]

I would ask also that in addition to documentation, &separator be added to the &recurse functionality as well :)

[armon]

I'll take a look, that shouldn't be too hard to support.

[jalaziz]

Any update on this? Seems odd that the &keys supports &separator for limiting, but &recurse doesn't.

[rocktavious]

+1 on adding &separator to &recurse - i would like to use this functionality too

[SH4DY]

Any news on this enhancement? Trying to GET only direct children. This would perform much better compared to retrieving all key-names of a node and then GET every node individually or as a txn.

[godefroi]

Late to the game, but as an inexperienced user, there are several behaviors here that feel strange:

• Recursion is on by default when specifying ?keys. This isn't the default in, for example, ls, and it feels unnatural here.
• When using ?keys, &recurse=false doesn't do anything. Given that recursion is on by default, this would be the most obvious way to turn it off.
• When using ?keys, recursion is turned off with &separator=/. This option isn't really clearly documented (what does it do, really?), and it's not obvious at all that this is how one would turn off recursion.

[nicolasgarnil]

@godefroi great explanation!
BTW, do you know if it is possible to select a subset of keys instead of an entire subtree?

[godefroi]

@nicolasgarnil I don't know, but I suspect not.

[i0rek]

Thank you for reporting! I understand that this behaviour is a a little confusing and the functionality is not fully fleshed out.
Right now our focus are on other areas of Consul and we don't have time to work on improving these APIs. I would be happy to review a PR if someone wants to provide one, that implements adds this. Having that said I am closing this issue now because for us there is no positive side in keeping it open.

Thanks.