Jq: Failure to parse .x.["y.z"] notation

[mcint]

mcint@work MINGW64 /c/git

The document

$ less test.json | jq -c '.'
{"a":"A","b":{"c":{"d.e.f":"DEF"}}}

The complex key

$ less test.json | jq -c '.b.c'
{"d.e.f":"DEF"}

One naive approach

$ less test.json | jq -c '.b.c.d.e.f'
null

Solution proposed by the docs

$ less test.json | jq -c '.b.c.["d.e.f"]'
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Windows cmd shell quoting issues?) at <top-level>, line 1:
.b.c.["d.e.f"]
jq: 1 compile error

Work-around

$ less test.json | jq -c '.b.c | .["d.e.f"]'
"DEF"

Boundaries of quoted key solution

$ less test.json | jq -c '.b | .c.["d.e.f"]'
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Windows cmd shell quoting issues?) at <top-level>, line 1:
.b | .c.["d.e.f"]
jq: 1 compile error

[mcint]

Trials with 2-part ("z.y") versus 3-part ("z.y.x") key

$ less test2.json | jq -c '.b.c'
{"d.e": "DEF"}

$ less test2.json | jq -'c .b.c.d.e'
null

$ less test2.json | jq -c '.b.c.["d.e"]'
jq: error: syntax error, unexpected '[', expecting FORMAT or QQSTRING_START (Windows cmd shell quoting issues?) at <top-level>, line 1:
.b.c.["d.e"]
jq: 1 compile error

[ghost]

The syntax is either .foo (for keys without extraneous characters) or ["foo.bar"] (note the absence of a trailing dot). That is:

jq '.b | .c["d.e.f"]'
jq '.b.c["d.e"]'

So, you may be wondering: how come jq '.b.c | .["d.e.f"]' does work? The answer lies in that the output of the previous filter is represented by .. Therefore, .["d.e.f"] performs ["d.e.f"] over .. The dot can be elided when using .foo as the first filter: that's why it's .b.c and not ..b.c.

[mcint]

Great, that completely addresses my question, thank you. Good to close.

For the sake of completeness, it looks like the dot _must_ be elided.

$ less test2.json | jq '..b.c'
jq: error: syntax error, unexpected IDENT, expecting $end (Windows cmd shell quoting issues?) at <top-level>, line 1:
..b.c
jq: 1 compile error

[ghost]

Yes, that is correct: the dot must be elided. I misspoke in my earlier comment.

[rogpeppe]

This seems to be a documentation issue. The reference documentation
describes the operator as .[0] and says:

When the index value is an integer, .[<value>] can index arrays

This appears to be true only if the dot is at the start. There
are no examples of an array index not at the start, and no
indications that I've found that the dot is not allowed in that
case.

[wtlangford]

I'll agree that it's a documentation issue, but the real issue is that we
may not be properly explaining what the dot is doing there. The dot means
"the input", and then you're indexing with [], so subsequent indexes in a
chain don't need to have a dot, since they're indexing off of the result.
In fact, as you've noticed, using the dot results in a syntax error. :)

On Thu, May 11, 2017, 07:31 Roger Peppe [e-mail] wrote:

This seems to be a documentation issue. The reference documentation
describes the operator as .[0] and says:

When the index value is an integer, .[] can index arrays

This appears to be true only if the dot is at the start. There
are no examples of an array index not at the start, and no
indications that I've found that the dot is not allowed in that
case.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/stedolan/jq/issues/1168#issuecomment-300762225, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADQ4V0NFWrbe-Lw4XpL8ivJLHBBeqoSWks5r4vGBgaJpZM4I7A3Y
.

[nicowilliams]

My question is: should we add .a.["b"]? I'm tempted to say "no", but this does come up frequently enough that maybe it'd just be for the best. What do we lose if we add this?

[MondayHopscotch]

This is something I recently ran into, as well. Simplified examples using JQPlay:

With dot (syntax error):
https://jqplay.org/s/itgHiQlNMP

Without dot (happy):
https://jqplay.org/s/AWemvC2Vsv

Is this something that there is a concrete reason behind disallowing? Or is it just a limitation of the current implementation?

Wording my question another way: Is there any reason that supporting this in the future would be explicitly unwanted?

[nicowilliams]

@MondayHopscotch There is no limitation of the current implementation that makes this difficult to support.

As to reasons not to support this, I don't think there are any, and this is a very very FAQ.

This trivial patch adds support for this:

diff --git a/src/parser.y b/src/parser.y
index 6255850..39c7836 100644
--- a/src/parser.y
+++ b/src/parser.y
@@ -685,6 +685,12 @@ Term '[' Exp ']' '?' {
 Term '[' Exp ']' %prec NONOPT {
   $$ = gen_index($1, $3);
 } |
+Term '.' '[' Exp ']' '?' {
+  $$ = gen_index_opt($1, $4);
+} |
+Term '.' '[' Exp ']' %prec NONOPT {
+  $$ = gen_index($1, $4);
+} |
 Term '[' ']' '?' {
   $$ = block_join($1, gen_op_simple(EACH_OPT));
 } |

$ ./jq -cn '{a:{b:{c:"d"}}}|.a.["b"]'
{"c":"d"}
$ 

[nicowilliams]

@wtlangford any objections to pushing that patch?

[rogpeppe]

The syntax is either .foo (for keys without extraneous characters) or ["foo.bar"] (note the absence of a trailing dot).

Perhaps the real issue here is that the documentation doesn't explain the grammar clearly. I can't see anything in the documentation that mentions that [] can be used as an operator at all.

Note that .a.b.c is the same as .a | .b | .c

AFAICS . is being used in two senses here - as an initial value (the first .) and then as a special case as a binary operator. Currently the filter documentation elides the operands for each operator. Perhaps if it didn't, then things might be more obvious.

One other thing: .x.y? works, and the docs document the .foo? form exactly the same was as .[<string>] so it's perhaps not surprising people get confused when .x.["<string>"] doesn't work.

[nicowilliams]

There is certainly a bug in the manual...

   For  example  .["foo::bar"]  and .["foo.bar"] work while .foo::bar does
   not, and .foo.bar means .["foo"].["bar"].

[nicowilliams]

@rogpeppe https://github.com/stedolan/jq/wiki/jq-Language-Description

[nicowilliams]

I've pushed 3e0a118 to allow '.foo.["bar"] syntax.