Jq: [newbie questions] JSON objects with single quotation mark, and keys with spaces

[starrysl]

Hi. I am new to jq. It is quite a useful tool. But I have two questions:

1. some of my json raw files use single quotation mark (') instead of double quotes ("). Is it possible to specify to use single quote in jq?
   e.g. {[u'foo': {u'bar, u'bat}]}
2. some of my keys contains spaces. How can I retrieve a value according to keys with spaces?
   e.g. {'user name': 'david'}

Thank you for your attention!

[ghost]

On your first question:

{[u'foo': {u'bar, u'bat}]}

That's not valid JSON! Even taking out the square brackets and fixing the single quotes (that is, sanitising it to {u'foo': [u'bar', u'bat']}), I'm assuming that's the output of some Python script which dumps its data structures instead of converting them first to JSON. While JSON values are valid Python dictionaries and objects (null being the exception), this doesn't hold the other way around. Namely, Python's unicode literals are not valid JSON, and neither are single quotes.

Here's a nice little Python script to convert from Python's JSON-looking data structures to real JSON. You can pipe to it or call it with filenames as arguments.

import json
import sys

def dump(s):
    print json.dumps(eval(s))

def main(args):
    if not args:
        dump(''.join(sys.stdin.readlines()))
    else:
        for arg in args:
            dump(''.join(open(arg, 'r').readlines()))
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Keep in mind that this script uses Python's eval for all its heavy work, so you shouldn't run it over untrusted input.

On your second question:

How can I retrieve a value according to keys with spaces? e.g. {'user name': 'david'}

.foo and .["foo"] are equivalents, the latter accepting spaces and other extraneous characters.

echo '{"user name": "david"}' | jq '.["user name"]' 
"david"

[Scringo]

I have a followup question:
What if my structure is: '{"data" : {"user name": "david"}}' ?
Doing:
jq '.data.["user name"]'
produced the following error:
error: syntax error, unexpected '[', expecting IDENT
.data.["user name"]
^
1 compile error

[starrysl]

Thank you @slapresta for your comments. I did get the "json" from a python dump (which I thought was also json...). Thank you for the script to parse it to real json.

I have the same problem as @Scringo mentioned. I cannot get the value according to a key defined in a secondary structure.

[starrysl]

@Scringo
it seems this works:

echo  '{"data" : {"user name": "david"}}' | jq .data | jq '.["user name"]' 

[Scringo]

This looks like a nice workaround which I might use, thanks @starrysl
However, it means invoking jq twice and in case of large data structure it might cost.
I wonder if this is a bug or the way it is intended to work ...

[starrysl]

@Scringo
I agree that calling jq once may be efficient. Let's wait for more reply.

[lluchs]

It's just like object member access in JavaScript, you either do object.member or object["member"], i.e. no dot before the [].

I guess the manual doesn't describe this clearly enough, as this question seems to come up quite often...

Also note that the pipe can be used in jq to combine filters:

echo  '{"data" : {"user name": "david"}}' | jq '.data | .["user name"]'
# is the same as
echo  '{"data" : {"user name": "david"}}' | jq '.data["user name"]'

[Scringo]

Works as you described. Thanks !!

[ghost]

@starrysl If you "control" the Python script, you can just pass the data structure through json.dumps first (remember to import json!) and then print it.

[nicowilliams]

I have a followup question:
What if my structure is: '{"data" : {"user name": "david"}}' ?
Doing:
jq '.data.["user name"]'
produced the following error:
error: syntax error, unexpected '[', expecting IDENT
.data.["user name"]
^
1 compile error

Use:

jq '.data["user name"]'

See the difference? You had an extra '.'.

[nicowilliams]

Also, you can use .data."user name".

This is becoming an FAQ, and I'm wondering if the syntax shouldn't just allow .data.["user name"] just to avoid this being an FAQ... It'd be a trivial change to parser.y to allow it...

[nicowilliams]

Another way in which Python literals aren't JSON: Python dicts allow the use of tuples and other non-string types as keys, but JSON objects only allows strings as names (keys).

[nicowilliams]

Another possibility is to add an error rule to parser.y so that .foo.["bar"] results in an error telling the user to try .foo["bar"]. I think I'll implement that.

[nicowilliams]

Another possibility is to add an error rule to parser.y so that .foo.["bar"] results in an error telling the user to try .foo["bar"]. I think I'll implement that.