Jq: Add an option to treat empty input as null input

[TylerRick]

Currently, if you provide an empty input, jq appears to not run your filter program at all, which can be very puzzling (see #1497 and #1142, for example).

There are times when it would be more useful to treat the empty input as a null value, so that your filter program actually gets a chance to run and you can therefore actually do something with the input (or lack thereof), react to the lack of input, and actually produce an output anyway — which currently isn't possible at all when the input is "empty".

Being able to produce an output value even when the input is empty means that you could even influence the exit code when using --exit-status (see https://github.com/stedolan/jq/issues/1497#issuecomment-372867248) — also something that you otherwise couldn't do (currently always exits with 0 but is supposed to exit with 4 (#1497)).

I propose adding a new --empty-input-as-null option to complement the familiar existing input options like --slurp and --null-input/-n. In fact, it would work identically to --null-input when the input was empty.

jq usually treats its input a stream of 0 or more JSON values (separated by whitespace) ... which is great when you do in fact have multiple values coming in. But oftentimes you'll have an input script that generates a single, well-defined JSON value, and it would be more useful and intuitive to treat jq's input as a single value as well in order to process that script's single output value.

This option basically gives you that: It allows users to treat the input as if it were a single JSON value, falling back to null if needed instead of skipping running your filter entirely and not giving you a chance to generate an output value.

Examples:

> command_that_may_produce_empty_input_or_json | jq --empty-input-as-null '.'
null

# same as: 
> jq -n '.'
null

This would be especially useful in combination with --exit-status.

> echo | jq --empty-input-as-null --exit-status '.'; echo $?
null
1

# same as: 
> jq -n --exit-status '.'; echo $?
null
1

jq <settings -e --empty-input-as-null ".some_setting == true" || do_something_else

Could use with // operator to provide your own fallback behavior/value if you want something other than null when there's empty input:

> echo | jq --empty-input-as-null --exit-status '. // true'; echo $?
true
0

# same as:
> jq -n --exit-status '. // true'; echo $?
true
0

[e1senh0rn]

Just encountered this with an empty file. If file is empty, filter is not processed at all:

touch file.json
jq '(. // {}) * {"new_key": "value"}' file.json

Is there any workaround?

[nocive]

bump

[pkoppstein]

@e1senh0rn asked:

is there any workaround?

What is the alternative behavior you have in mind? Would the -s option be useful for you? Or are you asking for new file-handing functions? Please be specific.

[nocive]

@pkoppstein maybe the report wasn't the most accurate but it seems to me that it still refers to the original (and quite detailed) report here.
I've come across this counter intuitive behaviour myself while using jq to parse all sorts of output from very common tools like gcloud or aws cli and the fact you can't use -e reliability because of this issue is a real pain and requires additional code and workarounds to cope with.

[stewartadam]

you can't use -e reliability because of this issue is a real pain and requires additional code and workarounds to cope with.

Exactly this, jq is often used to parse output from APIs and if the API returns no output (e.g. curl returned a 403 with no body due to missing token, or 503 due to service error) than jq proceeds happily along and we're none the wiser.

I think by far and wide, the user's expected behavior for -e is "exit on any false|null output" not "exit on any false|null output only if non-empty input". In fact, the docs don't even mention the non-empty quirk which makes it all the more confusing.

[nicpottier]

Just here to say this bit me as well, specifically when using jq to parse the output of an aws command that started failing. I get the purity argument that an empty string is valid JSON, but from a practicality perspective this sure would be a useful option.

[kolyshkin]

Let me rephrase the original issue; maybe it will make it more clear.

I have the following json input:

{
  "foo": true
}

and I want to use jq from a shell script, checking if foo == true and printing "fail" otherwise.

I read docs and come up with the following code (for those unfamiliar with shell, a || b checks the exit code of a and runs b if it is non-zero):

# valid input, foo is true
$ echo '{ "foo": true }' | jq -e '.foo == true' >/dev/null || echo "fail"

# valid input, foo is false
$ echo '{ "foo": false }' | jq -e '.foo == true' >/dev/null || echo "fail"
fail

# invalid input
echo '{ "foo": hoot }' | jq -e '.foo == true' >/dev/null || echo "fail"
parse error: Invalid numeric literal at line 1, column 14
fail

But it does not work for the case of no input:

$ echo  | jq -e '.foo == true' >/dev/null || echo "fail"

Obviously, foo is not true, it's not even there, but my code do not print "fail". :-1:

It seems that if -e is used, empty input should be treated like {}, otherwise we can't rely on exit code in case of empty input.