Jq: Exits with 141, despite executing correctly (when streaming?)

[tomfitzhenry]

I'm doing curl http://secret.example.com/ | jq -r '.Users[].UserName' | head -1. In cases where jq is able to parse and run the filter, it sometimes exits with 0 and sometimes with 141. Since I am running this with pipefail enabled, this causes the script to sometimes pass (as expected) and sometimes erroneously fail.

I cannot provide the HTTP endpoint I'm curling, nor even the JSON file, but I have managed to reproduce it below.

Reproducible test case

Let's try something simple.

$ cat input.json | jq -r '.Users[].UserName' | head -1
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
$ echo ${PIPESTATUS[@]}
0 0 0

... as expected.

Now let's simulate input.json being delivered over a slow stream, by having jq read from a FIFO.

In one terminal:

mkfifo input.pipe
cat input.pipe | jq -r '.Users[].UserName' | head -1 # this will block, on input.pipe being closed
echo ${PIPESTATUS[@]}

In another terminal:

sleep 50000 > input.pipe # keeps pipe from closing after first write

In another terminal:

split -l 700 input.json
cat xaa >> input.pipe
cat xab >> input.pipe

Now kill sleep in the second time, which will cause jq in the first terminal to terminate. PIPESTATUS[@] in the first terminal is:

0 141 0

... meaning jq is terminating with exit code 141, suggesting it exited due to SIGPIPE.

Why?

Also, if you modify input.json slightly, e.g. sed 's/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaa/', then suddenly PIPESTATUS[@] is 0 0 0 suggesting jq exits with 0.

What's going on here?

[tomfitzhenry]

@FauxFaux raised that cat has the same "bug", if it is considered a bug at all.

$ (seq 9000; sleep 0.1; seq 9000) | cat | head -c 5
1
2
3
$ echo ${PIPESTATUS[@]}
141 141 0

[nicowilliams]

This is not a bug. The problem is that head(1) exited without consuming
all of its input -exactly what head(1)'s purpose is- which leaves the
process producing that input to get EPIPE or SIGPIPE. If only head(1)
could back-propagate a non-error version of EPIPE... but it's 40 years too
late for that. (Shells back then didn't have set -o pipefail anyways.)

You can use the first(EXP) builtin in jq 1.5 instead of head(1) to avoid
this problem.

[tomfitzhenry]

That makes sense. Thanks.

[tokland]

Documented here: http://www.greenend.org.uk/rjk/tech/shellmistakes.html#pipeerrors

[timothybosse]

If you need to use something other than jq, I found awk 'FNR <= 1' did the trick as a replacement for head -n 1. Annoying "feature" but setting set +o pipefail fails according to my tests.