Jq: Piping into jq from some (continuous) commands hangs.

[abelnation]

I cannot get jq to output data during a multi-pipe shell command. My goal is to pipe either cut or awk into jq. This works:

while(true); do echo 'asdf {"type":"stats"}'; sleep 1; done | cut -f 2 -d ' '

This just hangs with no input

while(true); do echo 'asdf {"type":"stats"}'; sleep 1; done | cut -f 2 -d ' ' | jq '.'

Any workarounds? Thanks!

[pkoppstein]

Use the "--unbuffered" option, e.g.:

(echo 'asdf {"type":"stats"}'; sleep 1 ) | cut -f 2 -d ' ' | jq --unbuffered .

[nicowilliams]

@abelnation The problem is that cut(1) is buffering its output, so jq(1) never gets it. Trace cut and you'll see.

Instead you could use jq with the raw input option (-r, --raw-input) and then split the string in jq instead of depending on cut(1).

Or you could use stdbuf(1). See http://www.pixelbeat.org/programming/stdio_buffering/ .

@pkoppstein Close! That is the problem, but in cut(1), not jq(1).

[nicowilliams]

See also http://stackoverflow.com/questions/3465619/how-to-make-output-of-any-shell-command-unbuffered .

[abelnation]

@pkoppstein definitely tried --unbuffered to no avail. @nicowilliams's answer sounds correct. i'm very surprised that such a core unix tool cannot unbuffer it's output. i'll try the workaround, thanks

[abelnation]

@pkoppstein, also that stdbuf article was very helpful. i was able to get stdbuf working, but could not get the --raw-input solution working. here's an example log line that i'm inputing to jq via tail:

[2014-07-10 08:35:36.521] [INFO] 03_node_memwatch_hello - {"type":"memusage","data":{"rss":551342080,"heapTotal":542535944,"heapUsed":538198672}}

My command:

tail -f logs/03_node_memwatch_hello.log | jq -r 'split("\n")'

My error:

parse error: Invalid numeric literal at line 1, column 12

Any ideas?

[pkoppstein]

@abelnation asked:

Any ideas?

It looks like the problem is that you're trying to read non-JSON text as though it were JSON. If that's the case, you could use the --raw-input option. That splits at newlines for you. You'd presumably want to parse each lines to extract the JSON component, and then use fromjson. It looks like splitting at " - " might be the way to go, but note that the github version of jq includes support for regular expressions.

[abelnation]

I see, did not know -r (--raw-input) splits at new lines. I still get an error:

tail -f logs/03_node_memwatch_hello.log | jq -r 'split(" ")'

error (when i try to split on both '-' and ' '):

parse error: Invalid numeric literal at line 1, column 12

[abelnation]

For ease of reproducing:

echo '[2014-07-10 08:55:25.162] [INFO] 03_node_memwatch_hello - {"type":"memusage","data":{"rss":128286720,"heapTotal":113630448,"heapUsed":109014056}}' | jq -r 'split(" ")'

[pkoppstein]

$ /usr/local/bin/jq --raw-input -M 'split(" ")| .[length  - 1]|fromjson' log.txt

{
  "type": "memusage",
  "data": {   
    "rss": 551342080,
    "heapTotal": 542535944,
    "heapUsed": 538198672
  }
}

P.S. The abbreviation for --raw-input is -R, not -r.

[abelnation]

Got it, so looks like the prob was the trailing new line?

Thanks for the help. Great tool!!

Sent from my phone

On Jul 10, 2014, at 9:56 AM, pkoppstein [email protected] wrote:

$ /usr/local/bin/jq --raw-input -M 'split(" ")| .[length - 1]|fromjson' log.txt

{
"type": "memusage",
"data": {
"rss": 551342080,
"heapTotal": 542535944,
"heapUsed": 538198672
}
}
—
Reply to this email directly or view it on GitHub.

[pkoppstein]

I believe the problem was that you used -r instead of -R.

Of course, if the input has blank lines, then the jq program will have to take that into account.

If you have access to a sed that supports the -l option, then you should probably use that.

[abelnation]

doh!, -R def works fine. ok thanks so much :)

On Thu, Jul 10, 2014 at 10:38 AM, pkoppstein [email protected]
wrote:

I believe the problem was that you used -r instead of -R.

Of course, if the input has blank lines, then the jq program will have to
take that into account.

If you have access to a sed that supports the -l option, then you should
probably use that.

—
Reply to this email directly or view it on GitHub
https://github.com/stedolan/jq/issues/477#issuecomment-48637933.

Abel Allison
abel.[email protected]
713-557-4984