jq prints help when piping output

[evert]

I have a very strange issue. When I invoke jq as such:

cat input.json | jq -M

everything works fine, but when I invoke it as:

cat input.json | jq -M > output.json

output.json ends up being a 0 byte file and jq prints it's help page. I've never seen this before! How can I debug this?

[evert]

Note that this did work:

cat input.json | jq . -M > output.json

[pkoppstein]

@evert - Over time, there has been an attempt to make jq's behavior in the absence of an explicit filter more "intelligent". As you have discovered, the workaround is to be explicit.

(There's some irony in all this: the attempt to make things simpler for some people some of the time ends up making things more complex for some people most of the time.)

[evert]

hmm.. I suppose that makes sense, but why would jq have a different behavior based on where it's sending its output to?

[nicowilliams]

jq really wants a program. Defaulting to '.' is nice, but if you run it
with no arguments in an interactive shell then we really want to print out
an error message so the user doesn't see jq sitting there silently.

[evert]

I understand. This is not a big deal to me anymore, since I figured out what I was doing wrong, but I'm just saying that I think it would be good that whether jq is piped to stdout or a file, it would be less confusing if it behaved identically =)

[matti]

Ran into this as well. It's super confusing that you can do

$ cat file.json | jq -c

but not

LOL=$(cat file.json | jq -c)

[0xmohit]

Neither can you

diff -u <(jq -S < f1.json) <(jq -S < f2.json)

Using a UUOC revealed a broken pipe error. [Edit: Oh, this would be because jq bailed out before reading the entire piped input.]

$  cat f1.json | jq -S 2>&1 | tee 0
cat: write error: Broken pipe
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the
        given filter to its JSON text inputs and producing the
        filter's results as JSON on standard output.
        The simplest filter is ., which is the identity filter,
        copying jq's input to its output unmodified (except for
        formatting).
        For more advanced filters see the jq(1) manpage ("man jq")
        and/or https://stedolan.github.io/jq

        Some of the options include:
         -c             compact instead of pretty-printed output;
         -n             use `null` as the single input value;
         -e             set the exit status code based on the output;
         -s             read (slurp) all inputs into an array; apply filter to it;
         -r             output raw strings, not JSON texts;
         -R             read raw strings, not JSON texts;
         -C             colorize JSON;
         -M             monochrome (don't colorize JSON);
         -S             sort keys of objects on output;
         --tab  use tabs for indentation;
         --arg a v      set variable $a to value <v>;
         --argjson a v  set variable $a to JSON value <v>;
         --slurpfile a f        set variable $a to an array of JSON texts read from <f>;
        See the manpage for more options.

[wtlangford]

Jq makes some assumptions about what you want when you omit a program to
run and output !isatty. In these cases, we assume a default program of
.

In any case, it isn't a bug, but a surprising feature. I'm not inclined to
change it, if only because I feel this behavior is useful (albeit
confusing). It comes up fairly often, though. I'd welcome input on how to
improve documentation.

On Fri, Sep 9, 2016, 10:18 0xmohit [email protected] wrote:

Neither can you

diff -u <(jq -S < f1.json) <(jq -S < f2.json)

Using a UUOC http://catb.org/jargon/html/U/UUOC.html revealed a broken
pipe error:

$ cat f1.json | jq -S 2>&1 | tee 0
cat: write error: Broken pipe
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] [file...]

    jq is a tool for processing JSON inputs, applying the
    given filter to its JSON text inputs and producing the
    filter's results as JSON on standard output.
    The simplest filter is ., which is the identity filter,
    copying jq's input to its output unmodified (except for
    formatting).
    For more advanced filters see the jq(1) manpage ("man jq")
    and/or https://stedolan.github.io/jq

    Some of the options include:
     -c             compact instead of pretty-printed output;
     -n             use `null` as the single input value;
     -e             set the exit status code based on the output;
     -s             read (slurp) all inputs into an array; apply filter to it;
     -r             output raw strings, not JSON texts;
     -R             read raw strings, not JSON texts;
     -C             colorize JSON;
     -M             monochrome (don't colorize JSON);
     -S             sort keys of objects on output;
     --tab  use tabs for indentation;
     --arg a v      set variable $a to value <v>;
     --argjson a v  set variable $a to JSON value <v>;
     --slurpfile a f        set variable $a to an array of JSON texts read from <f>;
    See the manpage for more options.

—
You are receiving this because you are subscribed to this thread.

Reply to this email directly, view it on GitHub
https://github.com/stedolan/jq/issues/1110#issuecomment-245926050, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADQ4V2z-AK-_VGvkJS9-KWCj611Pc362ks5qoWqXgaJpZM4HrShD
.

[0xmohit]

@wtlangford I should have added that the workaround you mentioned (earlier in this comment) worked.

Thanks for the wonderful program!

[nmarley]

Wow, after 1/2 hour of diving into the source to try and fix this "bug", I find out it's a feature. Albeit one that completely breaks pipes, which are a core feature of UNIXish OSes since forever.

My suggestion is to not set a default based on isatty and require that users put '.' as a filter for all occasions, rather than break long-standing UNIX convention by trying to be an "intelligent" program.

[evert]

I'm with @nmarley. It makes really little sense and you should reconsider ;)

In any case, it isn't a bug, but a surprising feature. I'm not inclined to
change it, if only because I feel this behavior is useful (albeit
confusing). It comes up fairly often, though. I'd welcome input on how to
improve documentation.

How about:

WARNING, unlike every other unix utility, jq behaves differently depending on if you pipe stdout to a terminal, or a file. Because this makes extremely little sense, you should basically always specify the . argument as omitting that argument makes this program behave unreliably

It's a bit tongue-in-cheek, but frankly it's fair. This is just not a good user experience. Either make . implicit regardless of output, or force users to explicitly specify . every time.

I really like jq. I use it often and I'm glad somebody made it. I don't want to seem ungrateful. You're awesome for making this, but this is not the right decision ;)

[pkoppstein]

@evert and @nmarley and others - Please note that the problem may seem worse than it is, in that the confusing/buggy/wrong behavior (choose whichever adjective you prefer) evident in jq 1.5 has been made slightly less confusing/buggy/wrong since the release of jq 1.5.

In particular, the observation made by @matti in this thread, namely:

It's super confusing that you can do
$ cat file.json | jq -c
but not
LOL=$(cat file.json | jq -c)

is accurate with respect to jq 1.5 but does not apply to jq as of 5fe053671a768d28bc9773719cc4a9a35e72f30d
(Dec 2, 2015). That is, with this or later commits, LOL is what you'd get from jq -c .

All the more reason to upgrade to the latest version of jq on master, which is currently
0b8218515eabf1a967eba0dbcc7a0e5ae031a509 (March 21, 2016). There are many reasons for upgrading to this version, and I'm not aware of any reasons against doing so. It's a pity it hasn't been "officially" released.

[mblaschke]

Please fix this.

Current workaround for pipes:
cat input.json | jq . | grep foobar

[wtlangford]

Sigh. This isn't a bug, but a behavior we intentionally introduced. We thought it appropriate that stuff | jq would pretty-print for you (which is to say, assume a program of .), since the output was a tty and therefore ostensibly meant for human viewing. We also decided that when you piped the output of jq to something and didn't specify a program, that what you'd done was, in fact, an error.

tl;dr: We made a choice a while ago, and apparently the behavior is more surprising than we expected.

All this said, since 5fe0536, the behavior was changed. If you don't specify a program, the program is assumed to be . unless both stdin AND stdout isatty.
That is:

echo "3" | jq # program assumed to be "."
jq > someFile # program assumed to be "."
echo "3" | jq > someFile # program assumed to be "."
jq # program NOT assumed to be ".", so jq exits with a usage message instead.

The problem now is that this commit isn't in a released version yet. I'm hoping to release one very soon, but in the meantime, if you build master (which is generally better than jq1.5 _anyways_), you'll have the behavior you want.

[ghost]

There is a very simple explanation for the behaviour in this issue: jq '.' is a very useful filter for a human to inspect what a JSON file contains, but a very useless filter in a program. This is why it's the default when you use jq without a filter on the terminal; it's also why, if you're piping it to something, jq assumes you made a mistake, because jq '.' in a pipe is pretty useless. This is no different from python and echo print "foo" | python behaving differently, or grep coloring output when on a TTY but not coloring output when piped to something else (which is something jq also does, by the way).

Which brings me to my point: every other example that reproduces the issue in this comment section is using jq as a JSON pretty printer (I would suggest python -m json.tool for that instead! Mostly because it saves you having to do apt-get install jq) and using the fact that it pretty-printing separates each key and array element by newlines to do, frankly, gross shell things with the output as a result. (In fairness, shells are pretty gross by themselves, with their arbitrary streams of bytes as the only data type and whatnot, so almost every shell thing qualifies as a gross shell thing)

If you're looking to diff JSON files, you should probably use a tool for that purpose, such as json-diff. And if you're grepping foobar in the output, you should probably write a jq program instead to do a proper query on the JSON, such as .. | select(contains("foobar")) or .. | .["foobar"] or whatever works for what you're actually trying to grep.

In short, this was considered reasonable behaviour because jq is intended to be a programming language, not a pretty-printer.

[nicowilliams]

We could just assume that no arguments == "please pretty-print". We could print a one-line warning to stderr when isatty(stdin) and isatty(stdout) saying "use -h or --help to get a help message", and maybe only after a few-second timeout on stdin.

[evert]

Just wanted to confirm here that the behaviour on master is indeed the behaviour I wanted. Looking forward to a release

[Zaijo]

@nicowilliams Exactly! I definitely wish for such little change. Help message is really unhelpful for all who use jq for just pretty-print and then piping somewhere. E.g. grep for quick lookup, which was my case.

[atownley]

I just hit this again, and it's down to what @evert pointed out. If you write a command line tool for a target audience that isn't windows, "just sitting there waiting for input" is exactly the behavior of UNIX command line utilities since the beginning of C and the UNIX operating system. That's why all the other ones work the way they do.

Over time, some of the other tools will print a warning if you do something silly, but if you properly separate stdout from stderr, this warning doesn't impact your use of the tool in normal scripts, but it does give hints to people who expect the program to do something when run interactively.

I've only started using this tool in the last week, and when it's billed as "just like sed, awk...but for JSON" and then the behavior isn't really anything like sed or awk in this respect, it's pretty annoying and confusing.

If you want to leave the behavior as it is, then you MUST include an example in both the online documentation and the help file that says how to invoke it in a normal UNIX filter chain, and then make sure it works like a normal UNIX filter, e.g. separate stdout and stderr, don't pretty-print, etc.

Tools like vim, git, npm and others – even RedHat's popt – address this in the classic UNIX way: configuration files. If you want to add a global switch to do "interactive, non-UNIX filter behavior" as the default but allow us to use a .jqrc in our home directory that says "behave like a good UNIX shell citizen=1", I'd be ok with that. It's still annoying, but at least it means you can get the default behavior you need depending on the way you use it.

My sole use case for this isn't colors (I HATE THEM), it's visual verification of JSON output in two cases: 1) testing the initial behavior of the program and 2) getting sensible results from diff showing deviations in generated JSON output vs. what I expect.

This is what a "classic" UNIX filter is for, and, unfortunately, that's not what jq gives out of the box.

Please seriously consider the .jqrc file and adding the appropriate documentation where people first encounter the issue, e.g. jq help messages.

Otherwise, it seems like a fantastic tool with a lot of power, so thanks for making it available to the rest of us.

Cheers.

P.S. Just reading some of the above again where @purpliminal says "use other tools". I'm sorry, but I don't buy this. If I need/want the rest of the jq features as a programming language (like, say the power of awk), that doesn't stop me from writing 1-line awk scripts to do simple manipulations or printing just because there are other tools that might do that. If you have a Leatherman, unless you really, really need something else, it's probably good enough for the job if you need those features.

Again, I understand you have a vision for what jq should be, and that's fine, but there's a number of customers here that see a great tool and want it to behave the way they want. A configuration file would solve this problem.

Personally, I'm "one of those" that you say should use another tool, but when I go looking for things, I try and get the most functionality I can for anything I download so I don't end up with 500 things on my machine that more-or-less do the same thing. I chose jq because I thought it would support my simple use cases, and then, as I needed or wanted to get more creative, I could just increase my knowledge and reliance on the tool as part of my normal workflow.

[wtlangford]

@atownley I should point out that this comes up fairly often, that we know about it, and we agree that the mixed behavior is confusing.
I should then point out that master has greatly improved behavior surrounding this, where not specifying a program defaults to . in every case except when isatty(STDIN_FILENO) && isatty(STDOUT_FILENO) (which means you called jq like this: $ jq).

We're also aware that there's no formal release right now containing that change and we intend to release one very shortly (when one of us finds a bit of time).

[ttww]

Lost 2 hours 👎 about this "feature".
Don't break standard unix behavior!
But anyway, thanks for the program 👍

[cmckni3]

Any ideas on how to redirect the output to a file?

I have tried:

jq -M -s '.[0]' test.json > output.json

jq -M -s '.[0]' test.json | tee output.json

jq -M -s '.[0]' test.json |& output.json

These always result in an empty file.

[pkoppstein]

Your first try looks good:

jq -M -s '.[0]' test.json > output.json

It's difficult to know what the problem is when you don't give any hints.

By the way, if you wanted to overwrite the original file, the recommended approach, if you have or can get access to sponge in moreutils, is to use it, e.g.:

  jq .... test.json | sponge test.json

[cmckni3]

That actually seems to be the issue. I was trying to overwrite the file. Thanks! 👍

[OnkelTem]

My 5 cents: you shouldn't do any assumption about stdin or stdout. This is not like things work on Unix.

[Ralithune]

What the hell.

There is a very simple explanation for the behaviour in this issue: jq '.' is a very useful filter for a human to inspect what a JSON file contains, but a very useless filter in a program.

Are you kidding me?

How many people need to post about this unexpected behavior before you guys get it?

How about you include your "feature" default behavior as an option specified in a config file, and leave conventional linux behavior for the rest of us looking for a simple two letter linux command line utility to print JSON keys on separate lines?

You know that this is what makes linux is such a pile of trash, right? Because everyone has their own agendas and ideas about what will be best and bucks trends / consistency?

Maybe I just don't understand your genius in writing the behavior this way - I can't think of a reason you wouldn't want users to filter output in a terminal for a program whose entire job it is to make text pretty for terminal. What the hell.

[ghost]

I can't think of a reason you wouldn't want users to filter output in a terminal for a program whose entire job it is to make text pretty for terminal. What the hell.

@Ralithune jq's purpose is not _"to make text pretty for terminal"_. Consider alias jq=python3 -m json.tool. Two characters.

This is not the acceptable tone or behaviour for a technical discussion. You are using an open source tool developed by volunteers: you are in no position to make demands. Concerns were heard and the feature was amended accordingly; rudeness won't do you any favours.

[Macil]

This is not the acceptable tone or behaviour for a technical discussion.

Agreed!

Concerns were heard and the feature was amended accordingly

(re-reads thread more carefully)

Wait, this issue was already fixed on master three years ago? It looks like the fix has been ready for months before anyone in this thread ever ran into the issue. I feel like someone must be pulling my chain here. What has to be done for a new release to happen?

[NiKiZe]

I read here that "changing behavior when you use stdout is not bug but a feature" this is just wrong on so many levels. Never change behavior based on stdin or stdout

at least make this consistent and always require . if that is what is needed.
And I don't buy

but if you run it
with no arguments in an interactive shell then we really want to print out
an error message so the user doesn't see jq sitting there silently.

That is not how any other *nix program works, the simplest example here is cat
Eighter conform to how all other tools does it, or have all other tools behave the same.

[tvalenta]

There are a number of programs that change behavior when piping to stdout. The specifics of that behavior is another matter (I can't recall one that returns an error when output isn't a tty but that's another story).

ls(1) is a prime example, however. compare ls /etc and ls /etc | cat

[jasonkarns]

@tvalenta Besides eliminating ansi color codes (which I agree, is common)?

ls doesn't alter behavior for me, except to skip color codes.

$ ls /etc
/etc@
$ ls /etc | cat
/etc@

(BSD ls, on macOS)

I mostly agree with @NiKiZe

Never change behavior based on stdin or stdout

When scripting, the common scenario is to fiddle with a command till the output appears as desired, and then pipe it to something else. That will immediately break, if one's script is written to expect the output previously observed on STDOUT; and suddenly receives something entirely different when piped to the script.

And when it breaks, the first thing a dev will do is, yet again, run the command to visually inspect the output. Talk about bewildering.

[tvalenta]

The above example returned only the symlink itself, and not the contents of private/etc. However, the default behavior of ls from the MacOS manpage is By default, ls lists one entry per line to standard output; the exceptions are to terminals or when the -C or -x options are specified.. Otherwise, ls | while read name; do echo $name; done would do wildly different things depending on the phase of the moon.

Personally, I find the current behavior of jq troublesome and would prefer it not require a filter when atty is false. My earlier point was largely a pedantic response to the assertion that unix programs never altered behavior when changing stdout.

[LexVocoder]

I just ran into this. The pretty print works, unless I try to pipe it through less (very common). A simple fix would be to make the error message shorter. When I saw the wall of help text, with error: write /dev/stdout: The pipe is being closed at the bottom, I thought it was something wrong with how jq was handling pipes in git bash. If I had just seen a shorter message like this one, I'd be OK:

jq: no filters specified
Usage: C:\ProgramData\chocolatey\lib\jq\tools\jq.exe [options] <jq filter> [file...]

Other UNIX tools do this. Take grep, for instance:

$ grep
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.

You could downgrade the shorter error to a warning if connected to a console.

[brkeh]

Thanks very much for jq, it's good software.

This issue has been fixed years ago, but there hasn't been a release since?

As Macil said,

What has to be done for a new release to happen?

[brkeh]

Thanks very much for release 1.6. I look forward to it getting into Ubuntu, Red Hat, etc. repositories.

[brkeh]

@evert, is this a duplicate of #1028?

[nicowilliams]

This was fixed years ago and the fix was included in 1.5.

I object to the characterization of how Unix commands work in general. (@NiKiZe also noticed this.) Watch this:

$ echo 1 | sed 2>&1 | head
Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]...

  -n, --quiet, --silent
                 suppress automatic printing of pattern space
  -e script, --expression=script
                 add the script to the commands to be executed
  -f script-file, --file=script-file
                 add the contents of script-file to the commands to be executed
  --follow-symlinks
                 follow symlinks when processing in place
$ 
$ echo 1 | awk 2>&1 | head
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:          GNU long options: (standard)
        -f progfile             --file=progfile
        -F fs                   --field-separator=fs
        -v var=val              --assign=var=val
Short options:          GNU long options: (extensions)
        -b                      --characters-as-bytes
        -c                      --traditional
        -C                      --copyright
$ 
$ echo 1 | jq-1.4 -M

jq - commandline JSON processor [version 1.4]
Usage: jq-1.4 [options] <jq filter> [file...]

For a description of the command line options and
how to write jq filters (and why you might want to)
see the jq manpage, or the online documentation at
http://stedolan.github.com/jq

$ 

@stedolan billed jq as an awk for JSON. And look, that's how it used to behave. We, the maintainers, made it "better" since, though that's debatable, as @NiKiZe points out -- maybe we shouldn't have done that! Whatever your opinion of how Unix commands work or should work, there's certainly no need to yell at us about bugs, real or imagined, in old version of jq -- always check if you have the latest and greatest.

I agree with the deleted used who mentioned tone, and I mostly agree with @NiKiZe, though I don't mind some slight behavior selection based on whether a command is being run with isatty(STDOUT_FILENO) being true. We did end up assuming the program is . in that case when the program is absent because many jq users didn't really get it. Maybe we should have shortened the error message in that case a great deal -- the longer the message, the less likely it is to be read -- but what's done is done.

[OnkelTem]

Dying laughing at that comment about jq's purpose: prettifying json. To my mind JQ is one of the most challenging compact functional-style languages I even seen. I make programs on other languages for ages and yet I cannot cook JQ right! Pretty printing... what a folly it was %)

[brkeh]

@nicowilliams are you sure this was included in 1.5? Wasn't this fixed with 5fe0536 in 2015 December and 1.5 was released in 2015 August?

br@aa:~$ echo '{}'|jq
{}
br@aa:~$ echo '{}'|jq|cat
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]

    jq is a tool for processing JSON inputs, applying the
    given filter to its JSON text inputs and producing the
    filter's results as JSON on standard output.
    The simplest filter is ., which is the identity filter,
    copying jq's input to its output unmodified (except for
    formatting).
    For more advanced filters see the jq(1) manpage ("man jq")
    and/or https://stedolan.github.io/jq

    Some of the options include:
     -c     compact instead of pretty-printed output;
     -n     use `null` as the single input value;
     -e     set the exit status code based on the output;
     -s     read (slurp) all inputs into an array; apply filter to it;
     -r     output raw strings, not JSON texts;
     -R     read raw strings, not JSON texts;
     -C     colorize JSON;
     -M     monochrome (don't colorize JSON);
     -S     sort keys of objects on output;
     --tab  use tabs for indentation;
     --arg a v  set variable $a to value <v>;
     --argjson a v  set variable $a to JSON value <v>;
     --slurpfile a f    set variable $a to an array of JSON texts read from <f>;
    See the manpage for more options.
br@aa:~$ 

[nicowilliams]

@brkeh

$ jq -n '"foo"'|./jq
"foo"
$  jq -n '"foo"'|jq-1.5
"foo"
$ ./jq --version
jq-1.6-5-g8ea4a55-dirty
$ jq-1.5 --version
jq-1.5
$ 

I also built the jq-1.5 tag from scratch. Same behavior.

[brkeh]

Yes I see that behaviour too. Isn't the issue jq without -n?

br@aa:/tmp/aa$ jq -n '"foo"'|jq
"foo"
br@aa:/tmp/aa$ jq --version
jq-1.5-1-a5b5cbe
br@aa:/tmp/aa$ 

when I invoke it as:
cat input.json | jq -M > output.json
output.json ends up being a 0 byte file and jq prints it's help page.

[nicowilliams]

No. It's the jq without arguments that you want to assume . as the jq program in these conditions.

Honestly, I think we should have not fixed this at all. We should have made the jq error message really pithy, something like:

Error: no program given.  Use this to pretty print input to output: jq .

[brkeh]

Agree jq without arguments is when the problem exists. As far as I can see the problem only shows up in version 1.5 when you redirect standard out to standard in of another process or to a file.

[Macil]

When I ran into this issue, my first reaction was to simplify my command by removing everything I was piping jq into, which no longer reproduced the issue. I re-read about bash pipes and stderr to make sure my shell knowledge hadn't rotted. I then read jq's message (I had no idea why it was showing me its help message. I didn't understand it was an error message in response to me doing something wrong. I assumed some kind of bug was causing it to show its help message instead of doing stuff, so I didn't read too much into it.), which said you could give jq options and also a "filter". I knew this was optional because the command previously worked without one, and anyway, I didn't want to "filter" out anything from the output at all, so I moved on to reading about jq's other options, and tried every combination of them I could think of. I would test the option without piping jq into anything else, it would work, then I would re-add the pipe, and then it would fail. This reinforced my original idea that the issue wasn't actually anything about how I was using jq, but that something about my shell was horribly broken.

An actual error message would have helped me get my command working faster, but I would have assumed the inconsistency with the error message was some kind of bug and reported it immediately after. I like the current 1.6 behavior, but if the behavior were to be changed again, then my vote would be making it so jq with no arguments always prints the usage screen, regardless of pipes being involved. The best way to learn a tool is through using it, and it's weird to teach users that no arguments are needed and then surprise them later.

[brkeh]

Personally I'd rather not be forced to enter a filter. I think the latest code now to assume the . filter is good. I think the old way confuses, frustrates and maybe deters beginners.

[pkoppstein]

@brkeh wrote:

As far as I can see the problem only shows up in version 1.5 when you redirect standard out to standard in of another process or to a file.

Just to be clear, the official release of jq 1.5 behaves as intended with respect to one of the scenarios under consideration:

$ jq-1.5 --version
jq-1.5
$ echo 1 | jq-1.5
1

However there were changes made on "master" in the interval of time between the official releases of jq 1.5 and jq 1.6.

In my opinion (and I've been watching this wheel of misfortune spinning for several years now), the current (jq 1.6) semantics and implementation are much better than reasonable, and it would be just plain silly to reopen the can of worms. Which is not to say the documentation might not be improved in some way. If a simple message would do the trick, it would surely be: if in doubt, specify the filter.

[brkeh]

@pkoppstein wrote:

behaves as intended with respect to the scenario under consideration:

$ echo 1 | jq-1.5
1

Isn't the scenario under consideration, jq prints help when piping output, not jq prints help when receiving input from a pipe?

[pkoppstein]

@brkeh - True, there are several scenarios under consideration. I was responding to the suggestion by @macil (and others, elsewhere) that:

if the behavior were to be changed again, then my vote would be making it so jq with no arguments always prints the usage screen

[brkeh]

Maybe I'm misunderstanding something here but it seemed to me this ticket was all about piping output from jq and the issue still existed in jq 1.5. I'll stop talking, if you'd rather that though :) Thanks guys for all your work.

[nicowilliams]

I agree with @Macil. If we ever change this again, jq will print an error every time a program is not given. Perhaps the error will be pithier when isatty(STDIN_FILENO) || isatty(STDOUT_FILENO), but that will be the only difference based on whether you're running jq in a tty vs. not.

[1Hyena]

Arch Linux user reporting in. I never knew this bug even existed until I had to run my code on an "up to date" Linux Mint machine. After debugging it for an hour with no success I found this issue and the . workaround, which helped me a lot. Arch Linux has rolling releases, which means I always have the latest versions of all software, including jq. Apparently that is not the case with other flavors of Linux, causing such discrepancies. That said, it is unfortunate that even today, it is still advisable to always include the . program argument for jq, for the sake of compatibility.

[stdedos]

Or ... you could have added at least a warning message?

Or emphasize it somehow that:

You might like to have '.' as filter when piping!

[Ralithune]

Angry tone person reporting in.

I have jq 1.5 and I'm still seeing this issue. Should I be?

Clearly, I'm doing something wrong, because this was fixed years ago and I'm not allowed to make angry posts about it.

```[root@host ~]# jq --version
jq-1.5
[root@host ~]# echo '{"one": 1}' |jq
{
"one": 1
}
[root@host ~]# echo '{"one": 1}' |jq |grep one
jq - commandline JSON processor [version 1.5]
Usage: jq [options] [file...]

jq is a tool for processing JSON inputs, applying the
given filter to its JSON text inputs and producing the
filter's results as JSON on standard output.
The simplest filter is ., which is the identity filter,
copying jq's input to its output unmodified (except for
formatting).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Some of the options include:
 -c     compact instead of pretty-printed output;
 -n     use `null` as the single input value;
 -e     set the exit status code based on the output;
 -s     read (slurp) all inputs into an array; apply filter to it;
 -r     output raw strings, not JSON texts;
 -R     read raw strings, not JSON texts;
 -C     colorize JSON;
 -M     monochrome (don't colorize JSON);
 -S     sort keys of objects on output;
 --tab  use tabs for indentation;
 --arg a v  set variable $a to value <v>;
 --argjson a v  set variable $a to JSON value <v>;
 --slurpfile a f    set variable $a to an array of JSON texts read from <f>;
See the manpage for more options.

```

I love jq, for the record. I think anyone would take this 2 hour annual frustration over not having jq at all any day, but we really shouldn't have to.

@nicowilliams Your point about sed and awk behaving this same way is well taken; however: sed and awk don't have a default behavior, and jq does. I think everyone would agree that it's extremely useful to take a single-string JSON blob mash and separate it in to newlines and colorized key:value pairs. When a default operation is performed and prints output on my screen, I expect it to be pipe-able; just like any reasonable linux user, as evidenced by the dozen so people who've posted here complaining about this issue.

Ultimately though, we're all getting this functionality for free, so you're free to set things however you want, and I'm free to be angry about it.

[pkoppstein]

@Ralithune - If you love jq, you should have no problem feeding it a filter, even if it's only .; and if you really love jq, you should upgrade to jq 1.6.

[Ralithune]

If you love jq, you should have no problem feeding it a filter, even if it's only ..

@pkoppstein
"If you love driving, you should have no problem driving me to the airport any time I ask!"
"If you love pizza, you should have no problem eating it every night for the rest of your life!"
"If you love The Office, you should have no problem with it being only available in black and white!"

The "If you love X, you should put up with Y" argument is a logical fallacy, and falls apart when you realize that Y breaks convention / standard. Performing an operation by default that doesn't then work with piping in linux breaks convention / standard.

Thank you for the advice to upgrade to 1.6; that's not really an option on the systems I work on. Mainly, I was asking about whether this should be fixed in 1.5, or not, since after (carefully) reading the thread here, I gathered that it should be? Or is that wrong?

[Macil]

The problem was fixed in 1.6.

[Ralithune]

Ok, thank you @Macil.

[gschanuel]

i'm trying this:
iperf3 -s -J | jq -c ' del(.intervals) |add' >> ./iperf3.log
but it's not working yet. archlinux jq 1.6-2
what am I doing wrong?