Jq: sell it to us: is jq really handier than a program that just runs a JS lambda on parsed stdin?

[jedwards1211]

I'm not sold on jq; my gut feeling is that being able to write pure JS will be more convenient for most cases. But I want you to give me your best pitch. Maybe there are examples where it really shines in comparison to pure JS that I'm not aware of?

Here's a script I made that runs an ES6 lambda on the input:

#!/usr/bin/env babel-node

const {stdin, stdout} = process
const inputChunks = []

const fn = eval(process.argv[2])
if (typeof fn !== 'function') throw new Error('argument must evaluate to a function')

stdin.resume()
stdin.setEncoding('utf8')

stdin.on('data', chunk => inputChunks.push(chunk))

stdin.on('end', () => stdout.write(
  JSON.stringify(fn(JSON.parse(inputChunks.join())), null, 2)
))

With this I can easily do the equivalent of jq '[.[] | {message: .commit.message, name: .commit.committer.name}]':

$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | \
./jl 'j => j.map(({commit: {message, committer: {name}}}) => ({message, name}))'
[
  {
    "message": "Deal with strptime() on OS X and *BSD (fix #1415)\n\nstrptime() on OS X and *BSDs (reputedly) does not set tm_wday and\ntm_yday unless corresponding %U and %j format specifiers were used.\nThat can be... surprising when one parsed year, month, and day anyways.\nGlibc's strptime() conveniently sets tm_wday and tm_yday in those cases,\nbut OS X's does not, ignoring them completely.\n\nThis commit makes jq compute those where possible, though the day of\nweek computation may be wrong for dates before 1900-03-01 or after\n2099-12-31.",
    "name": "Nicolas Williams"
  },
  {
    "message": "Attempt to fix #1415\n\nOS X (and *BSD) strptime() does not set tm_wday nor tm_yday unless\ncorresponding format options are used.  That means we must call timegm()\nto set them.",
    "name": "Nicolas Williams"
  },
  {
    "message": "Add private my_timegm()",
    "name": "Nicolas Williams"
  },
  {
    "message": "Fix HAVE_TM_TM_GMT_OFF usage",
    "name": "Nicolas Williams"
  },
  {
    "message": "Use AC_CHECK_MATH_FUNC() for all math functions",
    "name": "Nicolas Williams"
  }
]

It may be a bit dense, but it has the huge advantage of being completely obvious to any experienced ES6 developer.

[ghost]

Cool! If you find that making a short JS program is easier for you or that it matches your use case better, go ahead! You're not obliged to use jq in any way, and there's no point in selling anyone anything or trying to pitch open source projects against each other. I haven't used lambduh, but I think it looks great!

[jedwards1211]

@slapresta haha okay, maybe don't try to sell it to me, but it would be cool if we could come up with some operations that are way more compact in jq than JS functions, because there are always tradeoffs, right? I'm sure there is something, and I don't want to dismiss jq library too quickly, because with 9,000+ stars, it seems like a lot of people find it useful!

[jedwards1211]

@slapresta for instance being able to pipe | one step into another in jq is nice because in a JS lambda, you'd have to use deeply nested function calls or a verbose procedural function body.

[jedwards1211]

@slapresta jq also has the advantage of being written in C, so I'm sure it's way faster.

[pkoppstein]

@jedwards1211 - Perhaps this is not the right forum, but it would be useful to document the relative merits of jq and what it is you are proposing. It might help if you could characterize more precisely what the alternative you have in mind is. From your original post, it almost looks like you are proposing an alternative implementation of certain components of the jq language.

Since there is quite a lot of documentation about jq, and many examples of its use are available (e.g. on stackoverflow.com and rosettacode.org), it might also help if you got the ball rolling by explaining what you see as the "selling points" of the alternative you have in mind.

As for efficiency and performance, if there are some instances where you believe your alternative would shine, please share some examples. Thanks!

[jedwards1211]

@pkoppstein oh, I doubt a pure JS alternative would e faster than jq in any case -- that's one downside of what I'm proposing (which I made into the lambduh package last night).

As I said my first post, the main selling point of running a JS function on the input JSON is obviousness; no experienced JS developer will need to read a manual to figure out what this means:

lambduh 'json => json.map(({commit: {message, committer: {name}}}) => ({message, name}))' < somefile

[ghost]

@jedwards1211 sorry, I never replied back to this; I think it's an interesting conversation! Personally I find that jq scripts are way shorter than their "real programming language" equivalent. Compare something like:

.files[] |= "\(.filename).\(extension)"

with:

json => json.files = json.files.map(({filename, extension}) => `${filename}.${extension}`)

(I'm not sure if this is entirely correct, I haven't tried; perhaps you'd still need to return the resulting json from the lambda)

This comes in super handy when you're just trying to get some insights about/from a JSON file in a quick and dirty manner. However, the maintainability of it is low (not only because jq has a Perl-ish vibe to it in that many things are implicit, but simply because nobody knows jq) so I could see myself fiddling around a JSON file with jq, then writing the equivalent lambduh script and committing _that_ instead. (lambduh also has as an advantage that it's part of the npm ecosystem, which is omnipresent these days, so I can just run npx lambduh ... and get lambduh for free)

Another use-case I've found jq works great for is as a templating language; that is, as a way to transform JSON (or YAML/CSV, if you plug yaml2json/csvjson before the jq invocation!) into _even cooler JSON_. For simple templates, the resulting jq script usually looks to the untrained eye pretty much like the end-result JSON, but with weird stuff in it; For more complex templates, jq's "language-ness" allows to be able to split it into individual functions that generate parts of the template, and compose those effectively, which is harder to do in many other templating systems.

As an example, I once wrote a reasonably-maintainable jq script (~100 lines) that transformed a JSON (but it should've been a YAML) containing a list of usernames into an AWS CloudFormation JSON that described a bunch of SQS queues and other AWS resources that we needed to create for each developer, to be able to guarantee independent development environments. (Yes, I'm aware mocks are the actual good way to solve this)