Jq: Add support for @base64d like gzip decompression @gzipd

[beckyconning]

echo '{"a": "eyJiIjogImMifQ==", "d": "e"}' | jq -c ". + {a: .a | @base64d | fromjson}"

gives

{"a":{"b":"c"},"d":"e"}

which is great but what if a is also gzipped?

e.g.

echo '{"a": "H4sIAOIs+VwCA6tOskqu5QIAlfuWnAYAAAA=", "d": "e"}' | jq -c ". + {a: .a | @base64d | @gzipd | fromjson}"

gzip works in a progressive streaming manner so this shouldn't cause any problems architecturally.

compression could also be supported e.g. @gzipc

[beckyconning]

might not be needed if we end up merging this https://github.com/sloanlance/jq/issues/1

in the meanwhile i did the following:

A=$(uuidgen); B=$(uuidgen); mkfifo $A && mkfifo $B && { aws dynamodb scan --table example --max-items 2 --output json | jq -r -c '.Items | .[]' | tee >(jq -r '.BinaryData.B // "H4sIABWa/lwCA6uu5QIABrCh3QMAAAA="' | while read bin; do echo $bin | base64 --decode | gzip -d; done | jq -c '{ BinaryData: { B: . } }' > $A) | tee $B > /dev/null & paste $B $A | jq -c '. + input' | gzip -9 | aws s3 cp - s3://example-bucket/example.json.gz && rm $A $B; }

[nicowilliams]

I've a branch, dlopen, in my fork that adds a way to shell out. I need to do a bunch more work on it, mainly to add a better way to specify what a jq program can do, fix some bugs, ...

[beckyconning]

I think that's incredibly important @nicowilliams. Did you see https://github.com/haochenx/jq/tree/sh-support as well?

[nicowilliams]

@beckyconning yes, it is. No, but I'll take a look.

The functionality in my dlopen branch is... extensive. There's C-coded modules, "file handles", file I/O, process stuff (very basic at this time, but I'll improve it, and I'll add some sort of sandboxing too), _coroutines_.

[beckyconning]

@nicowilliams is there anything i can do to help with your dlopen branch or shell support in general?

[trothwell]

Is there anything we can do here? Is it possible to add internal extensions (modules) that execute shell commands?

[beckyconning]

@trothwell yeah that would be very cool. would just gzip be enough for what you're working on or do you need other commands?

these external program prs such as @nicowilliams' are awesome but seem to taking a long time. i wonder if it would be faster to add specific commonly needed functionality such as gzip to jq directly.

it seems jq has base64 encoding and decoding now which is fantastic. if jq also had gzip then i could improve the performance of my scripts dramatically : )

[leonid-s-usov]

We are progressing on the dlopen PR, I took over it for now, but that will get out as 2.0 rc, lots of changes.
In the meantime, adding another filter to 1.7 could be an option. @pkoppstein please consider this as a candidate for the 1.7 issue list

[beckyconning]

Thanks @leonid-s-usov!