Jq: base64 decoding function

[Downchuck]

Add a function to base64 encode and decode a string result. Some serialization packages such as Jackson, a popular Java package, include base64 encoding for some data fields.

[nicowilliams]

@stdeolan FYI there's several variants of base64 encoding :( Also, a decoder would be equally nice now that we have fromjson.

[schroederc]

I find myself also wanting a base64 decoder. Are there any updates on this feature?

[nicowilliams]

base64 coding is certainly in master.

[antonosmond]

+1 for base64 decoding

[Overbryd]

As this is showing up on Google a lot, and good documentation on jq is sparse, here is to everybody who lands here:

Example:

echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '.foo | @base64d'

Or even use it when building new objects:

echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '{encoded: .foo, decoded: .foo | @base64d}'

[rmetzler]

Thank you @Overbryd! After building jq in a Docker container (fixing the Dockerfile first), this works for me.

[ameissnersofi]

@Overbryd , when I run your example of decoding, I get an error.

$ jq --version
jq-1.5
$ echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | jq '.foo | @base64d'
jq: error (at <stdin>:1): base64d is not a valid format

However, if I run @base64 to encode things, that works just fine. Am I doing something wrong?

[Overbryd]

@ameissnersofi My best guess is that you have to have a recent version of jq. @base64d was only added recently.

[ameissnersofi]

@Overbryd oh I'm sorry. I thought you were actively developing jq. My bad. :/

@l8nite - can you verify my issue (assuming you are developing - the commit history seems to indicate so lol) @stedolan ?

Thanks!

[l8nite]

@ameissnersofi I believe @base64d isn't in 1.5 release, it might be in 1.6 though.

Edit: I built latest from source and confirmed it is there and working:

$ ./jq --version
jq-1.6rc1-10-g7fd9e86-dirty
$ echo '{"foo": "Ym9iIGxpa2VzIGFsaWNlCg=="}' | ./jq '.foo | @base64d'
"bob likes alice\n"

[ameissnersofi]

@l8nite awesome! I see that 1.6 is in rc1 right now, which is probably why brew didn't pick it up for me. Thanks so much!

[pkoppstein]

@ameissnersofi - Have you tried brew install --HEAD jq ?

[Downchuck]

Great job all, on the feature and documenting its access.

[mterron]

@base64d seems to have issues with binary values, or at least does not match base64 -d:

$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64
57432c2f68706f20efbfbd1acd8032efbfbd4029efbfbdefbfbd3e5f0eefbfbd7e0e580122efbfbdefbfbd19550a
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | xxd -p -c64
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d1955
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | hexdump -C
00000000  57 43 2c 2f 68 70 6f 20  9c 1a cd 80 32 b1 40 29  |WC,/hpo ....2.@)|
00000010  a4 a3 3e 5f 0e b9 7e 0e  58 01 22 a7 e8 9d 19 55  |..>_..~.X."....U|
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | hexdump -C
00000000  57 43 2c 2f 68 70 6f 20  ef bf bd 1a cd 80 32 ef  |WC,/hpo ......2.|
00000010  bf bd 40 29 ef bf bd ef  bf bd 3e 5f 0e ef bf bd  |..@)......>_....|
00000020  7e 0e 58 01 22 ef bf bd  ef bf bd 19 55 0a        |~.X.".......U.|

[l8nite]

Thanks for the report @mterron - I'd suggest starting a new issue so we can track a fix for it there. I'll try to take a look at this tomorrow.

Edit: Actually, I remember now that if the encoded value is not a UTF-8 string than the results of the decode are undefined.

[mterron]

Thanks for the report @mterron - I'd suggest starting a new issue so we can track a fix for it there. I'll try to take a look at this tomorrow.

Edit: Actually, I remember now that if the encoded value is not a UTF-8 string than the results of the decode are undefined.

Then it's not actually base64 and should be renamed to something else maybe @utfb64 or something along those lines?

I opened https://github.com/stedolan/jq/issues/1931 to track this.

[pkoppstein]

@mterron -- jq is JSON-oriented with further limitations based on UTF-8 requirements. In particular, @base64 should only be expected to operate on UTF-8 strings. Similarly, @base64d has the necessary limitation that it has, as described in the documentation. Thus it's fair to say that both jq filters are appropriately named, despite these inherent limitations.

[kfox1111]

On my brand new centos 8 system, I yum installed jq, tried this and hit:
jq --version
jq-1.5
:(

[Dynom]

It works in jq-1.6

[kfox1111]

Yup. but top of the line, epel is only jq-1.5, so to a bunch of users, it doesn't exist. :( Anyone maintain the jq version on epel?