Jq: @base64d doesn't match based64 -d output

[mterron]

Describe the bug
The output of jq's @base64d does not match the output of base64 -d for the same string

To Reproduce

$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64
57432c2f68706f20efbfbd1acd8032efbfbd4029efbfbdefbfbd3e5f0eefbfbd7e0e580122efbfbdefbfbd19550a
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | xxd -p -c64
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d1955
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | hexdump -C
00000000  57 43 2c 2f 68 70 6f 20  9c 1a cd 80 32 b1 40 29  |WC,/hpo ....2.@)|
00000010  a4 a3 3e 5f 0e b9 7e 0e  58 01 22 a7 e8 9d 19 55  |..>_..~.X."....U|
$ echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | hexdump -C
00000000  57 43 2c 2f 68 70 6f 20  ef bf bd 1a cd 80 32 ef  |WC,/hpo ......2.|
00000010  bf bd 40 29 ef bf bd ef  bf bd 3e 5f 0e ef bf bd  |..@)......>_....|
00000020  7e 0e 58 01 22 ef bf bd  ef bf bd 19 55 0a        |~.X.".......U.|

Expected behavior
The outputs of echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | jq -Rr '@base64d' | xxd -p -c64 and echo "V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU=" | base64 -d | xxd -p -c64 should be equal.

Environment (please complete the following information):

• OS and Version: Alpine Linux
• jq version: jq-master-v3.8.0-3651-g18d55b6bda

Additional context
Seems to be related to binary values being encoded.

[pkoppstein]

In brief, @base64d has a documented and (in the context of jq) necessary limitation, which can be stated as follows:

Let $B be an arbitrary base64 string, then $B | @base64d is undefined
if base64 -D <<< $B is not a valid UTF-8 string.

Here base64 refers to the command-line program of that name, and I'm assuming that base64 -D acts as the inverse of base64.

Using the moreutils program isutf8 to test this condition, we see
that the given base64 string does NOT satisfy it:

$ S="V0MsL2hwbyCcGs2AMrFAKaSjPl8OuX4OWAEip+idGVU="
$ base64 -D <<< "$S" | xxd -p
57432c2f68706f209c1acd8032b14029a4a33e5f0eb97e0e580122a7e89d
1955
$ base64 -D <<< "$S" | isutf8
(standard input): line 1, char 8, byte 8: Expecting bytes in the following ranges: 00..7F C2..F4.

[mterron]

Fair enough. I'd make the suggestion to document that the results of @base64d is incorrect instead of merely undefined when decoding non UTF-8 strings.

Having put this to rest, is there any plans to have a way to "shell out" to call external utilities to aid in processing this edge cases?

[pkoppstein]

@mterron - I believe the author or authors of this particular section of the official manual did not want to commit to any particular behavior at the time of writing, no doubt because, as you say, there is much to be said for raising an error condition.

As for shelling out -- yes, there are plans to support this (see e.g. https://github.com/stedolan/jq/issues/147 and https://github.com/stedolan/jq/issues/1614), but it won't help much in the present case, for the reason already stated.

[mterron]

@pkoppstein give a way to shell out I'll make it work :)

Thanks!

[pkoppstein]

@mterron wrote:

I'll make it work :)

It appears you are not quite grasping the fact that the implementation of the shell-out function will (of necessity) be designed to prevent what I understand you want to do.

The "j" in jq can be understood as a commitment that, with two exceptions, every jq filter should produce strictly valid JSON, the exceptions being the values for NaN and Infinite, but even these two values are ultimately converted to valid JSON on output (e.g. echo Nan | jq . #=> null).

This is not to say that every version of jq is guaranteed to reject non-JSON strings, but that's not for want of trying :-)

[mterron]

Oh I understand very well. xxd will convert the binary back to an hexadecimal string that is valid json

[pkoppstein]

@mterron - Excellent. You might want to mention your use case on one of the shell-out tracking issues.

[itchyny]

I think this is worth fixing and it does not break compatibility. It is useful if we can decode the image data from aws ec2 get-console-screenshot with jq only.