Jsonnet: std.base64 fails to encode "§"

[clstb]

I experienced issues using std.base64().

For ex. std.base64("§") returns "pw==" which equals empty string decoded.
(Linux and Jsonnet commandline interpreter v0.11.2)

echo "§" | base64 returns "wqcK" which is the correct encoding for "§".

I don't know if other characters are affected but I think complete base64 encode/decode functions are desirable.

[sbarzowski]

Thanks for reporting. I think it's a bug, however it's a bit more subtle. Base64 operates on bytes, not on (unicode) strings. There are multiple ways of encoding a string as a sequence of bytes (e.g. UTF-8, UTF-16). Jsonnet, for the purposes of base64 encodes strings as a sequence of unicode-codepoints-as-bytes, while pretty much the rest of the world assumes UTF-8. Using naive encoding is clearly not ideal, because it's incompatible with other tools and disallows some strings (e.g. "ąę").

I think we should add std.encodeUTF8, std.decodeUTF8 and use that instead of naive encoding.

[sparkprime]

Having a dedicated bytes type would avoid any issues with stuff not being encoded, as well as improving the performance of large arrays of bytes (since at the moment an array of bytes is really an array of doubles). E.g. std.bsae64 would accept bytes, and if you gave it a string it'd complain that it had not been encoded yet. So then you'd have std.base64(std.encodeUTF8("foo")) or other options.

[clstb]

Thanks for your explanation and the quick gain of traction on this issue.

[seh]

I agree with your proposal, @sparkprime. Reading @clstb's original complaint yesterday, my first reaction was, "Wait, you can't just base64-encode a string like that. You have to first encode the string to a byte vector." I then got distracted by something else before I could reply here.

[sbarzowski]

I agree 100% with disallowing string input to std.base64.

Regarding a separate bytes type, perhaps we can make it an implementation detail? It could be a normal array semantically, but the interpreter could represent it differently.

Either way we should start with exposing encodeUTF8/decodeUTF8.

[clstb]

I saw the progress you are making on the PR.
Do you have an idea when i can expect this getting merged into the go branch of jsonnet?

When the encoding is implemented in C++ it will be extra effort porting it to go was my thought.

[sbarzowski]

In Go it's way easier (the difficulty is the result of implementation details in C++ version) - the PR for that is ready. I'm fairly sure keeping compatibility with C++ version is the only thing that prevents it from being merged.

I'll see if I can overcome the problem reasonably quickly and if not - a non-native implementation will be used.

[clstb]

How is it going?
Do you maybe have an idea for an workaround from within jsonnet in the meanwhile?
We use the base64 function to build kubernetes secrets so to relinquish it is quite cumbersome.

[sbarzowski]

Of course an implementation in pure jsonnet is possible. This implementation looks like a good starting point: https://github.com/truqu/elm-base64/blob/2.0.4/src/Base64/Encode.elm.

A simple, but very ugly workaround is possible if you have a small set of non-ascii characters that you may encounter. You could replace the offending characters with characters with codepoints in range (128,255) and replace them back after decoding.

Anyway thanks for bumping this, anyway. I'll try to finish the builtin magic soon.

[clstb]

I think the PR for this is already merged, so we could close this issue? Also when can the PR of the go fork expected to be merged since you said "I'm fairly sure keeping compatibility with C++ version is the only thing that prevents it from being merged.".

I'm grateful for your work and effort on this!

[Datamance]

Strange -

both of these:

  {testAgain: std.native("encode")("§")},
  {andAgain: std.base64(std.encodeUTF8("§"))}

With the native python function being

def encode(string: str):
    return b64encode(string.encode("utf-8")).decode("utf-8")

give me:

         {
            "testAgain": "wqc="
         },
         {
            "andAgain": "wqc="
         }

in contrast to the output of echo "§" | base64, which is wqcK as noted by @clstb above. In fact, after testing with a couple of strings, the last one or two characters seem to be different between python/jsonnet and the CLI's base64 output (usually uppercase "K" and "=" are switched). I'm sure there's a good answer to this given that lots of people are almost certainly using jsonnet to manage configurations and secrets in kubernetes.

[sbarzowski]

@Datamance

By default, echo adds a newline. If you run echo -n "§" | base64 you get the same answer as in Jsonnet and Python. You can also reproduce the shell result in Jsonnet std.base64(std.encodeUTF8("§\n")).

[Datamance]

That makes sense! Thanks @sbarzowski 👍