Node: Inconsistency concatenating string with int8 array

[realworld666]

• Version: 10.6 & 5.12
• Platform: Windows
• Subsystem:

I'm having a problem with this code since updating from node 5 to node 10 (also seen in node 8). I've attached the full code and a sample keys file but the problem code is displayed below.

GetScrambledKey returns an int8array object. We have been prepending "ECB_" to the beginning of this which converts it back to a string. This has then been used as a database key lookup. In the latest versions of node the behaviour of this admittedly not great code has changed and so our database lookups fail.
The first loop prints out the same results in both versions of node.

244, 249, 162, 181, 184, 167, 227, 212, 139, 133, 5, 136, 212, 185, 239, 142, 104, 241, 106, 186, 191, 225, 208, 250, 252, 18, 113, 160, 235, 230, 151, 51, 23, 38, 72, 215, 40, 43, 253, 190, 75, 185, 136, 252, 26, 147, 37, 13, 139, 37, 148, 214, 107, 200, 139, 49, 44, 18, 205, 242, 46, 81, 95, 71,

The second loop prints out different results. The node 5 version:

69, 67, 66, 95, 65533, 65533, 65533, 65533, 65533, 65533, 65533, 1291, 65533, 5, 65533, 1337, 65533, 65533, 104, 65533, 106, 65533, 65533, 65533, 65533, 65533, 65533, 18, 113, 65533, 65533, 65533, 65533, 51, 23, 38, 72, 65533, 40, 43, 65533, 65533, 75, 65533, 65533, 65533, 26, 65533, 37, 13, 65533, 37, 65533, 65533, 107, 523, 49, 44, 18, 65533, 65533, 46, 81, 95, 71,

The node 8 version

69, 67, 66, 95, 65533, 65533, 65533, 65533, 65533, 65533, 65533, 1291, 65533, 5, 65533, 1337, 65533, 104, 65533, 106, 65533, 65533, 65533, 65533, 65533, 65533, 18, 113, 65533, 65533, 65533, 51, 23, 38, 72, 65533, 40, 43, 65533, 65533, 75, 65533, 65533, 65533, 26, 65533, 37, 13, 65533, 37, 65533, 65533, 107, 523, 49, 44, 18, 65533, 65533, 46, 81, 95, 71,

Certain characters from the int8array seem to have been dropped rather than converted to 65533 as was the case in node 5.

This is a mission-critical piece of code for us and we're stuck on node 5 unless we can find a solution. Is this a node bug that can be fixed or is it something I can/should work around?

var eKeysCache = {};
var eKeys = null;
var dbKey = "[email protected]";

LoadKeys("MyKeys.ekeys");

var scrKey = _GetScrambledKey ( dbKey );
var stringOut = "";
for(var i=0; i < scrKey.length; i++)
{
    stringOut += scrKey[i]+", ";
};
console.log(stringOut);
var newKey = "ECB_"+scrKey;
stringOut = "";
for(var i=0; i < newKey.length; i++)
{
    stringOut += newKey.charCodeAt(i)+", ";
};
console.log(stringOut);

NodeEncrpytBug.zip

[mscdex]

This was a change in behavior in V8 5.5. There's not much that can be done about it. See https://github.com/nodejs/node/issues/10300.

[realworld666]

This is not good news :(

[realworld666]

Is there any way to emulate the old behaviour? I can migrate all my keys to the new way but I'll need a way to find them

[mscdex]

Not really. The only solution would be to write your own UTF-8 decoder to ensure you get the old output in all cases, which would not be an easy task.

[realworld666]

What about the original node code that did the conversion? Would I be able to reuse that? I'm not sure where I would find it to even look at it

[mscdex]

The conversion you were doing does not involve node at all, it's all V8.

[addaleax]

@realworld666 I think https://github.com/v8/v8/commit/fd40ebb1e64274ae3529f8bbe6dad6adc76cb391 was the main change here.

So: If you need to need keep the exact results, you probably need your own UTF-8 decoder (in JS or otherwise). You can probably find an existing one out there and adjust it to your needs, or write your own – it’s work, but it’s probably less a lot less work than manually reverting the change in V8 and maintaining that.

On the other hand, if you don’t need to keep the exact results, I’d suggest replacing your code by one that doesn’t involve conversions between binary data and character strings at all (and therefore doesn’t have to rely on the exact number of � replacement characters).

[ryzokuken]

@realworld666 exactly as @addaleax said. Because there's nothing left to be done here, I'm closing this. Please feel free to reopen this issue if you think I missed something and there's still something actionable on our part.