Openpgpjs: Streaming decryption fails on files larger than 2GiB

[ama4tf]

Running decrypt on large files fails due to array overrun. This was generated by calling openpgp.decrypt using an encrypted file-stream of 2.1gb in length, a private key, and the binary format. Failure message is below:

Error decrypting message: Invalid typed array length: 2322369312

[twiss]

Hey :wave: This is a limitation of Chrome / V8; it has a max Uint8Array size of 2GiB. It's possible that you could work around this by using streaming decryption: https://github.com/openpgpjs/openpgpjs/releases/tag/v4.0.0.

[ama4tf]

We found a workaround by decrypting the larger files in bash but still not a clean fix. I tried switching to streaming both the input encrypted file and output decrypted stream but still yields the same error

[twiss]

@ama4tf For the streaming solution, please see https://github.com/openpgpjs/openpgpjs/releases/tag/v4.0.0, specifically "Streaming CFB". I think it should be possible to stream files larger than 2GB, although I haven't tried it.

[ama4tf]

I tried Streaming CFB and that did yield a different kind of error, so still not successful but maybe progress

RangeError [ERR_INVALID_OPT_VALUE]: The value "3148715314" is invalid for option "size"

Traceback on the error looks like this

at Object.concat (\node_modules\openpgp\dist\openpgp.js:24175:23)
at Reader.readBytes (\node_modules\openpgp\dist\openpgp.js:24067:51)
at async Reader.peekBytes (\node_modules\openpgp\dist\openpgp.js:24086:17)
at async Object.read (\node_modules\openpgp\dist\openpgp.js:38472:26)
at async \node_modules\openpgp\dist\openpgp.js:38571:22

[twiss]

OK yeah, this is a bug in our streaming implementation, that we should fix.

[twiss]

Hey @lekhacman, yes of course, if you want to take a stab at it feel free.

AFAIK, the root cause of this error is this line: https://github.com/openpgpjs/openpgpjs/blob/25118c3f2859c4f5b4c911e7fcd495e20d1cce4f/src/packet/packet.js#L272

There is a long comment above it explaining what it does and why. Under normal circumstances, I don't think this line should be reading 2GB even if the message is larger than 2GB. The first step to fixing this is probably to find out why it does so.

I would recommend working from the v5 branch, and if you do find a fix, sending a PR there. Otherwise this may lead to a lot of rebasing work in the future. Good luck!

[lekhacman]

@twiss thank you for the hint :)
I let me try to fix this and create a PR.
By the way, i encounter corrupted file after setting CFB streaming config. Not sure what happened.

[fluxquantum]

@lekhacman Did you did you make any strides while working on this? Thank you for your time.

[lekhacman]

@fluxquantum i'm still reading & trying to understand the structure of the library which is difficult.
meanwhile, I realize that the PGP (or at least this library) is intended to be used to encrypt emails and not large files. But i believe this algorithm is such a masterpiece and should be used for general purposes. So i try to give a hand to fix this.
If you are in hurry, you can use AES (stream reading) as i did. I was trying to encrypt a 20GB file with this library but failed to decrypt so i switched back to AES which solved the problem.

[fluxquantum]

@lekhacman Really appreciate you for the quick follow up. May I know which library you are using to perform the streaming encryption/decryption? Just to share some more context, in my case there's some more complexity, because I have a java app performing the encryption and javascript for decryption. I am hoping the java solution can be compatible with the javascript approach. Thanks again for the guidance.