PrivateBin exceeds memory

[privatebin12]

Steps to reproduce

1. Upload a file

What happens

If the file size isn't a fraction of the memory_limit, upload will fail with the following error:

PHP Fatal error: Out of memory (allocated 604905472) (tried to allocate 200236432 bytes) in Unknown on line 0

(the file size was 200MB, base64 included, and the available RAM was 600MB)

The more I increase the memory limit, the less often it happens, but it still happens randomly when uploading the same file over and over.

What should happen

Upload of a 150MB (+base64) file should not fail when having 1GB free memory.

Additional information

Ubuntu 18.04

Apache (from repository)

PrivateBin version: 1.3.4 (latest)

[elrido]

You will need to increase your PHPs memory_limit setting, if you really want to handle such large files. Keep in mind that PHP will store the full paste in memory, plus the decoded JSON and the gzip compression result used to check if it contains encrypted data and the paste is in a valid format. For this very reason the default upload limit is 10 MiB (and was 2 MiB before). I would not recommend to allow such large files/pastes to be stored, as it will exhaust your system with just 2 - 3 simultaneous requests.

This design, using in-memory operations, is fast, avoids complexity and is reasonable for small texts, images and office documents. The tradeoff is memory consumption. For safe handling of such large files, the system would need to be redesigned both on an API level (to support chunked uploads) and backend (to perform the JSON checks in a streaming fashion over a temporary file). I consider this to be out of scope for the project - It's intended to be a pastebin, not a filehosting service at it's core.

[privatebin12]

I understand completely.

It's just that I

1. didn't expect that you cannot at all upload files that are about 1/4 the memory_size
   and
2. noticed some inconsistency. E.g. if I upload a 77MB file repeatedly to a 512MB memory_size installation, it works most of the time, but sometimes fails with the above error.

Since 1. seems to be normal, what could be the reason for 2.?

and perhaps 3. would then be a "feature request" to seek for ways to improve the situation a little, without having to rewrite everything. Or is that not possible at all?

Thanks for your response. I really appreciate PrivateBin the way it is already. I just have a lot of files larger than 10MB, such as uncompressed music or photos, and the usual 128MB web hosting memory_size seems to be not enough for it.

[elrido]

Reg. 2, assuming the POST contains about 200MiB, then that gets loaded into memory once when the request is loaded:
https://github.com/PrivateBin/PrivateBin/blob/fb66e2392bc833b0e43ac8b553e587c06351a2c6/lib/Request.php#L111-L113
then duplicated when json_decode is applied to it:
https://github.com/PrivateBin/PrivateBin/blob/fb66e2392bc833b0e43ac8b553e587c06351a2c6/lib/Json.php#L49-L54
then in validation the ciphertext (which contains the attachment) gets base64 decoded and stored in memory again:
https://github.com/PrivateBin/PrivateBin/blob/fb66e2392bc833b0e43ac8b553e587c06351a2c6/lib/FormatV2.php#L67
and finally compressed as a test for entropy (encrypted data should be indistinguishable from random data):
https://github.com/PrivateBin/PrivateBin/blob/fb66e2392bc833b0e43ac8b553e587c06351a2c6/lib/FormatV2.php#L111-L114
So for that 200 MiB of raw message, you add 200 MiB for the decoded JSON, 150 MiB for the base64 decoded encrypted payload and 150 MiB for the gzip-test requiring up to 700 MiB of memory.

Reg. 3, how it could be improved:

• php://input is already a streaming API and could be read from in chunks
• the PHP builtin json_decode would need to be replaced by a streaming JSON implementation and the output buffered into a file on disk - I assume such a library for PHP exists? This is probably the most tricky part.
• similarly base64 decoding would need to be chunked, but AFAIK base64 can be split into multiples of 6 bits, i.e. 4,8 kiB (=800 * 6 bits) and each could be decoded using the regular function and again buffered on disk
• zlibs gzdeflate could be replaced with the streaming gzopen, gzread, etc. - we only count the bytes that come out of it and wouldn't need to store it yet again
  Downside: Saving pastes would be slower and the DoS potential larger (limited by disk I/O, instead of memory bandwidth). Maybe we could use large chunks of 10 MiB so that small pastes easily fit into memory and remain fast.

If anyone wants to give this a try, I would certainly support such a PR.

[intrepidx]

Hello,

I'm a big fan of PrivateBin, it elegantly fills a niche - it's certainly the kind of software I love self-hosting. I believe I'm impacted by some limitations of the current attachments implementation however. I would like larger and faster, and I have just started reading things over to see if I have any options to self-help.

Regarding "Lines 111 to 114 in fb66e23" (entropy check) - and accepting that encrypted data should appear random as you said - are you sure this check is carrying it's weight? This is an unfortunate computational expense, gzipping what's always expected to be random ciphertext. I don't believe this check is actually sufficient for anything because highly non-random looking data (e.g. the binary digits of PI) would appear quite incompressible to gzip and still pass this test, although subject to very concise description (a small program to emit C/D). To my understanding (and please correct me) this check does not promote any strong cryptographic guarantee of the system.

Does this intend (even loosely as a server-side heuristic) to protect against a client somehow submitting corrupt / adversarial data that is not the output of AES? If so, doesn't the crypto just fail on the receiving client? Isn't that acceptable in such unusual/rare circumstances? If that happens a lot, is this possibly addressed by rate-limiting (or similar?) To be excessively pedantic, maybe a better function name would be "isApproximatelyValid" since you _can never_ know (because of the crypto) whether it's valid AES output or "random fake AES" output... by design!

Asked another way: Why is it the servers job to determine if the random data is the right random data? I think that's the wrong expectation. If the ciphertext is "aaaaaaaaa...a" the decryption is going to fail, and .... life goes on in that rare case. Once a recipient has _successfully_ completed decryption with provided parameters, the regret we feel for doing the slow, loose entropy check is very high IMO, because the user is satisfied with the cryptographically sound answer anyway.

My compliments on this excellent project,

Dave

[elrido]

Why is it the servers job to determine if the random data is the right random data?

Basically, the servers job is to protect the server admin from accusations of hosting data that they could have validated before publishing. If the data is encrypted and the key/password not known to the server admin, that is the case. If one can just upload unencrypted data (see next paragraph), some jurisdictions will oblige the server admin to validate the content before publishing it.

Given that the server, by default, deals with encrypted data, in that default case the server can't know what is in that segment of the JSON, other than that it is something base64 encoded (which is also checked server side).

But it would be trivial to modify one of the third-party CLI clients to use a PrivateBin server to store other types of data, then the intended AES encrypted one, and retrieve it again using the same modified client. A use case for this would be if some people wanted to share data via PrivateBin instances without intent of making it accessible in the WebUI, but only via that modified client. If one happened to open that paste URL in the browser, it would just fail to decrypt, while the users of that alternative client could still access the data. This would serve as an obfuscation (steganography, but in this case in encrypted data instead of an image) from parties searching for that malicious payload on websites, search engines, etc. Those parties and the server operator would just assume the data to be corrupted or such, if they would attempt to look at it.

Anyhow - Yes, the test is weak. If such non-standard data would be stored, it would for example easily pass the test, if it were compressed (be that by storing data like JPEGs or PNG, which do inline compression, or just compressing the data without encrypting it).

How could this be improved?

We could explore if AES encrypted data can be recognized, by attempting to decrypt it in PHP with an empty key (as the server doesn't have the key). Since we are using an authenticated data string, stored along side the encrypted message, we should be able to distinguish between the data failing to decrypt due to the incorrect key or because it isn't actually encrypted data of a type that we expect. This would require openssl support in PHP and the decryption error should be provided via openssl-error-string(). I'd expect this would be the only expected error, when the key is incorrect, but it is otherwise an AES encrypted message:
https://github.com/openssl/openssl/blob/62f27ab9dcf29876b15cdae704c3a04b4c8a6344/engines/e_capi_err.c#L25

Or we could do a MIME magic check (using the Fileinfo module, which should now be bundled by default in PHP >= 5.3) on the Base64 decoded data: If the data is gzip compressed (for example), it should detect it's deflate header, while actually encrypted data should yield an "application/octet-stream".

Other ideas?

[intrepidx]

Thank you for your explanation (sorry for my delayed reply). That's a very interesting use-case that I didn't consider. I trust my small number of users not to upload anything unencrypted to my hidden instance (so I might disable this particular check) but I realize my case isn't typical. Even with such server adjustments, this still leaves open the unavoidable memory limitation in the browser (going back to your previous comments about streaming, which sounds like the right approach, also a big change).

If you haven't looked, the source code to Firefox Send is very inspirational. I loved Firefox Send while it was alive, before it was unjustly killed by Mozilla (except it lacked the also-desired encrypted pastebin, for text description - files only!) The source code is really instructive, they're making use of websockets, there's a service worker to do the decryption/unzip. It's a Node.js backend which is not my preference, I am working towards minimal PHP examples of similar functionality. From PrivateBin's standpoint, I know backwards compatibility is a separate major concern (that I'm still learning about).

So in short, I don't currently have anything to help, but if that changes I will chime in for sure...

Best wishes,

Dave

[rugk]

Yeah FF Send source code is good. Especially their JS encryption on the client though. The server of course should do not much – just as in PB's case.