Privatebin: wrong content type

Created on 28 Mar 2017 · 13Comments · Source: PrivateBin/PrivateBin

Steps to reproduce

set up a server with modsecurity and set a limit on SecRequestBodyNoFilesLimit

What happens

When uploading a (large) file, the web-server gives me this error message;
«Request body no files data length is larger than the configured limit (1310720).. Deny with code (413)»

With modsecurity-audit-logging I saw this:
Content-Type: application/x-www-form-urlencoded; charset=UTF-8

According to https://support.tigertech.net/413-error, this is because the data in the POST request is not the correct type. "Content-Type" should be "multipart/form-data".

This limit is also described in https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual#SecRequestBodyNoFilesLimit and it exist to prevent (D)DoS attacks on public web-forms.

What should happen

Uploaded file should been saved.

Basic information

Apache 2.2 on debian wheezy

1.1

I can reproduce this issue on https://privatebin.net: No (it does not support file upload)

full mod-security-audit-log:

--0b54db46-A--
[28/Mar/2017:17:32:32 +0200] WNqCEMI-@WYAAjnbEBwAAAAJ x.x.x.x 45557 10.0.249.
102 443
--0b54db46-B--
POST / HTTP/1.0
Host: xxx
X-Ds-Client-Ip: x.x.x.x
Connection: close
Content-Length: 5836760
Accept: application/json, text/javascript, */*; q=0.01
Origin: https://www.xxx
X-Requested-With: JSONHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/56.0.2924.87 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.8,nb;q=0.6,no;q=0.4,sv;q=0.2,da;q=0.2


--0b54db46-F--
HTTP/1.1 413 Request Entity Too Large
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 238
Connection: close
Content-Type: text/html; charset=iso-8859-1

--0b54db46-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>413 Request Entity Too Large</title>
</head><body>
<h1>Request Entity Too Large</h1>
The requested resource<br />/<br />
does not allow request data with POST requests, or the amount of data provided in
the request exceeds the capacity limit.
</body></html>

--0b54db46-H--
Message: Request body no files data length is larger than the configured limit (1310720).. Deny with code (413)
Stopwatch: 1490715152355197 12222 (- - -)
Stopwatch2: 1490715152355197 12222; combined=77, p1=71, p2=0, p3=1, p4=0, p5=4, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.6 (http://www.modsecurity.org/).
Server: Apache/2.2.22

--0b54db46-Z--

question

Source

idarlund

Most helpful comment

"the server should be agnostic on what he stores" - this is something you should consider a wishlist item, that will never get implemented on servers.

Servers are never going to be agnostic about the data they receive.

Here is a possible way out:

Why not treat all JSON structures as "files"?

The filename could be a cryptographically generated string, and don't need to reveal information about the file content.

jani on 31 Mar 2017

👍2

All 13 comments

These are two distinct issues, IMO:

Same as for nginx you have to set the client_max_body_size setting to match the maximum file size I want to allow being uploaded on the server, with modsecurity you will need to adjust SecRequestBodyNoFilesLimit so that it matches the upload limit you want to allow.
You are correct, that when we use jQuery $.post() requests, they are sent as application/x-www-form-urlencoded and not as the application/json that would match the content. This is not really a problem, since our php server application does not check on that. We would need to write our own wrapper to solve that in jQuery, but this is only a cosmetic thing and not a technical problem for our application. Switching to multipart/form-data would be the incorrect MIME type and one still needs to configure a matching amount of POST data on the server.

elrido on 29 Mar 2017

Your solution for the first issue is regrettably non-workable, as the point of SecRequestBodyNoFilesLimit is, among other things, to reduce risk of overloading a server due to maliciously large content. The common limit for this is 128 KiB, so relying on it as something more seems somewhat weird.

Regarding the second issue, you seem to have missed a change in how file uploads are handled on the web, that was introduced 22 years ago, and subsequent HTML standards. Please see RFC 1867 from 1995, and https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1 for more details.

If you need additional sources, please see http://php.net/manual/en/features.file-upload.post-method.php and https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types

jani on 30 Mar 2017

The POST used by our JSON based REST API isn't a form upload. It is a REST API call and yes, it transfers application/json data, not application/x-www-form-urlencoded or multipart/form-data. The form upload documentation does not apply here at all.

Regarding raising that limit, yes, if you raise that limit there is an increased risk for your server to get overloaded with too many malicious POST requests. Either:
a) increase your servers capacity, eg. using a load balancer and multiple application servers behind and increase that post limit or
b) reduce the sizelimit of PrivateBin to match the allowed POST size on your webserver.

In any case, all these transfer limits - the PrivateBin sizelimit configuration, the webservers maximum POST size (LimitRequestBody in apache, client_max_body_size in nginx), PHP maximum POST size (post_max_size and upload_max_filesize in your php.ini) and, as in this case, the security plugin's maximum POST size - all need to match (or be larger then PrivateBins sizelimit) for the upload to work. This is exactly the same as with any other WebCMS that offers some kind of file upload to the server.

Basically what I get from this request is: My webserver is configured to only accept 128 KiB POST requests. If that is the case and it is not an option to change that setting, then yes, you will not be able to upload messages larger then 128 KiB with the current API.

elrido on 30 Mar 2017

It may not be a form upload per se, but it is a file upload, and as such, should adhere to the standards for file uploads.

You shouldn't just cram your own custom file upload solution on the Internet, you should follow the standards.

Fiddling with size limits in various software is only a temporary mitigation, it's not a solution.

The solution is to fix the code to perform a file upload.

jani on 30 Mar 2017

😕1

Regarding "custom file upload" and "RFC 1867 from 1995" I would kindly ask you to learn some more recent concepts we have come to use in the decades since, like RESTful APIs and content-negotiation, in our case a JSON-LD based one. And in general please read up on RFC 7231s section on POST and RFC 2045 section on content-type headers for more recent information on the topic.

Regarding "temporary mitigation": According to your judgement all of the below CMS apparently don't do file uploads correctly, because they all explain that you need to adjust your PHP and webservers settings according to your needs:

If you decide that your webserver should not accept POST requests larger then 128 KiB then yes, PrivateBin will work with pastes up to that size, but no more. Or please share with us, how you would circumvent that limit. Switching the content-type won't reduce that size...

elrido on 30 Mar 2017

👍1

You seem to be missing a point here. POST requests in themselves are not limited to 128 KiB anywhere that I know, but the 128 KiB limit affects POST data except file uploads.

When you insist on uploading files without using the file upload methods, you run into this 128 KiB limit (or in @idarlund's case, 1.28 MiB). Otherwise, you get to the limit that you keep focusing on, the POST data limit.

The default POST data limit for PHP is 8 MiB, and is not typically user-changeable, but most hosting providers that I've run into, limit it to 32, 64, or 128 MiB.

The default limit for POST data in ModSecurity is 12.8 MiB, more than sufficient for PHP's defaults, and hosting providers that change PHP's limits typically also adjust this limit. Ditto for Apache/Nginx limits.

If PrivateBin were to do file uploads like almost everyone else does, following the standard for file uploads, this problem would be less, you would instead only get problems with the general POST limits, as almost everyone else.

jani on 31 Mar 2017

From the servers perspective in the case of PrivateBin there is no difference between a regular paste and a paste with file attachment. The data sent just tends to be larger.

Of course we can't do what you call a "regular" file upload, since that would imply the server receives unencrypted data, filename and other information on the contents of the file. But the whole point of this project is that the server should to be agnostic on what he stores. It just receives a more or less large JSON structure according to the API specification and will store it if it matches certain criteria.

elrido on 31 Mar 2017

"the server should be agnostic on what he stores" - this is something you should consider a wishlist item, that will never get implemented on servers.

Servers are never going to be agnostic about the data they receive.

Here is a possible way out:

Why not treat all JSON structures as "files"?

The filename could be a cryptographically generated string, and don't need to reveal information about the file content.

jani on 31 Mar 2017

👍2

"the server should be agnostic on what he stores" - this is something you should consider a wishlist item, that will never get implemented on servers.

This is something you have as "Key features" item, PrivateBin has it implemented right now, service has zero knowledge of anything except date, size (after encryption), and eventually some metadata. Server must be agnostic on what he stores, because he has no other option.

Servers are never going to be agnostic about the data they receive.

Any server implementing end to end encryption must be agnostic about data, because data is encrypted. Server may not be agnostic about way which data comes and formatting of container.

Rest of your comment - +1'd, AFAIR it's easy doable with vanilla jQuery. We can push encrypted blob via multipart form.

marsjaninzmarsa on 3 Apr 2017

I am sorry, but I do not agree with @jani's view on how large data (>128 KiB) on the web should only ever be transferred using multipart/formdata encoding. I have documented two different ways how the original problem of @idarlund can easily be resolved and am closing this issue.

elrido on 7 Apr 2017

Are you for real @elrido ? You have not documented any solution on the original post. Both are non-workable and that is what @jani is trying to get you to understand. @jani, @marsjaninzmarsa and I seem to agree on that.

Please re-open the issue so that it can be fixed.

idarlund on 7 Apr 2017

Dear @idarlund, the solution to adjust the apache mod configuration was documented by @jani. Changing the mime type to an incorrect one would break several RFCs and would not fix your the problem, as your server would be still as vulnerable to DDOS attacks as with the current mime type and a raised limit in that one mods configuration.

This being open source software, you are of course free to implement any other solution and I certainly will reconsider a PR on this, if it does something more clever then messing with the mime type, e.g. by extending the API with a chunked upload feature or some other mechanism that allows circumventing the usual server upload restrictions.

elrido on 11 Apr 2017

Dear @elrido, as I have already attempted to explain to you, changing the Apache mod configuration is not an option that is open to everyone. It's open if you're self-hosting your own Apache server, and only then.

Additionally, "break several RFCs", yes, that's what your decision does, by "messing with the mime type" in a non-standard, incompatible manner.

jani on 16 Apr 2017

Was this page helpful?

0 / 5 - 0 ratings