Owasp-modsecurity-crs: Rule 960000 blocks file names that contain a ";". However this is a valid character

[xformerfhs]

Rule 960000 blocks file names that contain a ";". However, there are valid file names that contain html encodings like e.g. "D:\x5cTestdaten\x5cÜbersichtspläne.txt". These valid file names are blocked, as well.

CRS version is 2.2.9.

[csanders-git]

Hey @xformerfhs - Finally going back through a number of issues. This one is very interesting. This was originally reported because some webservers and modsecurity (if i recall the facts) were mishandling multipart requests. The specification is overly broad about delimiters which uses the ';' within that area. Therefore in order to prevent these issues we prevent ';' within the filename area. If you are passing filenames that may have legit ';' colons, in this case I think the best situation would be modify the rule.

[xformerfhs]

Hi Chaim

and thanks for your answer. Well, I did not realize that this discussion software changed the file name It shoud have been

Übersichtspläne.txt

This is a valid file name and IMHO the CRS should not prevent ';' within the filename area. Maybe it is possible to fix this by checking wether an ampersand ('&') precedes the semikolon and there are a few letters inbetween.

Regards,
Frank

[csanders-git]

hmmm, good point this requires thought. I am slightly hesitant to touch the multipart form stuff as I know a lot of interesting test cases went into it but i'll leave this open to keep it on my to-do list

[lifeforms]

File names with ; could also trigger OS command injection when passed to a shell. Generally I'm not so happy with them, though checking this does trigger some false positives. :(

[dune73]

The have a similar same issue with single quotes in filenames (-> used a lot in French).

However, that one triggers the MULTIPART error flags which are now in the domain of the recommended rules distributed by the ModSecurity project itself.

[dune73]

This is still an issue in CRS3 at PL1. I think we need to treat this as FP.

[lifeforms]

I don't disagree, it is a recurring source of complaints. Since the default modsecurity.conf also blocks this, even before the CRS rules are executed, we are likely not able to fully solve the issues users are having. Maybe the best we can do is to add this to our FAQ.

[csanders-git]

Well it's a common FP @lifeforms brings up a good point. I also don't really see us switching ';' to PL level2, it is just to commonly used for injection imho

[xformerfhs]

As I said 'ä' is a valid part of a filename and therefor a ';' should not generally be forbidden just because it is abused. This would make this rule simply useless. The specific circumstances should be checked.

As a side note, we will be abandoning modSecurity next year because it is the main reason for software failure of our web services. Mostly the reason is a buggy modSecurity CRS rule and not a bug in our software. So, I really would like to see this fixed.

[dune73]

I do not like the way you word your criticism, @xformerfhs. You are being paid for your work, while two of the three people you are talking to, @csanders-git and @lifeforms, gave their Summer holidays to work on CRS3 in order to reduce the amount of false positives and give you a better experience.

Almost all rules in the CRS work on semantically valid HTTP traffic that can be abused. The difficulty is to find criteria which identify attack traffic and ignore benign traffic. Standard compliance is no such criteria. In the end it is a _Trennschärfenproblem_ and we are weighing pros and cons.

Threats and un-constructive criticism will only spread frustration, so please stop it. If you migrate away from CRS because of false positives, I wish you well. If you care to look into CRS3, then take https://www.netnea.com/cms/apache-tutorial-8_handling-false-positives-modsecurity-core-rule-set/ as a guide on how to solve the remaining false positives of a fresh installation.

We are trying hard to deliver a Paranoia Level1 free of false positives. But we do not want to compromise on Security. So we need to discuss this. Feel free to contribute in a constructive way. But your latest contribution was not constructive and I do not like this.

[csanders-git]

Now @dune73, @xformerfhs has been giving back to the community and for that I thank him/her. I am sorry you have been finding so many issues, we have done our best to address some of them and we are sorry to see you leave. In this case its a particular issue with the report, as we don't have the context to determine if the actual filename is malicious or proper. The issue is of course we can't simply allow a char that is abused in, to do this would create a false negative, which is against the entire point of the rules. It is something we need to consider how to recraft the rule in such a way that we can detect true positives and also remove false positives. Any ideas you have are welcome.

As a side note on many instances we prohibit valid chars that can be used for malicious purposes. A simple example is %00, this is allowed by HTTP spec but we prohibit it as its often associated with null byte attacks. In the case you are mentioning it seems our prohibition has been overly aggressive, but before we remove it completely we must find a better solution. Until such a time, the capability to add exceptions is possible as @dune73 mentioned. thank you.

[xformerfhs]

I am a firm proponent of open software and, indeed, I reported all the bugs I found to make the CRS better. I did not have to do this. I could just have switched to something else without reporting the bugs.

My side note was not diplomatic and I am sorry you found it offensive. That was not my intention. It was triggered by the "we will not change the behaviour" note. You see, I and a few others at my company had a hard time with the CRS. As I said, the CRS was (and in some instances still is) the leading cause of web service malfunction. I thought, maybe you do not know about that, so I wanted to tell you. Not fixing this causes a lot of frustration. I just wanted to let you know. It is entirely your decision if and what you do about it.

There were really many issues: E.g., I had to disable all sql injection and xss rules because they were triggered again and again by our AJAX framework:

SecRuleRemoveByTag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"
SecRuleRemoveByTag "OWASP_CRS/WEB_ATTACK/XSS"

I also had to remove 29 rules which were buggy. And I really hated it when my email program greeted me again with "You have a new ticket"/"Urgent: Client unable to make a bid"/"Very urgent: Web request fails" etc. messages which caused me to spent hours on analyzing only to find out that, once again, there was a CRS rule that blocked totally legit HTTP requests.

If you are interested in it, I could post the exceptions.conf file that was the result of many hours of analysis.

I understand that the ";" can be abused. But I think it should be possible to specify the cases where it is allowed. I am not a RegEx expert, so I can not really say whether my vision of how it should work is even possible, but my suggestion would be to allow something like '&\w+;', i.e., creating some sort of white list of hazard-free uses.

I know, you do your best and I wish you the very best.

Oh, and my name is Frank. Sorry, that I forgot to introduce myself.

[dune73]

Thank you Frank, now we are talking.

Yes, CRS2 are a pain in the ass. I have over 100 services in prod, all running in blocking mode and an estimate of 4K rule exclusions. We do about 40 per service on average. With the ones we have migrated to CRS3, the number comes down to 3-4. Seriously, CRS3 is a huge improvement over CRS2.

If you run into service outages in PROD due to the CRS, there is something wrong with your process. The CRS were published without any documentation on a useful tuning policy, so this is not your fault. But the process seems seriously broken. Like with all the other ModSec setups that I have seen and doing ModSec consulting I get to see a lot.

I have developed my policy after years of experience. And it really took this time to come up with something which works in real life. The key elements are:

• Always work in blocking mode; but start with a very high Anomaly Score Limit
• Highest scoring requests are tuned first
• Work in several iterations

The long explanation and the reasoning is in the tutorial. The process allows to tune a service down to a very tight blocking mode without ever blocking a legitimate request coming from a benign user.

But back to your post. The fact that the CRS have too many false positives are no big news. We know that. Hence the focus of the CRS3 release: Kill the false alarms. You submitted your welcome bug reports in a moment the project was in a drought. Ryan was on the way out and Chaim not yet really in. Walter and I were users, but not committers, so nobody even acknowledged your problems. What a shame.

But when we worked on the CRS3, we should have paid attention to your bug reports. However, we were overwhelmed with work and getting the release out the door was really hard work. And it is not like we did not know where the problems are. We had very clear statistical data where the problems are. If you happen to know this blog post: https://www.netnea.com/cms/2016/01/17/most-frequent-false-positives-triggered-by-owasp-modsecurity-core-rules-2-2-x/ It does the frequency of false positives across the board for CRS2.

It's just that the FPs you reported were surprisingly exotic. In the sense, that most of them are not being washed away like 90% of the the other FPs. So yours seem to stick and now with the release out the door I returned to the issue tracker to clean out all the old issues so we can start over with a clean record and your findings are still there.

So we want to cover them for 3.0.1. And this particular one makes it very hard. Your idea has some merits, but I wonder if we can come up with such a whitelist. There are multiple problems associated with it. One of them is a semantic one. If an attacker couples an Umlaut with a XSS attack, you need to make sure the Umlaut does not trigger the whitelisting so that the XSS goes unnoticed. As it happens, this is very tricky with the limitations of the ModSec Rule language.

We will see what we can come up with. My users use German filenames just like yours and the French speaking part of Switzerland has a habit of using single quotes in filenames, which is even worse.

So let's think some more and we will see.

[franbuehler]

Hi,

I had a look at this issue this week.

Here is my proposal for the rule id 920120 (960000 in 2.2.9):
SecRule FILES_NAMES|FILES "(?<!&[aAoOuU]uml);|['\"=]" \

I’m not quite sure about what is meant by lifeforms and dune73 about the recommended rules.
It’s possible to trigger this rule 960000 (2.2.9) or 920120 (3.0.0) without a multipart processing error.
Maybe this does not happen in reality, but we can solve this issue independently anyway.

I have a solution for the German Umlaute problem. But of course we could extend the rule for more/frequent/all possible html entities.
If we want to just close this issue, it’s solved with my proposed rule. And we wait for other issues or real examples of other users.
But maybe it’s best to proactive solve other problems as well.

It’s not harmless to allow these html entities, but at PL 1 we should avoid false positives.
My proposal is to shorten the regexp on PL2 or PL3 to ['\";=] again. I would propose on PL2.

Now, before I can do a Pull Request, we should come to an agreement:

• Solve only this issue with German Umlaute or in general for more html entities?
• Shorten the regexp to be more strict on PL2 or PL3?

Regards,
Franziska

[dune73]

Thank you for picking this up again.

I second your approach to handle this problem separately from the recommended rules.

If I get you right, you are proposing a pragmatic approach covering the most frequently reported html entities in filenames for PL1 in 920120 (-> tune it down a bit) and then add a stricter sibling at PL2/3 which is in fact the 920120 we have now.

That sounds reasonable to me. Of course, having an all-encompassing global solution would be sweet, but the related recommended rule problem is not solvable for us right now, so the pragmatic approach seems right.

I would probably look through the html entities and cover a few more cases, though. And then we wait and see.

[franbuehler]

I now added the following html entities, of which I think, they are used frequently:

&[aAoOuU]uml;
&[aAeEiIoOuU]circ;
&[eEiIoOuUyY]acute;
&[cC]cedil;
&[aAnNoO]tilde;
&
'

My rule proposal now looks like:

SecRule FILES_NAMES|FILES "(?<!&(?:[aAoOuU]uml)|&(?:[aAeEiIoOuU]circ)|&(?:[eEiIoOuUyY]acute)|&(?:[cC]cedil)|&(?:[aAnNoO]tilde)|&(?:amp)|&(?:apos));|['\"=]" \
  "msg:'Attempted multipart/form-data bypass', \
  severity:'2', \
  id:'920120', \
  ...

First I had some concerns because of the performance of the negative look behind, but this rule is not as expensive as I first thought.
As an example the sqli rule 942410 is a few times more expensive. I only tested with a normal regex tester. I didn't run tests against real content.

Because a negative look behind doesn't allow alternatives of variable length I had to repeat some patterns.

[dune73]

That looks good. I have not done performance tests so far, but I think we need to do real world traffic tests. Glad you seem to agree. Would you mind extending a bit with the &_grave group of html characters?

[franbuehler]

Oh, yes, of course. I forgot them!

[franbuehler]

&_grave; html entities added and &[yY]uml; as well.

SecRule FILES_NAMES|FILES "(?<!&(?:[aAoOuUyY]uml)|&(?:[aAeEiIoOuU]circ)|&(?:[eEiIoOuUyY]acute)|&(?:[aAeEiIoOuU]grave)|&(?:[cC]cedil)|&(?:[aAnNoO]tilde)|&(?:amp)|&(?:apos));|['\"=]" \

[dune73]

Very good. Feel like doing a PR? We could then run it with real tests easily.

[franbuehler]

Great! I will create the PR this week.

[dune73]

691 solves this. Thanks to @xformerfhs for reporting (and sticking around!) and to @franbuehler for the fix.

[csanders-git]

@franbuehler / @dune73 wouldn't it be easier to just do an htmlEntityDecode transformation? this should solve the issue more completely (depending on if the modsec method of htmlEntityDecode is correct)

[dune73]

Yes, you are probably right. However, @franbuehler's new rule 920120 also deals with ' and ", so htmlEntityDecode does not do away with 920120 completely.