Owasp-modsecurity-crs: Irresponsible Choices for User Agent Scanner List

[nicksloan]

The logic behind the security scanner user agent rule seems fundamentally flawed to me, and actually irresponsible.

One inclusion is the "python-requests" user agent. This user agent is the default value supplied by an extremely popular HTTP request library for Python: http://docs.python-requests.org/en/master/

Rules like this are a slippery slope that encourages attackers to co-opt more specific user-agent strings instead, and while no one would ever add major browser or search crawler user agents to this list, it is not hard to imagine that small, yet legitimate, well-behaved crawlers might have their user agent misappropriated and subsequently added, which could in turn harm their business or reputation.

210 demonstrates that python-requests was added with absolutely no scrutiny, and if that is typical of how business is conducted in this project, and within this company, I think it's fair to describe this project as a sham.

I really, really hope I'm missing something here, but I suspect I will be disappointed.

[csanders-git]

@nicksloan well I don't necessarily appreciate the way you've brought forth your concerns I think they are valid and therefore deserving of a response.
Perhaps your most valid concern is that we are encouraging attackers to become more complex. I actually take great pride in this. While historically I myself have made similar arguments it turns out, as you might agree, that many attackers are lazy. As a result if we can get CRS to block even 50% of lazy attackers very cheaply, I think this is a big win. Similar arguments are made when discussing almost every open source security defense mechanism. I don't think that the community gains anything by not forcing attackers to put forth the most amount of work necessary to be successful.
Onto the specifics, python-requests. The question as to why we added python requests so liberally is due in part to the existence of python-httplib2 within the list. In general CRS is designed for a common web server administrator, common administrators expect traffic from web browsers or other well formed user agents. While we are aware of python-requests (in fact we use it within our unit tests), general testing of this rule when it was added showed that a number of attackers did indeed use this user-agent. Additionally, CRS is designed to be configured, if you find that you often use CRS in an environment with Python-Requests, it makes sense to add an exception (which ModSecurity is designed to do easily). Generally, i'd hope you agree that most users would not use or expect to see python-requests as a day to day user-agent for the predominant amount of web traffic. This is what these rules are meant to accomplish, they are not intended in all cases to indicate that the library is malicious, merely not the normal use case.
Onto the project as a whole. This is an open source project, you are invited to submit pull requests that will be evaluated by the project leader/collaborators to determine if a rule/data is too specific, in the very same way the author of #210 did. In general this conversation often, and at the behest of the requester may, involve the community and can be accomplished via the mailing list.
I invite you to join our community constructively. Before you do so, you should note that the OWASP CRS project, although under the SpiderLabs git group is actually not involved with Trustwave, save that a Trustwave employee happens to be the project lead. It is designed (but not exclusive to) a product that Trustwave owns the trademark for in ModSecurity.
I hope you are not disappointed, but please do consider the average use case, keeping in mind that in even a slightly more advanced use case, if these become a false positive they can easily be removed. Generally we feel it is better to be reserved in this sense, but perhaps this might be a candidate for a higher paranoid level.

[nicksloan]

@csanders-git I apologize for my tone, but I passionately disagree with the decision to include rules like this. One of my servers had just been reported to AWS for abuse due to this rule despite our efforts to be well behaved in other regards. As a side note, using a more specific user agent is intended for an upcoming release.

The problem with forcing attackers to be more sophisticated in this instance is that it doesn't scale well. Attackers may be lazy, but they aren't completely stupid. If the default user agent for various libraries starts getting rejected, their next step won't be to make up some other unique user agent that you can add to the list. They will simply spoof a known-good user agent and move on.

The trouble is that the security benefit is extremely low at best, but the damage it does to good web citizenship is significant.

Consider, for example, that if I know that malicious user agent strings will be added to a list like this, and I am able to discover a competitor's crawler's user agent string, I can potentially upset their operations with targeted malicious, or malicious-looking activity. The whole idea is just too gameable.

On the other hand, if user agents are not given special significance by projects like this, they are less likely to be misused. I'd argue that most user-agent abuse that exists today comes from the common misunderstanding that a user agent actually means anything of significance.

I also take issue with the notion that it is abnormal in the average use case to see user agents that are not browsers and the most well known bots. I fear that attitudes like this from respected projects will lead to user agent paranoia, and will force every new service to lie about who they are.

I apologize for the aggressive tone in my report, but I don't think we are going to see eye to eye on this issue at all. I think this question likely warrants broader debate over the merits of this approach.

[kennethreitz42]

They will simply spoof a known-good user agent and move on.

Yep, just grab whatever your browser is sending. Always go-to UA.

I'd argue that most user-agent abuse that exists today comes from the common misunderstanding that a user agent actually means anything of significance.

This.

[csanders-git]

This is exactly the point @kennethreitz, user agents are easily spoofed client side elements, as a result we don't rely on these alone, they are just part of the greater anomaly score. However, an attacker can always put MORE effort into hiding their presence - to say that they always do is disproven by the evidence, which is why these have been added to the list. Again if an admin wants to use any UA string on the list for internal testing, or as a UA, it is as simple as adding an exception - the typical use case for the vast majority is that these are abnormal requests. The argument that they are easy to spoof therefore we shouldn't even look doesn't seem to equal being "Irresponsible"

[dune73]

This user-agent list is one of many, many rules. A hit on UA is only a _hint_ something could be amiss. It's the combination of alerts, which drive up the anomaly score and thus the indication you are facing an attack. The crs are thus best used in anomaly scoring mode.

I do not want to re-iterate @csanders-git, but second his argumentation.

One thing which has not been brought up is splitting the UA list into two distinct files, and enable the common http library user-agents only on a higher paranoia level. See here for the details. The paranoia level construct is in active development at the moment and you would be very welcome to submit two split files.

[kennethreitz42]

@dune73 ah, splitting into a separate level of paranoia (e.g. making it opt-in, as opposed to default) sounds like a great solution.