Owasp-modsecurity-crs: overly broad scanners matching

[ygrek]

Hello,

258 added AhrefsBot (which is a general purpose crawler, not a vulnerability scanner) to the scanners-user-agents.data which contradicts the purpose of this list, afaics :

#
# -=[ Vulnerability Scanner Checks ]=-
#
# These rules inspect the default User-Agent and Header values sent by 
# various commercial and open source vuln scanners.
#

People are using Ahrefs to explore their own sites to find known problems (like duplicate pages, missing title tags, etc). modsecurity is often installed and configured by the hoster without actual website owner control of its configuration, and due to this rule they lose the ability to use Ahrefs services for their own websites. This is rather unfortunate.

Please remove AhrefsBot from the list of vulnerability scanners.

[dune73]

This list of scanners in upcoming CRS3 has quite a few entries for tools which can also be used to legitimate purposes.

AhrefsBot is on the list as it is being known to be used for reconnaissance, vuln scanning and has a bad habit of ignoring robots.txt changes for days on end. (This is information from google. I have not observed the bot in the wild myself.)

I think it is best to issue a rule bypass for AhrefsBot:
SecRule REQUEST_HEADERS:User-Agent "@contains AhrefsBot" "id:XXX,phase:1,pass,nolog,ctl:ruleRemoveById:913100"

[ygrek]

being known to be used for reconnaissance, vuln scanning and has a bad habit of ignoring robots.txt changes for days on end

Said the internets. Seems legit (not really).

I have one more candidate for this list of vuln scanners - googlebot. Why it is not included? And there are actually documented cases when google results are used to search for vulnerabilities, unlike AhrefsBot. What are the criterias to be included in this scanners list?

[dune73]

You do have a point, the list is fairly arbitrary. It's been assembled during the years mostly based on Ryan Barnett's experience. AhrefsBot is a fairly new addition (not even released yet) after a suggestion by a user.

If you think we ought to approach this scanner problem differently, then please reopen this issue and let's have a substantial discussion. Picking out a single UA you think should not be on the list won't cut it as relief is fairly easy as outlined above. Splitting the UA list by severity and pushing AhrefsBot and friends to a higher paranoia level could be such an option. Your contribution to such an effort would be welcome as you seem to have the expertise to bring sense and reason to the list of scanner UAs.

[csanders-git]

This is a good question and @dune73 keeps beating me to the punch so I won't repeat what he said.
Generally I evaluate based on the following (although it isn't formalized (i just put it together now) and maybe it should be):

• Reputation
• How often it's seen in the wild as a nuisance
• Faith in our contributors
• False Positives that would be/are being reported
• If it is a security scanner
• If it can be produced by a normal browser

By this list googlebot would not be added due to reputation, not being seen in the wild as a nuisance, no one contributing it, and massive false positives that would be reported.

Some of these are hard though. For instance urllib and requests are both on the list and are used far more than AhrefsBot. Generally, if the people who feel it is a true positive for their site would outweigh the people who think its a false positive, it's a pretty good indicator. Based on just googling the word "AhrefsBot" I have a feeling most people feel its a true positive. I myself have had customers request that this bot be prevented from scanning their sites in the past. For those that don't, exceptions are easy enough to craft as @dune73 pointed out in his first response.

[ygrek]

@dune73 I cannot promise that I will do a comprehensive filtering of all scanner user-agents and I don't know CRS structure well enough, but I see that there was a list of marketing bots (which also included googlebot) which was removed in 2014 - that would be a proper fit if there is an analogue for such list in 3.0.0

@csanders-git reputation is an interesting topic, I believe for googlebot you are evaluating the reputation of google as a whole, not the googlebot (which is known to be faked constantly actually). In that case Ahrefs is one of the top tools for online marketing (I think it will be present in every "top 10 SEO tools" collections one can find). Also consider that google is a well-known public company, while many bots are run by smaller companies with niche business which one may not know about until doing a research.
The problem with reporting false positives is that users who get hit by this problem may not (1) notice that it happened (2) may not have an ability to understand why it happens (3) even with knowledge may have no permission to fix things - in case of shared hosting where apache is configured by a hosting provider. In best scenario complains will be sent to bot authors, not here.
What bothers me is that sysadmins are enabling CRS thinking of it as a protection from security breaches and hacks but instead will be doing arbitrary censorship for their users based on crowdsourced criteria..

All in all and given the above considerations, I understand the best way to resolve this would be for me to create a pull request and continue discussion with more specifics in there.

Thank you.

[dune73]

@ygrek: Thank you for you considerate followup. Your initial message sounded a bit unfriendly, but this one offers some help and support is what we need.

I agree with you, that it is not very smart to put vulnerability scanners, exploitation tools and search engine crawlers into the same basket. From my point of view, they should be split into separate files. And then applied separately, probably at different paranoia levels for CRS3. (Exploitation tools definitely at paranoia level 1, search engine crawlers including AhrefsBot probably only at PL 2 or 3.

If you would propose such a set of split files, we could then discuss it and I am sure it would find it's way into the upcoming CRS3 release.

We are currently in quite a rush to get CRS3-RC1 out of the door. So the project is a bit unresponsive. After RC1, there will be time to update this for RC2 with your support.

[lifeforms]

I agree with @dune73, I would definitely support moving these kinds of crawlers to a separate list.

@ygrek If you could help us by splitting off some of the more useful crawlers I would be delighted to move them to a higher paranoia level. Then the real paranoid users can still block them.