Cidram: Wrong Block for duckduckbot.duckduckgo.com

[eurobank]

ID: 1624167506-332675-7976283197 »
Date/Time: Sun, 20 Jun 2021 08:38:26 +0300 »
IP Address: 50.16.247.234 »
Hostname: duckduckbot.duckduckgo.com »
Signatures Count: 1 »
Signatures Reference: module_botua.php:L176 »
Why Blocked: Banned UA! »
User Agent: 'Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)' »

[eurobank]

I have switched the log to daily format, since it seems that many legit bots are blocked. This is the second i have found.

I will review each and every block daily.

[mikeruss1]

I suspect that BOT should only come from specific IPs, like Amazon and MS Clouds.

[eurobank]

@mikeruss1

https://www.abuseipdb.com/check/50.16.247.234

[eurobank]

This is strange since here:

https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

the IP i posted is not mentioned. But the IP comes from their subdomains. Not sure what happens here.

[mikeruss1]

abuseipdb is still counting it as suspicious - maybe fake hostname. If it was the real amazon it would not be suspicious. DuckDuckbot as far as I know only use AWS and MS clouds

[eurobank]

@mikeruss1 But it IS AmazonAWS

[Maikuolan]

The reason for that block is the apostrophes on either side of the UA. There's a signature in the Bot UAs module intended to catch UAs which follow that pattern, due to many spambots, hacktools, etc which follow that same pattern.

I'm not sure why DuckDuckBot seems to be suddenly now following that pattern, seeing as they've never followed that pattern before AFAIK, but the IP address which that request originates from is one of the IP addresses which DuckDuckBot uses, according to their own information.

Worth noting too, that this block event occurred because of a module, but not due to search engine verification, and that CIDRAM's search engine verification mechanism seems to have had no problem with the request at all.

I would be inclined to say that this is a legitimate DuckDuckBot request (and therefore, that the block event is a false positive). However, after seeing the reports from AbuseIPDB.. I'd be really curious, and interested, to see what the exact justifications were for some of the comments attached to those reports.

"Attacks websites by trying to access known vulnerables of plugins, brute-force of backends or probing of administrative tools."

Access logs would've been nice there, to have a better idea about exactly what kinds of "attacks" and "vulnerabilities" the report is referring to (though wishful thinking, I know, and there's also a character limit for the comments included with AbuseIPDB reports, which limits how much evidence could be theoretically provided in a single report, unfortunately).

"Exploiting vulnerabilities."

Ditto. Access logs, etc, would've been nice.

"Lurking piece of shit."

Okay.. I'm actually not even entirely sure what that is supposed to mean. What qualifies something as being a "lurking piece of shit"? I have no idea. But whatever.

But, based on the reports which do actually provide some access log snippets, I'm guessing there were some XMLRPC attacks from that IP address (e.g., checking for XMLRPC vulnerabilities at WordPress websites). XMLRPC attacks are a good reason to block a request, and a good reason to block IP addresses from which such attacks originate.

I'm disinclined to believe that a legitimate DuckDuckBot request would be responsible for such requests. But, the reports exist, and the IP address in question is, according to DuckDuckGo itself, one of the IPs used by DuckDuckBot. Given, however, that the IP address in question belongs an AWS data center (i.e., owned and operated by Amazon), there exists the possibility that other AWS customers are using that same IP address, and maybe someone there has figured out that they're operating from the same IP address as one of DuckDuckBot's IP addresses, and has decided to exploit that as a cover for their own online misbehaviour. (But, I'm just speculating on that, and have no evidence either way).

So, maybe this is DuckDuckBot, or maybe this is something else, which is simply operating from one of the same IP addresses as DuckDuckBot, and pretending to be DuckDuckBot in order to exploit operating from the same IP address as a cover. I'm not sure.

While writing this, I've already written a bypass for it, which I'm ready to commit at any time. Not sure whether I should though, since I'm not sure whether I would actually be bypassing DuckDuckBot, or just something pretending to be DuckDuckBot. It is one of their IP addresses, but that's also an AWS IP address, which could be being shared with unrelated malicious entities, which are just pretending to be DuckDuckBot.

Thoughts..?

[mikeruss1]

looks suspicious to me, and Abuseipdb support that view

[eurobank]

@Maikuolan "but the IP address which that request originates from is one of the IP addresses which DuckDuckBot uses, according to their own information".

Nope, the IP is not there (or i,m blind, very possible in my age)

[Maikuolan]

Nope, the IP is not there (or i,m blind, very possible in my age)

[eurobank]

https://cleantalk.org/blacklists/50.16.247.234

Apparently people report whatever they think it is malicious and it is not. It can't be spam from that IP.

[eurobank]

https://seoapi.com/duckduckbot/50.16.247.234/

[Maikuolan]

abuseipdb is still counting it as suspicious - maybe fake hostname.

Well, it doesn't reverse back to itself, which fits the bill for how I'd normally define a hostname as being "fake". So, fair call.

50.16.247.234 resolves to duckduckbot.duckduckgo.com, which doesn't resolve back to anything at all. The parent domain, of course, just resolves back to 20.43.111.112 (DuckDuckGo's main IP address, and the IP address for their homepage).

That said.. Not useful for search engine verification at least, since DuckDuckGo itself recommends just checking the IP address against their provided list, rather than performing a full reverse-DNS lookup (Googlebot, Bingbot, Baidu, Yandex, etc all recommend performing a full reverse-DNS lookup, so DuckDuckGo is the odd one out there; though understandable, seeing as they're using AWS and Azure addresses, which are controlled by Amazon and Microsoft respectively, meaning that DuckDuckGo has no control over the hostnames for those addresses).

[Maikuolan]

I think, I'll go ahead and commit the bypass. Can't be 100% sure whether it's a legitimate DuckDuckBot request, but even if it's not, if the entity which is pretending to be DuckDuckBot tries something bad, hopefully one of the other signatures should catch it anyway, or we should at least be able to notice if something goes wrong as a result of letting it through. Seeing as the signature in question is only matching against user agents anyway, rather than specific behaviours or specific addresses, the risk of harm in bypassing it specifically for user agents which also match against DuckDuckBot should be very minimal, and the risk of accidentally blocking search engines is more problematic anyway.

[Maikuolan]

Done. :-)

[eurobank]

I think this is ok to fix. Despite the reported issues (that some are just reports from inexperienced users at various sites), i would take some risk (i doubt if any) instead of banning any legit SE. And especially that duckduckdot.