Cidram: Fake petalbot?

[eurobank]

Can you please check and verify if it is fake or not?

ID: 1629286057-198436-9265357264 »
Date/Time: Wed, 18 Aug 2021 14:27:37 +0300 »
IP Address: 114.119.133.251 »
Hostname: petalbot-114-119-133-251.petalsearch.com »
Signatures Count: 1 »
Signatures Reference: functions.php:L1580 »
Why Blocked: Fake PetalBot! »
User Agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) »

[Maikuolan]

Not fake. Looks like a false positive. I'll fix this now. Cheers.

[Maikuolan]

BTW, for anyone wondering how I managed to allow this kind of false positive to occur, at Aspiegel's own documentation for verifying PetalBot, they write:

1.5 How to Judge PetalBot Crawling

You can verify if a web crawler accessing your server really is PetalBot.
To verify PetalBot as the caller:

1. Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
2. Verify that the domain name is in aspiegel.com.
3. Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.

Special note: "aspiegel.com" mentioned in step 2, but no mention about "petalsearch.com".

Back when I'd first implemented verification for PetalBot, I based the implementation entirely upon their own official documentation (seeing as the official documentation for such things should, in theory, be the most accurate and trustworthy source of information, although in reality, it sometimes isn't; Bingbot is a particularly notorious culprit for that).

They currently have an example included below that text, and "petalsearch.com" is cited in that example (though still not anywhere in the actual steps). I don't recall seeing that example at all back when I'd first implemented PetalBot verification. I'd like to say it wasn't there, but maybe I just didn't look properly and didn't notice it; I'm not sure. My immediate thought is, "maybe they forgot to mention the second domain, so then later decided to add the example, but still forgot to update the actual mentioned steps?", but I don't have evidence for that, and could possibly just be wrong about it.

The example:

$ host 114.119.128.10
10.128.119.114.in-addr.arpa domain name pointer petalbot-114-119-128-10.petalsearch.com

$ host petalbot-114-119-128-10.petalsearch.com
petalbot-114-119-128-10.petalsearch.com has address 114.119.128.10

Anyway, seeing as I'd based the implementation on what they wrote for the steps, it would check for "aspiegel.com", but not for "petalsearch.com". But, due to the presence of the example, as well as evidence (e.g., your reported false positive here and connected traffic events elsewhere), it's obvious currently that both domains are valid domains from which PetalBot may potentially send its requests.

My fix will have both domains included as valid domains. :-)

And, in the unlikely event that the developers of such bots somehow end up reading this discussion here at some point in the future: Please keep your bot's documentation as up-to-date and accurate as possible! False positives like these are one of the reasons why accurate documentation is important. </Rant> '^.^

Anyway.. Back to the fix. I'll commit the fix and reply back here very shortly.

[Maikuolan]

Done. :-)

(Just noticed too, that the link cited in the user agent seems to be a verbatim copy of that same aforementioned documentation. I'll add that link into the YAML file a little later too, seeing as each could serve as a mirror for the other).

[eurobank]

ID: 1630346543-139348-7585815522 »
Date/Time: Mon, 30 Aug 2021 21:02:23 »
IP Address: 114.119.151.1 »
Hostname: petalbot-114-119-151-1.petalsearch.com
Signatures Count: 1 »
Signatures Reference: 114.119.128.0/19 »
Why Blocked: Cloud service ("HUAWEI CLOUDS", L26284:F0, [SG])! »
User Agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) »

Doesn't seem fixed, can you please check again?

[Maikuolan]

Seems like I'll need to implement a bypass for PetalBot at the Huawei Cloud signatures. Working on it now.

[Maikuolan]

The bypass has now been added, but when I test the new changes at my end, I notice that the AbuseIPDB module is now blocking the IP address, due to the listings there. So, this false positive is partially fixed now, but I think I'll need to make some further adjustments with that module (and possibly others) to fix it thoroughly. I'll keep this issue open for now.

[eurobank]

@Maikuolan You mean the block is for those you use AbuseIPDB module? Or not?