Cidram: Signature to block

[mikeruss1]

New one ?

User Agent: Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html)

[Maikuolan]

I agree; worth blocking. That particular UA should already be blocked by the "optional security extras module" though. :-)

If it isn't being blocked currently, it's possible that the module mightn't be installed, or might be outdated.

Responsible code of the matching signature:

Block test results:

[mikeruss1]

umm - I dont use that because the description seemed a little obscure, and the risk of false positives medium. To be a little argumentative, shouldnt it be with bots like proximic, admantx etc ?

[Maikuolan]

I see your point.

Those other bots mentioned (proximic, admantx) are both actually also included in the module too, but I agree, that the module description is a little obscure.

Reopening this issue.

My current thinking: Split out the bot-specific UAs into a separate module, just for signatures that target known bot UAs, and nothing else. Everything else in the module can stay where it is. Update module descriptions accordingly. Thoughts?

[mikeruss1]

sounds good to me

I am pretty sure proximic etc are also elsewhere, maybe botspiders_cidram.inc

[ghost]

Just to confirm for mikeruss1:
Semrush, AdmantX and promixic are all in my botspiders file.

Maikuolan, probably a good idea for modularity. Some will surely want the bots only, others are happy for the added security layer. Most probably would want both though, so I think that the most essential is a more specific description for the module(s) to clarify any confusion.

[mikeruss1]

just to explain my interest in this ... proximic etc are necessary in order to get revenue from Adsense. So I need to have a bypass for each occurrence.

[Dibbyo456]

@mikeruss1 why would proximic is necessary? I think AdSense does not rely on others analytics.

For a couple of months, I have whitelisted them but later blocked them again.

[mikeruss1]

depends who you believe, if you look around it is thought that advertisers use them and others to determine the best places for their adverts. I blocked them and others for a long time. Since opening again I have seen an increase in traffic and revenue.

[Dibbyo456]

That’s great. Can you give me a list of bots I should whitelist?

[mikeruss1]

your choice, but I allow Proximic, A6-Indexer, Admantx, and Grapeshot

there are others - see https://www.incapsula.com/blog/most-active-good-bots.html

[mikeruss1]

havent seen this one before

Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

[Maikuolan]

havent seen this one before

Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)

@mikeruss1 I hadn't ever seen that one before either, but I've seen it hitting up several of my honeypots over the past 48 hours, for the first time, and quite a number of times (one of these honeypots is literally just one static page, with no other content, images, or anything like that, and I had a dozen or so requests with that UA within a one hour period to that particular honeypot, for example). The website address cited in the UA isn't loading at all on my end though, and appears to be effectively dead, but a bit of quick googling suggests that it's likely either a variant of the Sistrix bot, or a ¹new, alternative UA for it.

^1. First time I can recall seeing it myself, though the information I've been able to find via Google suggests that it's been in circulation since at least 2014, so maybe not so new really, and maybe just less active than its other UAs, or I've just been lucky/unlucky(?) enough to have not encountered it until now.

I'm thinking I'll add that one to the new bot UAs module that I'm currently working on splitting away from the currently available security extras module (and I should have that done shortly).

[Maikuolan]

your choice, but I allow Proximic, A6-Indexer, Admantx, and Grapeshot

there are others - see https://www.incapsula.com/blog/most-active-good-bots.html

I'll need to take a closer look at that when I've got a spare moment, I think. Right off the bat, I can say that quite a few of the SEO bots listed there (as well as some of the mentioned search engine bots which aren't currently verified, though not all), have UAs which frequently get ghosted by spambots, scrapers, and various others (I can recall a few arguments in the past, on the now defunct SBS forums, where a few of them got blocked, due to being associated with bad behaviour, due to the bad behaviour of these bots which ghosted their UAs, which sometimes, on some rare occasions, caused the supposed authors and operators of the real thing to sign up and argue their case for why they shouldn't be blocked, sometimes nicely, sometimes very rudely, with the same often offered back and such).

It may be difficult to do anything in cases where the real thing doesn't provide any reliable way to be verified, but for cases where it does.. it may be possible to improve the way that CIDRAM handles such requests.

[Maikuolan]

The "optional security extras module" has now been split into two distinct modules.

All user agents previously blocked by that module, are now instead blocked by the newly created "optional user agents module". Nothing else has been changed in the modules though.

Note: Any users that want to continue blocking any potentially dangerous UAs previously blocked by the extras module, should also install this new module.

[Maikuolan]

Regarding bypasses: I've slightly changed the description for the signature that blocks Proximic, SemRush, and Admantx, from "Scraper UA" (its previous designation), to "Backlink/SEO/Scraper UA" (its current designation). This is because the signature in question actually blocks quite a few different things, and some of those UAs, are used by different bots, which try to fake each other's UAs, or bots with different purposes all sharing similar UAs and that sort of thing, and the new designation is perhaps a little more fair than the previous designation (e.g., not necessarily accusing something of being a "scraper", if, although engaging in scraper-like behaviour, it happens to be primarily used for something else, etc). This does mean though that the "Scraper UA" part of some bypasses might need to be changed to "Backlink/SEO/Scraper UA" accordingly to continue bypassing any relevant UAs (assuming, of course, that said bypasses will continue to be needed at all, considering the module split).

[Maikuolan]

Kudos to Grapeshot.. They seem to have a proper verification process in place, similar to Google, Yahoo, et al, so verifying them should be very easy (to block fakes, and to prevent banning the real thing, similar to as we do with Googlebot, Pinterest and such already). I'll add in a verification check for them later today (and I've never seen any bad activity from them before, and all their IP ranges seem clean against the various databases I would usually check against, so I have no problem there labeling them as "good"). :-)

Some of the bots mentioned on that page though, I would disagree about being "good". Anyway, I'll write more a little later.

[mikeruss1]

positive moves I think...

when I decided to open to these SEO bots I checked their activity. They replicate the GET that someone else has used and arrive immediately after it, and only request that page and nothing else.

I seem to remember at least some of these bots provide their IP ranges

[mikeruss1]

should this have been blocked ? .....

User Agent: Mozilla/7.0 (compatible; MSIE2.00; Windows 2002)

[Maikuolan]

Yeah.. I'd wager so. MSIE2 was released in 1995, and MSIE3 was subsequently released in 1997 (thus making MSIE2 effectively outdated since 1997), which is.. 21 years ago now. AFAIK, there shouldn't be any valid, current, modern browsers anywhere citing that as their UA (unless I'm really, really missing something here).

[ghost]

Mozilla/7.0... well that is a bit ahead of time. Back in the days of IE2 the Mozilla token was still at 3 if I recall correctly. Almost all browsers identify as a Mozilla/5 today.

Definitively a case of "should have been blocked".

[mikeruss1]

it did get blocked with Cloud service - but no UA block

[ghost]

Hmm... looking at the UA more carefully... If you pasted it exactly as it appeared, it actually looks like it is tailor-made to slip through many checks.

I think I will update some of my signatures. @Maikuolan you might want to check yours as well if you have anything that should catch it. Specifically look for the lack of whitespace in the IE token f.ex.

[mikeruss1]

yep, copy and pasted

[ghost]

If you use my optional lists then my commit c5e5fb0 should include the necessary changes to catch such an UA.

Thanks for sharing!

[mikeruss1]

thanks, tested and works

[Maikuolan]

All of mine have now been updated. :-)

Keeping this issue open for now as a reminder for myself to finish working through all the UAs mentioned by the page linked above at some point. Otherwise, everything else covered by this issue has been resolved now, I think. I'll close this issue after finishing going through the UAs mentioned by that page linked above (plus any related things that might be added in the meantime).

[Maikuolan]

Should be okay to close this now, I think. I haven't covered everything yet, but those which haven't yet been covered, I haven't really seen many fakes instances of anyhow, so not really super urgent to be able to verify them at the moment anyhow. We can always expand on social media and search engine verification and whatever else needed in the future anyhow, regardless of what happens at this particular issue.

Facebook is one in particular I wouldn't mind adding to the list of support social media platforms, but some more work elsewhere needs to be finished first before this will be possible. So, I'll go ahead and close this now.