Cidram: Question about Robots.txt

[eurobank]

I would like to ask if before any block from ANY module (optional or not) happens before robots.txt access.

I ask that because i see many bots from legit (?) bots to be blocked but they are already Denied in my Robots.txt. So apparently there is no check if some bot or search engine FIRST access robots.txt. So bot/SE never know if allowed or not.

Can you pls help on this?

[737simpilot]

Robots is a text file. CIDRAM ONLY protects PHP files. I'm not sure how robots can be protected via CIDRAM.

[eurobank]

@737simpilot Not what i asked.

[mikeruss1]

its up to the bot to access robots.txt, and up to the bot to obey or not, No overlap with Cidram.

[eurobank]

@mikeruss1 But if the bot tries to check 1st robots.txt and CIDRAM blocks it ? The bot can't access robots.txt

Am i wording this right? It seems i get all kind of unrelevant replies here.

[mikeruss1]

Cidram ignores robot.txtcheers.... MikeOn 12 Jun 2021 22:28, eurobank @.*> wrote:
@mikeruss1 But if the bot tries to check 1st robots.txt and CIDRAM blocks it ? The bot can't access robots.txt
Am i wording this right? It seems i get all kind of unrelevant replies here.

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[Maikuolan]

I'm not sure how robots can be protected via CIDRAM.

It's possible to do this if a website owner chooses to run all their scripts through a "router" (not talking about a hardware router, but rather, talking about a PHP router; same word, but two completely different things). A "router" (in this context) stands before any/all other scripts/software/etc at a website (typically, just a single entry point only will be provided), and all 100% of all traffic to that website is redirected (typically, via .htaccess) to the router. The router then decides what to do with that traffic, whether to drop it completely, whether to forward it on to another script, etc. Many PHP frameworks and CMS operate as routers (e.g., WordPress). It's also possible to have CIDRAM handle robots.txt requests by creating a simple PHP script which calls CIDRAM first, and then outputs the content intended for such requests (assuming they haven't yet been blocked by CIDRAM by that point yet), and having any direct requests to robots.txt be redirected to that newly created PHP script via .htaccess. Similar things are possible for most other non-PHP content at websites too, if websites owners want to do that.

This can actually be a really great idea for honeypots, really useful for anyone seeking to gain deeper insights into the exact behaviours of bots, spammers, hackers, and whichever other malevolent/unwanted traffic occurs at the websites in question, and potentially result in us having much more information to utilise for making better decisions about what to block or not block.

However, I would recommend caution here, because there are potential security risks involved (depending on whether the context allows for non-PHP content to be directly handled by PHP, and especially if that content originates from third-parties and/or hasn't been vetted by the website owner with extreme scrutiny), and having PHP handle everything at the website, in many cases, is likely to incur very noticeable (and sometimes quite severe) performance degradation at the server in question. I would also strongly recommend against it at very high-traffic websites, any websites offering essential services, or for that matter, any websites where it simply isn't necessary.

[Maikuolan]

But if the bot tries to check 1st robots.txt and CIDRAM blocks it ? The bot can't access robots.txt

Am i wording this right?

Understood, and no worries. :-)

(Couldn't reply earlier due to work. Got back home not too long ago today).

Generally, CIDRAM won't ever block any requests to robots.txt, regardless of where the traffic originates, bot or otherwise. This is because generally, CIDRAM isn't even aware of such requests in the first place, due to such requests not being routed through CIDRAM. However, there are some exceptions to this (note my previous reply about routing). Depending on the kinds of software/frameworks/CMS/etc which you use at your website, it's theoretically possible that those requests may be being routed through CIDRAM, and that CIDRAM may therefore be aware of such requests (and therefore, may potentially block some of those requests). This is particularly true for any systems which utilise their own internal "routing" features (e.g., WordPress). Though, in the complete absence of any such systems, any .htaccess redirections, any routing scripts and etc (i.e., totally vanilla/standard/traditional kinds of websites), CIDRAM shouldn't ever be aware of such requests.

For in the event that CIDRAM is aware of such requests, and subsequently encounters requests from search engines for robots.txt, as to whether it would be a problem, depends on a few factors.

• Does CIDRAM's "search engine verification" feature support that search engine? If so, then, legitimate requests from that search engine should never result in that search engine being banned by CIDRAM (although, even in such cases, it's still possible for it to be merely blocked for a particular request if some installed module has reason to block it, e.g., behavioural-based signatures, request patterns matching known malicious bots, hackers, etc). If not, then, requests from that search engine would be treated just the same as any other request, with no respect to its status as a search engine.
• Does any of CIDRAM's signature files have signatures matching the IPs from which those requests originate? If so, then that's obviously a problem, since it would be blocked. If not, then it shouldn't generally matter (assuming the requests aren't blocked by modules).

[eurobank]

So, how do we know if a bot respects robots.txt if he can't access robots.txt ?

[Maikuolan]

So, how do we know if a bot respects robots.txt if he can't access robots.txt ?

I don't think it would be possible to know with certainty.

It may be possible, in some cases, to rely on testimony from others (e.g., posts at forums written by people which claim a bot does/doesn't respect robots.txt, based on their own experiences, or third-party websites which track that kind of information; there are a few of such websites around, such as udger.com, which I've found to be reasonably useful a number of times before, although there aren't any 100% reliable such websites around, I think). However, newly discovered bots will most likely have only very limited information publicly available like that, or no information at all, and we also can't always be sure how reliable the information is to begin with anyway (e.g., someone reports wrong information, or the bot has updated since the time of that information and behaves differently than before, rendering the information useless, or someone intentionally shills for the bot, for example; and from my own experiences over the years, I've encountered all three of those examples before).

It may be possible to speculate, purely by analysing our access logs, even if bots aren't able to access robots.txt: Checking whether they at least attempted to access robots.txt before accessing anything else (even if they've been blocked from doing so). Such a method, at least, will tell us which bots are not even bothering to try accessing robots.txt. However, such a method is not a reliable way to determine whether they adhere to robots.txt, because there are many bots, too, which will make a point of accessing robots.txt before accessing anything else, in order to appear as if they're adhering to it, only to then completely ignore it. Some particularly malicious bots will even access robots.txt for the purpose of actively seeking out sensitive resources, and any resources/pages which they aren't allowed access to (we've confirmed this in the past by inserting "Deny" clauses into our robots.txt files, which pointed to hidden honeypot resources, which were not linked anywhere other than from the robots.txt file itself, and should've therefore been nearly impossible to locate normally through any means other than through actively probing robots.txt for denied resources; and subsequently seen certain bots firstly accessing robots.txt, and then almost immediately after accessing the denied resources in question; the lesson there being to never link there any truly sensitive or vulnerable resources, since it'll tell the good bots, but also the bad bots, too, exactly where those resources are located).

The only way to be 100% certain, I think, is to be able to put such bots to the test, providing them with a well-written robots.txt, and analysing our access logs, the behaviour of such bots, both prior to and subsequent to accessing robots.txt (or their lack thereof), and attempting to make such determinations ourselves. Which, of course, isn't possible if they can't access robots.txt to begin with.

[eurobank]

@Maikuolan First, many thanks for spending time to explain to me. It is crystal clear now all the benefits and limitations.

[Maikuolan]

You're welcome. :-)

[eurobank]

@Maikuolan

The following happened. A few days back i used robots.txt to ban 2 SE bots. In all my sites but one.

Today i see that there are no more visits blocked (no visits that is) for those i edited robots.txt with deny and only for the one i left to allow.

That can only mean the robots.txt is not blocked in any way ever from anything. IPs, modules, whatever.