Cidram: Auto block scrapers (rate limiting)

[glennish]

I suffer from lots of scrapers that slip through the net. Sometimes it's obvious, other times I notice they are trying very hard to get around CIDRAM. Is it possible to add a feature where is automatically adds an IP to a blocklist after X amount of clicks/pageviews?

[Maikuolan]

It sounds like what you're looking for is a "rate limiting" feature, I think. This would definitely be possible to implement.. but would also be something you'd want to be quite careful with when using, I think, because as well as the scrapers and other unwanted traffic, it could also potentially block particularly enthusiastic human visitors and other more wanted traffic, in cases where a normal, human visitor happens to access X too many pages within Y given time period beyond whatever limit gets defined or configured as the rate limit.

One of the other big problems, is that search engines and related crawlers, in the context of rate limiting, can easily be indistinguishable from scrapers if care isn't taken when implementing them (and to be certain that search engines don't get blocked, any rate limiting feature would probably need to be tightly coupled with search engine verification, which isn't generally a problem.. except when it is, for some specific servers, which sometimes have obscure, specific limitations, or for search engines like Bing, f.ex, which frequently and ruthlessly break their own rules regarding how we're supposedly meant to verify their requests).

My general feeling about implementing a rate limiting feature to CIDRAM, is that it would technically be very easy to do, and could work, but that there'd be a very high chance of unforeseen problems arising as a consequence of it, and that such a feature would inherently present quite a high risk of false positives.

If I implemented this, I'd likely also need to mention something in the documentation about the possible risks, including some warning that users use it "at their own risk", I think.

Let me think on it a bit, and I'll see whether I can come up with something workable, or any ways to improve the idea at all.

Also, if anyone else wanted to share their input, they're welcome to do so.

[soumsps]

What is differnce between Rate Limiter and Infraction Limit?

CIDRAM Documentation says this..

"infraction_limit"

Maximum number of infractions an IP is allowed to incur before it is banned by IP tracking. Default = 10.

Cloudflare has rate limiter that can be configured to ban any user. If user made X no. of page views in Y minutes.. and block will remain active for Z minutes...
Cloudflare rate limiter test page : https://www.cloudflare.com/rate-limit-test/

I am very interested in this feature

[soumsps]

For some unknown reason Cloudflare test page is not working for me :( .. but it was working approx 2 months back.

[ghost]

Rate limiter = a process that refuses to server more than a certain amount / time, like when the free downloading on certain file-hosts limits the download to 50 kpbs no matter how fast your connection is.
(Limiting can f.ex. be done on either bandwidth or parallel requests)

Infraction limit = how many times you can "knock on the wall" (trig some rule) before you are "permanently" blocked, as in "come again next year".

At least that's my view of how it is separated.

I would be very very careful with blanket limiting. Way too easy to catch humans that fast-click or uses parallel tabbing in the browser unless a lot of headroom is left in place (which makes the whole limiting less useful), and there will definitively be a risk with accidentally smashing some search engine spiders unless those get a special bypass.

[Maikuolan]

Let's say that there's a feature, whereby for X many "events" (the "limit"), the "entity" (the inbound IP address, user agent, or whatever else) will be denied any further access to the "system" (the website or server that we're protecting).

In both of these cases ("rate limiting", and the "infraction limit"), the "entity" we're working with is the inbound IP address, but the "event" that we're measuring is different.

In the case of "rate limiting", the "event" is any request or pageview whatsoever to the system. Generally, one request or pageview represents one event.

In the case of the "infraction limit", the "event" is an "infraction", which (in the context of CIDRAM) is a value corresponding to a "block event" (i.e., an "infraction" occurs when something is blocked). This can be either something blocked specifically by a module, or by something blocked by CIDRAM more generally, depending on package configuration. Generally, one block event will generate one infraction, but this can be artificially increased to other values, for signatures which correspond to particularly dangerous behaviours (e.g., recognised hacking attempts, etc).

This means that an entity that gets banned because of exceeding the infraction limit, has already been doing bad things (or at least, has been behaving suspiciously, or doing things which provide reasons for us to distrust the entity), but an entity that gets banned because of exceeding the rate limit, potentially might've not been doing anything bad, but might simply be overzealous with their pageviews, or accessing too much at once.

The theory, is that "accessing too much at once" is a good way to detect scrapers, and is something that scrapers often do. The theory here is not wrong, but we would need to be careful, because working from that theory alone, without more broad regard for context, the types of normal, expected traffic at websites and so on, could result in the wrong entities (i.e., your users) being accidentally blocked.

[Maikuolan]

@macmathan beat me to it (we must've both been writing at the same time). But yeah; What he wrote. (Or what I wrote, for the TL;DR version of the same thing). :P

[glennish]

Thanks for all the feedback and debate.

I agree any such feature should default to OFF with a warning as this might be specific to certain types/styles of website. Search engine (and related) verification would be a must but I presume this is something that is done once per IP and the verification stored locally? Taking this one step further, having the ability of turning on and off verification for different spiders and crawlers might be a good idea too.

I would like the infractions to be be based on page views and 'clicks' per page (two separate settings). Initially, any block logs would need to be monitored quite tightly (and separate from existing log?), but I think a workable setting value could be found quite quickly.

Ability for a test IP would be great too to make sure everything is working without blocking yourself.

[737simpilot]

It sounds like an interesting idea. Perhaps it can be combined with the Google captcha facility so that if a legit user is clicking the hell out of your site, they are allowed to respond to a captcha to not block them.

[ghost]

CAPTCHA is pretty dead as security measure, and annoying as heck to regular users.

Many bots can bypass the older style (using letter, numbers etc.) and if you want to bypass the reCaptcha image system there's always cheap labour from unscrupulous parts of the world ;)

Bottom line, don't rely on CAPTCHA.

[737simpilot]

CIDRAM uses it and an invisible captcha already. I would say that the invisible captcha would be the way to go.

[glennish]

Hi Maikuolan, are there any updates or anything more that needs discussing in regards to this?

Thanks.

[737simpilot]

I'd like to see rate limiting myslef since I do have it on in CloudFlare, but if I go above 10.000 requests I get charged. LOL

[Maikuolan]

Currently I'm thinking this could be something for CIDRAM v1.9.0, when released. Got a few small things that I'd promised to finish off before the next tagged release (CIDRAM v1.8.0), and I'm hoping to tag that release immediately thereafter (I anticipate that happening at some point within the next few weeks, maybe earlier). I could work towards implementing a rate limiting feature after that though.

[Maikuolan]

Ability for a test IP would be great too to make sure everything is working without blocking yourself.

This is already possible. :-)

If you've enabled the front-end, you can test IPs via the IP test page.

Otherwise, for anyone that wants to roll their own code for testing whether CIDRAM blocks any particular IPs, I've build a simple API wrapper for CIDRAM that can be used to leverage CIDRAM's own test functions in other contexts:

https://github.com/CIDRAM/CIDRAM-Extras/tree/master/api

[Maikuolan]

(Worth noting: Considering that CIDRAM utilises a flatfile database system currently, I do anticipate that there could be some problems when using it as a rate limiter, in cases where IPs might send significant numbers of requests simultaneously, due to possible race conditions when multiple PHP processes simultaneously attempt to access the file to be responsible for tracking number of requests from any particular IP address. I can build this feature, but I can't promise that it will always be effective).

[737simpilot]

Hmmm, that's a good point. I guess in effect it would be a DDoS anyway. I guess layer 7?

[DanielRuf]

Not in particular. Also there are solutions for DDoS attacks. In this case many different IP addresses are used.

[glennish]

Not wanting to be _that_ guy, but how far off is CIDRAM v1.9.0 and implementing 'auto block scrapers'?

[dev-101]

https://github.com/CIDRAM/CIDRAM/issues/98#issuecomment-437025248

[Maikuolan]

I've been slowly making a start on this during my spare time, but yeah.. I don't plan on committing anything until I get back home. Too much risk of something going wrong due to not having enough time and rushing it as a result of that, and then not being at a machine to be able to fix said thing going wrong. Should be committed (and released, if all goes well) around early or mid December.

[Maikuolan]

Quick question (opinions invited): Worth including a way to enable/disable this for specific IPs/CIDRs (for those likely to use this feature), or would those likely to use this feature be more inclined to apply rate limiting across the board for all visitors/users?

[soumsps]

I am more inclined towards usage of rate limiting for all visitors/users.

But one question... suppose I am having WordPress site .. and as an administrator ... there is high probability I could exceed rate limit value ... So, in this case what CIDRAM will do ?

@Maikuolan can we have url based rate limiting rules?

[DanielRuf]

Sounds like what all the other security plugins for WordPress already do and which is not based on CIDRs.

[737simpilot]

Perhaps it might be prudent to create IP/CIDRs as an exclusion in case that feature is needed by someone else in the future.

You wouldn't face a rat limit in WordPress as an Admin if you whitelisted your IP in the IPv4_custom.dat file or IPv6_custom.dat file. Or you can enable CIDRAM's maintenance mode while doing Admin things in WordPress.

I don't use the WordPress plugin for CIDRAM so not sure about its options there.

[Maikuolan]

My proposed warning message, to be included in the configuration section of the documentation, immediately before the configuration directives relating to rate limiting (these directives also haven't been added yet; will likely update the whole lot at once):

>

This feature was implemented to CIDRAM because it was requested by enough users to warrant being implemented. However, as it is somewhat outside the scope of the purpose originally intended for CIDRAM, it most likely won't be needed by most users. If you specifically need CIDRAM to handle rate limiting for your website, this feature could be useful for you. However, there are some important things you should consider:

• This feature, like all other CIDRAM features, will only work for pages protected by CIDRAM. Therefore, any website assets not specifically routed through CIDRAM can't be rate limited by CIDRAM.
• Don't forget that CIDRAM writes it cache and other data directly to disk (i.e., saves it data onto files), and doesn't use any external database system like MySQL, PostgreSQL, Access, or similar. This means that in order for it to track usage for rate limiting, it would effectively need to be writing to disk for every single potentially rate limited request. This could contribute to lower disk life expectancy in the longterm, and is not ideally recommended. Instead, ideally, a tool used for rate limiting could utilise a database system intended for frequent, small read/write operations, or could retain information persistently across requests, without the need to write data to disk between requests (e.g., written as an independent server module, instead of a PHP package).
• If you're able to use a server module, cPanel, or some other network tool to enforce rate limiting, it would be better to use that for rate limiting, instead of CIDRAM.
• If a particular user is very keen to continue accessing your website after being rate limited, in most cases, it will be very easy for them to circumvent rate limiting (e.g., if they change their IP address, or if they use a proxy or VPN, and assuming that you've configured CIDRAM to not block proxies and VPNs, or that CIDRAM isn't aware of the proxy or VPN that they're using).
• Rate limiting can be very annoying for actual, real end-users. It may be necessary if your available bandwidth is very limited, and if you discover that there are some specific sources of traffic, not already otherwise blocked, that are consuming the majority of your available bandwidth. If not necessary though, it should probably be avoided.
• You may occasionally risk blocking legitimate users, or even yourself.

If you feel that you don't need CIDRAM to enforce rate limiting for your website, keep the directives below set as their default values. Otherwise, you can change their values to suit your needs.

What do you reckon?

[Maikuolan]

Perhaps it might be prudent to create IP/CIDRs as an exclusion in case that feature is needed by someone else in the future.

I've been thinking a bit on that actually, since first asking the question earlier. Technically should be possible already to do, as long as I make sure that signature files are processed before rate limiting is processed. This is because of the ability to modify configuration values per request instance via signature files if needed (i.e., someone could create a new signature in their custom signature files targeting their own IP or some CIDR containing their IP, and then just include some YAML afterwards to disable all the relevant rate limiting configuration directives for any requests which trigger that particular signature). I'll probably need to provide examples and documentation of course, but it does also mean that I won't necessarily need to write any actual new code in order to be able to disable rate limiting for specific IPs or CIDRs. So, we're probably okay with IP/CIDR exclusion, at least.

Becomes a little more complicated when dealing with specific URLs though. I'll need to think on that a bit more.

[737simpilot]

Looks good thus far. I guess I'll keep the rate limiting on in CloudFlare then sense you give that warning based on CIDRAM's flat files approach. CloudFlare charges me about 30 cents a month anyway. However, I reckon that if the IPs are from Cloud sources and what have you, they'd all get blocked by CIDRAM anyway. With my current configuration it's three strikes and no more logging.

[glennish]

Quick question (opinions invited): Worth including a way to enable/disable this for specific IPs/CIDRs (for those likely to use this feature), or would those likely to use this feature be more inclined to apply rate limiting across the board for all visitors/users?

Probably across the board for me. I'm looking for a way to stop people/bots/scrapers clicking all of my listings.

My proposed warning message, to be included in the configuration section of the documentation, immediately before the configuration directives relating to rate limiting (these directives also haven't been added yet; will likely update the whole lot at once):

This feature was implemented to CIDRAM because it was requested by enough users to warrant being implemented. However, as it is somewhat outside the scope of the purpose originally intended for CIDRAM, it most likely won't be needed by most users. If you specifically need CIDRAM to handle rate limiting for your website, this feature could be useful for you. However, there are some important things you should consider:
...

Reads well enough to me :)

I'm amazed at the time and dedication you put into this, so again a big thank you.

[Maikuolan]

You're welcome. :-)

(Currently progress: About halfway there. Just been a little busy this week since getting back from overseas, and probably a little next week too. Definitely should have this done before the end of this month though).

[Maikuolan]

Rate limiting has been implemented. :-)

I'll leave this issue open for a while, to allow for further testing, and in case any unexpected problems emerge, but AFAICT from my own testing thus far, everything works as expected and it should now be ready to be used.

If you use it, let me know how it goes, whether you have any questions, need any help with it, whether you encounter any problems, etc. Cheers.

[glennish]

Boooom!

Thanks very much for listening.. I'm not in the office until tomorrow afternoon but will have a play and let you know.

Kind Regards.

[glennish]

Hi Maikuolan,

Just checked a WP verion of CIDRAM and not seeing the settings for rater limiting in config.ini? Maybe I have to wait for update update cron to fire again?

Current System information:

CIDRAM version used: 1.9.0
Branch latest stable: 1.8.0 – Latest stable: 1.8.0 – Latest unstable: 1.9.0
PHP version used: 7.2.12
Branch latest stable: 7.2.13 – Latest stable: 7.3.0 – Latest unstable: -
SAPI used: cgi-fcgi

[Maikuolan]

What's your preferred method of modifying the configuration? E.g., manually editing the config.ini file, modifying configuration via the front-end configuration page, etc.

If modifying configuration via the front-end configuration page, the new settings should appear under a newly added "rate_limiting" category. Simply changing as needed and then clicking "Update" should see the new settings added to the config.ini file.

If manually editing the config.ini file, the newly available settings won't appear until you add them in manually, due to that the updater itself doesn't modify the config.ini file during updates, to avoid the risk of breaking a user's customisations, settings and so on between updates.

You can copy/paste the following data to the bottom of your config.ini file and modify as needed if modifying manually:

[rate_limiting]

; The maximum amount of bandwidth allowed within the allowance period before
; rate limiting future requests. A value of 0 disables this type of rate
; limiting. Default = 0KB.
max_bandwidth='0KB'

; The maximum number of requests allowed within the allowance period before
; rate limiting future requests. A value of 0 disables this type of rate
; limiting. Default = 0.
max_requests=0

; The precision to use for tracking IPv4 usage. Value mirrors CIDR block size.
; Set to 32 for best precision. Default = 32.
precision_ipv4=32

; The precision to use for tracking IPv6 usage. Value mirrors CIDR block size.
; Set to 128 for best precision. Default = 128.
precision_ipv6=128

; The number of hours to track usage. Default = 0.
allowance_period=0

:-)

[glennish]

I'm happy to use either to modify the configuration, but neither option is showing for me?

[glennish]

ahh, just re-read - will try adding the options to config.ini now :)

[glennish]

Ignore me - found them in the front end now too..

[Maikuolan]

Cool cool. 👍

[Maikuolan]

Haven't noticed any problems come up thus far, and seems to all be good, so, I'll go ahead and close this as implemented.