Cidram: 503 error on v1 branch

[adrianmihalko]

Please use this template when reporting bugs, problems, issues, etc.

Server information.

Caddy, Apache/2.4.18

CIDRAM version used (if known, please also include the exact commit hash).

git cloned branch v1 and release v2.3.0.zip

Signatures version (if known).

PHP SAPI used (<?php echo php_sapi_name(); ?>).

fpm-fcgi

PHP version installed (<?php echo phpversion(); ?>).

7.2.24

Operating system used (<?php echo php_uname(); ?>).

4.15.0-72-generic #81-Ubuntu

---

Issue

Please describe the problem here.
I am trying to run Cidram v1 branch on two different machine (Caddy and Apache server), but I can't make it work. I always get blank page & error 503.

What I tried:

• I enabled web interface and I am trying to open /loader.php
• created test.php and added

Apache, Caddy error log doesn't show anything.

[adrianmihalko]

chown www-data:www-data cidram/ -R

[Maikuolan]

chown www-data:www-data cidram/ -R

For the benefit of others, and for non-Linux users: chown(1) - Linux man page.

[adrianmihalko]

@Maikuolan I still need to read a lot about CIDRAM (and it's massive readme file), but I want to ask two thing:

• it would be nice to check against an IP if it's blocked or not, is this somehow possible?
• enable some test mode to check how website works from "attacker" viewpoint

[Maikuolan]

it would be nice to check against an IP if it's blocked or not, is this somehow possible?

Yep. :-)

Generally, the easiest way to do this would be to use the "IP test page" provided by the front-end.

Alternatively, for those that don't want to enable or use the front-end, for those that would prefer to create their own interface, or for those that would prefer to interface CIDRAM with something else, the "CIDRAM API loader" could be useful, too.

It's also possible to do this by faking datapoints, too. E.g., creating a PHP file, replacing existing $_SERVER variable's values with other things, and then calling the loader:

<?php
$_SERVER['REMOTE_ADDR'] = '192.168.0.1';
require 'loader.php';

(But, this latter solution, can become inconvenient sometimes, due to always needing to edit files whenever the tested datapoints change, and particularly when needing to test against multiple IP addresses. I would generally recommend using the "IP test page" instead, due to being able to test multiple IP addresses at the same time, due to there being less work involved, etc).

[adrianmihalko]

This is awesome. Thank you. I need to use it on my OpenCart server, because I get many fake spam registered accounts from China and Russia. From a quick search I haven't found a clear answer, from where Cidram gets the "bad guy ip" data?

[adrianmihalko]

And it's catched my bad guy's IP:

Very cool.

Testing more, looks like it doesn't know every IP, but it's still better than no protection at all. For example 197.50.77.154 is not blocked (some African spammer).

[Maikuolan]

This is awesome. Thank you.

You're welcome. :-)

From a quick search I haven't found a clear answer, from where Cidram gets the "bad guy ip" data?

I haven't added anything to the documentation about that yet, but that's something I should probably try to do at some point.

The source of the information behind the decisions regarding the addition and removal of signatures in CIDRAM is varied. At this time, I manually maintain CIDRAM's signatures myself, and the data which goes into maintaining those signatures isn't acquired from any one specific, singular source, but is acquired from a range of different sources, including other people, various platforms, databases (e.g., Stop Forum Spam, Google Malware Dashboard, Spamhaus, AbuseIPDB, and others), often involves cross-referencing records from different databases and so on. I run a few honeypots too, which sometimes helps with discovering new things to block, and I sometimes share information with some other people that I know which run their own honeypots too.

Sometimes, users will mention something here at this issue tracker about some specific IPs/CIDRs/networks/etc, or will suggest something new to block, so I'll try to investigate it, and sometimes that'll lead to new signatures being added to CIDRAM's signature files.

Sometimes I'll leave brief notes in the signature files, explaining why some specific signatures were added, the logic/reasoning behind the decisions to include those signatures and so on (but this is something I've really only just started doing since the past year or so, so often, many of the older signatures from prior to that don't have these notes; I hope to add more notes and to improve on this over time though).

The signature files are also being constantly revised, new things added, old things removed, etc (this is necessary, both in order to be able to block new threats which emerge over time, as well as in order to avoid false positives, to clean out stale data, removing formerly bad IPs that might've become inherited by other, benign entities since the time they were first listed).

For example 197.50.77.154 is not blocked

I'll add that IP to the next signatures update. Thanks for the heads up. :-)

[adrianmihalko]

Thank you for the detailed answer. If there would be a easy way to report bad IPs I would like to contribute.

For example today's result:

India:
https://pastebin.com/raw/vqdidPD1

Russia:
https://pastebin.com/JYkzvvTc

Egypt:
https://pastebin.com/raw/uQeNKxwv

Thailand:
https://pastebin.com/raw/vbDqQCUS

Interesting how user agent is identical for these spammers.

Just one more question which is still not clear to me: I can update database via 1. cron 2. automatically? It's automatically downloaded after some time? Isn't this slows down the web page when updating?

My web page is fully cached at first view of a given page, so I can't use this script on my whole website (if attacker views first, then 403 will be cached I think). Fortunately some parts on my web page are not cached, like the register page and I can use it there. It's not a big problem, if require loader.php is not at the top at the page, right?

Because you have a whole database of bad IPs, you didn't think about to extend this project to something universal, like an iptables / htaccess blocking solution? I don't know any more project like this.

👋

[DanielRuf]

Because you have a whole database of bad IPs, you didn't think about to extend this project to something universal, like an iptables / htaccess blocking solution? I don't know any more project like this.

This would be an extra project and much data.

See https://github.com/DanielRuf/CIDR-rules

There are also scripts for the generation of the rules.

[Maikuolan]

(Writing a short, partial reply for the moment, because I need to walk out the door for work in a few minutes, but I'll write a more thorough reply later today, when I get home again).

Just one more question which is still not clear to me: I can update database via 1. cron 2. automatically? It's automatically downloaded after some time?

This is possible to set up. :-)

The CIDRAM front-end provides an "updates" page, allowing users to install/uninstall/update various components, modules and so on provided for CIDRAM. CIDRAM doesn't natively interact with any cron managers, and doesn't utilise "poorman's cron", but I've provided a separate tool, "Cronable", which can be used with cronjobs to completely automate the process of updating CIDRAM (Cronable works by sending requests to CIDRAM to log into the front-end itself and utilise CIDRAM's "updates" page to update everything); that could be useful in this case.

Isn't this slows down the web page when updating?

Depends. Pseudo-cronjobs, utilising "poorman's cron", etc would probably slow down webpages when updating or when performing similar tasks, seeing as they're usually triggered by normal webserver requests, but a true cronjob, executed by a cron manager, shouldn't generally slow down webpages, seeing as they're generally triggered by the cron manager, rather than by a webserver request, so normal users shouldn't notice anything, as long the server is relatively decent and not running the entire server instance, as well as the cron manager itself, all through the same, single PHP thread.

Anyway, need to run, but I'll reply again later today. :-)

[Maikuolan]

It's not a big problem, if require loader.php is not at the top at the page, right?

It's not a problem if the loader isn't protecting every page, but it's best to have it execute as early in the execution chain as possible, for those pages which it protects. After all, if a page has already been sent to the output buffer by the time that CIDRAM executes, "blocking" the request won't be much use at that point, because the request would've been already effectively served. There's also a risk that PHP might produce "headers already sent" warnings if CIDRAM executes after data has already been sent to the output buffer, due to CIDRAM sending response codes (403, 503, etc; configurable) when requests are blocked. So, it's best that CIDRAM is executed before anything is sent to the output buffer. But, as long as that happens, it shouldn't be a problem if the loader isn't required right at the very top of the page. :-)

Because you have a whole database of bad IPs, you didn't think about to extend this project to something universal, like an iptables / htaccess blocking solution? I don't know any more project like this.

I do provide an htaccess version of CIDRAM's signatures (available here), because there are some users which prefer to be able to use them at htaccess level, but I try to not promote it too much, because the amount of data is quite large (currently, 32,509 rules), and my primary concern is that the way that htaccess rules are processed by servers means that such a large amount of data could risk creating significant bottlenecks when processing requests, risk slowing down websites and so on (the way that CIDRAM processes its own data means that for CIDRAM, such large amounts of data aren't a problem at all).

My secondary concern is that, whereas CIDRAM's various configuration options provide some degree nuance for exactly how the signatures are handled (e.g., being able to enable/disable signatures on the basis of the block reason, e.g., block_cloud, block_bogons, block_generic, etc; the ability to offer a reCAPTCHA instance to users when blocked, in case of false positives, and the ability to modify the conditions by which that would occur; being able to run supplementary code on specific ranges when necessary, such as to bypass "good" bots and so on), that nuance doesn't exist at all for the htaccess version of CIDRAM's signatures (because it's literally just a list of htaccess allow/deny rules).

But, that all said, the actual ranges in question are exactly the same, and it's provided for the convenience of anyone that wants it, so anyone is welcome to use the htaccess version if they want to do so. :-)

[adrianmihalko]

Thanks for the extra detailed information.

Instead of .htaccess I think it would be better to block at iptables level, as you said because of speed. If someone wants to try that, something like:

iptables -A INPUT -j HTACCESS_TEST
grep deny .htaccess | awk {'print $3;'} | while read i; do iptables -A HTACCESS_TEST -i eth0 -s ${i} -j DROP; done

...should work to convert form .htaccess to iptables. Maybe I will try this later.

One question left from your previous reply:

"but a true cronjob, executed by a cron manager, shouldn't generally slow down webpages, seeing as they're generally triggered by the cron manager, rather than by a webserver request"

How? 🤔 (Pseudo-crontab is nice, but I don't want to enable web interface).

[DanielRuf]

Instead of .htaccess I think it would be better to block at iptables level, as you said because of speed. If someone wants to try that, something like:

Or use one of my PHP scripts ;-)

[adrianmihalko]

I am using it. 😀 But I am thinking on usage on my servers, where I don't run PHP, just to add some extra protection.

[DanielRuf]

What does your server run then (CIDRAM is written in PHP)? =)

[adrianmihalko]

I mean bad IP list with iptables would be useful even if you don't run a webserver, to stop malicious/bad traffic.