Cidram: An easier, more user-friendly way to generate signatures.

[Maikuolan]

Regarding an idea mentioned via CIDRAM's Gitter.im channel:

just a thought - have you considered providing an Instaban interface via the front end? So it allows me to define one of a UA, whyblocked, referrer, IP, hostname etc, and then it automatically blocks everything for that entity with a 503, or whatever the setup defines.
simpler than coding a custom mod.

Earmarking for a future v1.8.0 release.

[Dibbyo456]

That would be dope. Also a "whitelist" would be nice addition.

[mikeruss1]

if you wanted to take it further, how about a screen interface to generate Triggers?

so you could select the parameters to form the "IF" -
maybe UALC - proximic - not false
and whyreason - Amazon - not false
and it generates the code.
this probably requires a test harness to prove it works and hasnt broken anything.

[Maikuolan]

Definitely, those ideas would all be pretty cool additions too. :-)

[Maikuolan]

So.. hoping to work on this before the next release (v1.8.0). Still trying to wrap my head around exactly how this should all look on the front-end, whether it should be part of existing functionality, or should implement some new subsystem into CIDRAM, whether it should have its own page, or integrate into existing pages and such, lol. Haven't started coding it yet, but I've been thinking about it a bit, trying to figure it out before I start on it. Thoughts from others always welcome of course.

[mikeruss1]

its completely different from the rest of the front end so suggest its own section

needs to be elegant and easy to use, so probably needs an example line followed by boxes to complete

take the technology / programming out and request simple entries

not easy !!

[Maikuolan]

Starting work on this now.

[Maikuolan]

Committed. :-)

After updating, you should see a new page appear at the front-end, "Auxiliary Rules". This can be used to block, bypass, greylist, and whitelist requests based on whatever criteria is specified when creating new rules there.

I'm marking this as "implemented", but I'll keep the issue open for a while longer yet. Everything on the page has been tested prior to committing it, and it seems to all work properly, but there are a few more things I'd like to still add to it before closing the issue and a few small details I'd still like to improve. It can be safely used though. :-)

If you use it, and have a spare moment, let me know what you think, whether you feel it is easy enough, or whether it needs more work and such. Cheers. :+1:

[Maikuolan]

(I should mention though, this is actually a new subsystem I've built into CIDRAM, so it has nothing at all to do with signature files or modules. Auxiliary rules execute after all signature files and modules have finished executing. I haven't had time to document anything yet, but I'll get that done when I've got some more time to spare).

[mikeruss1]

looking forward to this, will implement when I have time to test it properly, probably the weekend

thank you !!

[mikeruss1]

apart from the fact I cant get it to work, it seems easy and intuitive to use. Easier than I expected and once used to it will convert everything I have in custom mods

might be worth an example page showing how each of the block, bypass etc work. I feel that someone coming into this as a new user would need a bit of a tutorial, and encouragement

[Maikuolan]

Cool cool. :-)

Good point about tutorials and such. Definitely could help.

[Maikuolan]

apart from the fact I cant get it to work

I noticed your messages on Gitter. I'll reply to them there, for better conversation readability.

[mikeruss1]

couple of thoughts ..

consider an rule amendment option, imagine a complex multi line rule you are testing, every small change requires a delete and recreate

"are you sure you want to delete this rule", deleting a complex rule by mistake means you have to remember it

[Maikuolan]

100% agreed. That's actually one of the things I was hoping to implement before closing this issue. ^.^

Main delay currently is trying to figure out the best to implement an edit/modify/amend option into the interface as such that we still retain a good experience for the user (e.g., to avoid an ugly or counter-intuitive implementation). Also hoping to implement a way to test syntax validity, and maybe a way to rearrange rules, in case the order needs to be changed (currently, this can only be done by manually modifying the auxiliary.yaml file).

[mikeruss1]

it might also be neat to be able to define a GET which tests the rule works as expected. I suppose it would be too complex to actually run it through the website, so maybe a front end test only?

and maybe the reverse, ensure its not blocking stuff that shouldnt be blocked ?

[mikeruss1]

plan to incorporate $InstaBan ?

[mikeruss1]

I have the following in custom mods which blocks short UA - is there a way to use the regex option for this ?

$Trigger(preg_replace('[^.{0,15}$]', '', $CIDRAM['BlockInfo']['UA']) == '', 'Invalid UA - 004');

[mikeruss1]

tried a grapeshot bypass with 2 conditions and it works

[Maikuolan]

I have the following in custom mods which blocks short UA - is there a way to use the regex option for this ?

$Trigger(preg_replace('[^.{0,15}$]', '', $CIDRAM['BlockInfo']['UA']) == '', 'Invalid UA - 004');

Yep. Super easy. Just put the [^.{0,15}$] part in as the value to be matched, select "User Agent" from the drop-down list for sources, select "use regular expressions to test the conditions" from the drop-down list for matching methods, select "if the following conditions are met, block the request" from the drop-down list for actions to take when successfully matched, put the Invalid UA - 004 part in as the name and reason for the new rule, create the new rule, and you're done. :-)

[Maikuolan]

plan to incorporate $InstaBan ?

Maybe.. but I'll need to think on this a bit. I can see the benefit of adding something like $InstaBan to the auxiliary rules page, but it's yet another thing that I'll need to explain to beginners, so that they'll understand its purpose, why it's there, what it does, etc. The fact that "banning" in CIDRAM already inherently doesn't work the same way as it does in other systems, namely in that the way that $InstaBan is currently implemented is just a hack way to artificially modify tracking rules on specifically marked module signatures (seeing as it only would apply to modules anyway, because "banning", in the context of the original CIDR blocking functionality by itself, actually makes no sense at all), I see adding $InstaBan as an option directly to the auxiliary rules point as something which could potentially cause a lot of confusion for beginners.

Seeing as the original goal of auxiliary rules was to provide "an easier, more user-friendly way to generate signatures", I want to careful about what gets added there too, as to avoid creating too much complexity there, or too many things which would need explaining to beginners (otherwise it'll eventually just end with "why don't I just use modules instead?" from users anyway).

Probably better to sort out the other things first, I think, and then come back to the subject of $InstaBan a little later.

[Maikuolan]

it might also be neat to be able to define a GET which tests the rule works as expected. I suppose it would be too complex to actually run it through the website, so maybe a front end test only?

and maybe the reverse, ensure its not blocking stuff that shouldnt be blocked ?

I could always add request method as a possible source for conditions.. that could definitely be useful, and would allow users to further customise rules on the basis of whether GET, POST, OPTIONS, etc. I'll see to that soon. :+1:

For other things: You can already test for rules which use IP address or user agent as conditions via the IP test page. Rules which use the request query as a condition can be tested by actually adding the value being tested into the request query via your browser (though unlike when testing IP addresses or user agents via the IP test page, testing queries by actually using them would cause a real block event to occur, rather than a simulated block event, so.. would probably need to be careful about accidentally banning yourself via IP tracking when doing it that way). Some of the other possible condition sources might be a little trickier to test though (short for actually waiting for something in the wild to trigger the rule, of course).

[Maikuolan]

Support for request method added and committed the fix for the bug I mentioned before on Gitter.

[mikeruss1]

I have the following in custom mods which blocks short UA - is there a way to use the regex option for this ?

$Trigger(preg_replace('[^.{0,15}$]', '', $CIDRAM['BlockInfo']['UA']) == '', 'Invalid UA - 004');

works fine - thank you

[mikeruss1]

couple more thoughts ...

if you code a rule line and then add another line, the previous code is lost
could you consider a way to delete or amend a rule line - you may already be working on this as part of modifying a rule

[mikeruss1]

the Grapeshot bypass I coded yesterday seems to be including itself in other blocks

ID: 1084
Date/Time: Sun, 30 Sep 2018 02:22:22 +0100
IP Address: 46.229.168.145
Signatures Count: 2
Signatures Reference: 46.229.160.0/20, botspiders_cidram.inc:L47, module_botua.php:L208, functions.php:L1854 Why Blocked: Cloud service ("DataWeb Global", L11453:F6, [NL]), Suspected bot detected!, Backlink/SEO/Scraper UA, Grapeshot bypass!
User Agent: Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html)
Reconstructed URI: https://www.xxxx.co.uk/Dday/cindy_story.php3 reCAPTCHA State: Disabled.

note the Grapeshot bypass!
there are a few of these in the log

this is the aux bypass
If the following conditions are met, bypass the request.
Use Windows-style wildcards to test the conditions.
User Agent (lower-case) = grapeshotcrawler
Why Blocked = Suspected bot detected

maybe its matching against "Suspected bot detected" ?

for my usage its preferable for a bypass to match against all lines in a rule in order to execute ?

[mikeruss1]

and for blocks as well, I dont have any multiline rules, but I suppose the same should apply, all need to be true for the block to execute ?

[Maikuolan]

if you code a rule line and then add another line, the previous code is lost
could you consider a way to delete or amend a rule line - you may already be working on this as part of modifying a rule

Definitely, I agree there too. It's a UI problem which needs to be fixed.

maybe its matching against "Suspected bot detected" ?

Yeah.. I reckon that's what's happening here.

Noting this, seen on the auxiliary rules page – Any "equals" (=) conditions may trigger the rule, as long as all "not equals" (≠) conditions are also met. – basically, because one of the conditions matched, the rule is triggered (i.e., it's matching A or B, instead of A and B).

for my usage its preferable for a bypass to match against all lines in a rule in order to execute ?

There should be a better way to go about sorting this out. I suspect that matching against all lines in a rule might result in a bypass which is too narrow, and mightn't always trigger when needed.

I need to walk out the door in about 5 minutes to get to work on time, but leave it with me for the moment, and I'll reply again in about an hour or so.

[mikeruss1]

> There should be a better way to go about sorting this out. I suspect that matching against all lines in a rule might result in a bypass which is too narrow, and mightn't always trigger when needed.

I have bypasses with 5 and 6 lines, matching against for instance Amazon, Cloud service, Known bot user agent, and Suspected bot. If these are matched individually any valid block containing these will be bypassed.

there has to be a bypass line for each of the errors found in the Signature count, otherwise the bypass doesnt work.

[Maikuolan]

I might need to add some more options for different logic to use when matching, I think. I'll work on this tonight and comment here again when it's committed.

[Maikuolan]

Sorted. Can now specify whether any or all conditions should be matched, and also fixed the way new conditions are added so that already written values aren't lost anymore. :-)

[mikeruss1]

that was quick - thank you
have recoded will let you know as soon as the bypass has been used

[mikeruss1]

that worked - great thanks

a minor GUI suggestion, move "Create new rule" to the bottom right after the rule lines.

it hurts my brain every time to see it above the rules

[Maikuolan]

Sounds like a good idea. :-)

I'll do that with the next commit.

[mikeruss1]

would you mind checking this Auxiliary bypass for me - looks OK to me !

If the following conditions are met, bypass the request.
Use Windows-style wildcards to test the conditions.
User Agent (lower-case) = proximic
Why Blocked = aws_cidram.inc-IPv4
Why Blocked = Amazon.com, Inc
Why Blocked = Scraper UA
Why Blocked = Suspected bot detected

In order to trigger the rule, all conditions must be met.

its not working, the Sig count has been reduced by 1, and its found the bypass, Aux Proximic bypass2! is its name

ID: 1252 Date/Time: Tue, 02 Oct 2018 15:18:00 +0100
IP Address: 54.165.41.130
IP Address (Resolved): -
Hostname: ec2-54-165-41-130.compute-1.amazonaws.com
Query: -
Referrer: -
Signatures Count: 3
Signatures Reference: 54.160.0.0/11, 54.160.0.0/12, botspiders_cidram.inc:L70, module_botua.php:L208, functions.php:L1854
Why Blocked: Cloud service ("aws_cidram.inc-IPv4", L243:F3), Cloud service ("Amazon.com, Inc", L10710:F6, [US]), Suspected bot detected!, Backlink/SEO/Scraper UA, Aux Proximic bypass2!
User Agent: Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
Reconstructed URI: https://www.xxx.co.uk/camo.php3
reCAPTCHA State: Disabled. 

[mikeruss1]

one odd thing, I had to change the bypass from the original one in custom_mods.

the second signature which reports Amazon.com, works OK with
$Bypass($CIDRAM['BlockInfo']['SignatureCount'] && strpos($CIDRAM['BlockInfo']['WhyReason'], 'ipv4.dat-IPv4') !== false, 'Proximic + cloud bypass');

I deleted the Aux bypass and reinstated the custom bypass, including the line above, and the bypass is working.
Very strange !

[mikeruss1]

noticed the wildcards were missing from the rules, put them in and it still fails in the same way

[Maikuolan]

the Sig count has been reduced by 1, and its found the bypass

I think it's probably working then (as a bypass, at least). Generally, it's assumed that one bypass = bypassing just one signature or rule (so, good to use bypass, when you're wanting to bypass a specific rule).

Because in this case, prior to executing any auxiliary rules, 4 signatures had already been triggered via signature files and modules, try using "greylist" instead of "bypass", add an extra condition of Signature Count = 4 (as a precaution, in case there are additional triggered signatures or matched rules in the future which we don't want bypassed). Like with whitelisting, greylisting will reset all detections and reduce the signature count to 0, but unlike whitelisting, greylisting doesn't prevent subsequent signatures from triggering or subsequent rules from matching.

[mikeruss1]

greylisting solved it, thanks

odd thing, I have a bypass with 2 signatures and that works.

did you see the point about the signatures changing with an aux bypass ?

[Maikuolan]

did you see the point about the signatures changing with an aux bypass ?

Not 100% sure. Could you quote it in a reply, or point it out to me again? Just to make sure I'm thinking of the right thing here. (Sorry, haven't had great sleep the past few days, and not 100% focused at the moment).

[mikeruss1]

one odd thing, I had to change the bypass from the original one in custom_mods.

the second signature which reports Amazon.com in the aux bypass,, works OK with
$Bypass($CIDRAM['BlockInfo']['SignatureCount'] && strpos($CIDRAM['BlockInfo']['WhyReason'], 'ipv4.dat-IPv4') !== false, 'Proximic + cloud bypass'); (in the custom_mods bypass)

I deleted the Aux bypass and reinstated the custom bypass, including the line above, and the bypass is working.

[Maikuolan]

Ah, that point. Cheers. :+1:

Not so sure why that one wouldn't work as an auxiliary rule. Bypasses written as modules, and bypasses written as auxiliary rules, both leverage the same $Bypass() closure in CIDRAM's codebase, in the same way (other than that auxiliary rules provides an interface and an indirect rather than a direct way of calling it, of course), so it's probably not that.. I think.

Though in this case, the module version checks the "truthiness" of the signature count, rather than trying to compare its literal value against an exact string or an expected pattern, and auxiliary rules don't currently provide a way to check solely the truthiness of a value as the basis for a condition (this is something I could possibly work on at some point though). In the case of signature count though, an equivalent could be found by way of just checking whether it doesn't equal 0 (seeing as it should always be an integer anyway).

But yeah.. Not sure what's going on there currently.

[Maikuolan]

I'm thinking I might just go ahead and tag v1.8.0 today, to get it out of the way, and then aim to get all the other things planned in regards to auxiliary rules (for example, the ability to edit existing auxiliary rules on the page) finished for maybe a subsequent v1.8.1 or v1.9.0. Feeling a bit foggy at the moment and not sure whether I'll necessarily get this finished within the immediate next day or so, and I'm a little concerned about dragging things out for too long between releases, due to the increased risk of false positives and such that can occur as a result of people not updating, which can be the case for anyone relying on actual releases that doesn't use the front-end updates page and such. Plus, having a clean slate in the changelog for Hacktoberfest might be nice, too.

[mikeruss1]

Ok, whatever works best

[Maikuolan]

Closing in deference to #125.