Cidram: Reworking output generator, whitelist logic, etc.

[Maikuolan]

Working on some improvements and fixes currently. Creating this issue early to be able to track changes and discuss if necessary. Bit short on time right now (need to get some things done today and won't be at a computer for a few hours), but I'll post more information about this a little later today, when I've got some time.

[Maikuolan]

I use httpstatus.io, to see headers and some other useful things all the times. I've whitelisted their hosting ip so cidram won't block it. It worked fine until I tried their fetch as Googlebot option.

So, basically they set the User-Agent as Googlebot or other search engines and try to retrieve the page. But now cidram blocking it.

what i don't understand why cidram blocking the request when I already whitelisted the IP?

I thought Whitelist takes highest priority or it's not the case?

Whitelist should take higher priority, but currently, in the codebase (I checked over everything again after you mentioned this problem), search engine verification and social media verification are basically ignoring any priority set or implied by earlier parts of the process. Fixing this now (I'll mark it as a bug-fix in the changelog when pushed).

[Dibbyo456]

What is the difference between whitelist and bypass?

[Maikuolan]

What is the difference between whitelist and bypass?

If a hypothetical request triggers 3 different hypothetical signatures, the "signature count" for that hypothetical request should be 3.

Whereas a signature or rule that utilises whitelist or greylist reduces the total "signature count" to 0 (regardless of whatever previous rules or signature may or may not have been triggered), a bypass reduces the total "signature count" by exactly 1 (meaning that a "bypass" effectively acts like a "negative" signature).

Assuming that W, X, Y are standard CIDR "Deny" signatures, and that Z is a "whitelist" or "greylist" rule or signature:

Begin checking signatures.. signature count = 0.
Signature W matches.. signature count = 1.
Signature X matches.. signature count = 2.
Signature Y matches.. signature count = 3.
Signature Z matches.. signature count = 0.

(Request shouldn't be blocked anymore).

Assuming that W, X, Y are standard CIDR "Deny" signatures, and that Z is a "bypass" rule or signature:

Begin checking signatures.. signature count = 0.
Signature W matches.. signature count = 1.
Signature X matches.. signature count = 2.
Signature Y matches.. signature count = 3.
Signature Z matches.. signature count = 2.

(Request should still be blocked, because multiple, non-bypassed signatures were still triggered).

Bypasses can be useful when there are different rules or signatures that block things for different reasons, and you may sometimes agree with one particular rule or signature, but not the other, and the things that those rules or signatures block sometimes intersect (i.e., one agrees with reason A, but disagrees with reason B, but sometimes A + B can both block the same thing; a bypass, in such cases, when all the rules and signatures start to become a little complex, provides a means to fine-tune it all a bit).

[Dibbyo456]

Wow...! Understood. Thank you for giving the examples, wouldn't have understand without it.

[Dibbyo456]

There is a bug at IP Test. I had set a aux rules with 54.155.177.72 = Whitelist

Now, if we test the IP with the user-agent "googlebot" it shows NOT BLOCKED, but in the real world usage the IP is BLOCKED because of _Fake Googlebot_ (as of 1.13.1-DEV+19192727) which is indeed a BUG but shouldn't IP Tester show same output as in real usage?

So IP Tester not returning the correct result. Or, maybe IP Tester is returning correct result but the search engine verification and social media verification causing the problems? Does IP Tester also test with search engines and social media rules?

---

There are same row (buttons) in aux page, it's better if you omit the first one.

[Maikuolan]

Does IP Tester also test with search engines and social media rules?

It does.. but looking at the GUI, it hasn't really been made entirely clear that it does, so.. that's something to be worked on, in any case. I probably need to add another button to the page, to allow users to choose whether or not to check against that.

[Maikuolan]

Now, if we test the IP with the user-agent "googlebot" it shows NOT BLOCKED, but in the real world usage the IP is BLOCKED because of _Fake Googlebot_ (as of 1.13.1-DEV+19192727) which is indeed a BUG but shouldn't IP Tester show same output as in real usage?

Definitely, it should. If it isn't, then yeah.. there's a bug in there somewhere. Thanks for pointing it out. Looking into that now. :-)

[Maikuolan]

New option has been added. :-)

Also, not sure whether I've missed anything, but results from the IP test page seems to match the reality now at my end for when trying to whitelist IPs to prevent the search engine verification being triggered. Mind updating to the latest available version and testing again, to see how it looks at your end, and reporting back what you find? Cheers.

[Dibbyo456]

Ah. The latest update seems to fix the issue that I was having with the IP Tester. Thank you.
But you forgot to remove the same rows in AUX page that I mentioned earlier.

[Dibbyo456]

A question. How do I get the current logfile path cidram is using as variable?
I'm writing a php script for cidram, i need to get the current/latest logfile path cidram is using/logging to.

[Maikuolan]

Ah. The latest update seems to fix the issue that I was having with the IP Tester. Thank you.

Awesome. :-)

But you forgot to remove the same rows in AUX page that I mentioned earlier.

I'll take a look at that tomorrow (ready to call it a night, I think).

A question. How do I get the current logfile path cidram is using as variable?
I'm writing a php script for cidram, i need to get the current/latest logfile path cidram is using/logging to.

Internally, there are a few different variables at play. $CIDRAM['Vault'] is used to determine exactly where the vault directory is located (for logs, as well as just about everything else that deals in any way with the vault directory). The output generator knows which filename to use for logfiles by pulling the information from the configuration array and then parsing it through CIDRAM's "TimeFormat" closure (to convert the {placeholders} that signify the day, month, and whatever else into their corresponding values):

        /** Determining date/time information for logfile names. */
        if (
            strpos($CIDRAM['Config']['general']['logfile'], '{') !== false ||
            strpos($CIDRAM['Config']['general']['logfile_apache'], '{') !== false ||
            strpos($CIDRAM['Config']['general']['logfile_serialized'], '{') !== false
        ) {
            $CIDRAM['LogFileNames'] = [];
            list(
                $CIDRAM['LogFileNames']['logfile'],
                $CIDRAM['LogFileNames']['logfile_apache'],
                $CIDRAM['LogFileNames']['logfile_serialized']
            ) = $CIDRAM['TimeFormat']($CIDRAM['Now'], [
                $CIDRAM['Config']['general']['logfile'],
                $CIDRAM['Config']['general']['logfile_apache'],
                $CIDRAM['Config']['general']['logfile_serialized']
            ]);
        } else {
            $CIDRAM['LogFileNames'] = [
                'logfile' => $CIDRAM['Config']['general']['logfile'],
                'logfile_apache' => $CIDRAM['Config']['general']['logfile_apache'],
                'logfile_serialized' => $CIDRAM['Config']['general']['logfile_serialized']
            ];
        }

When logging actually occurs, the complete path is built something like:

do_something($CIDRAM['Vault'] . $CIDRAM['LogFileNames']['logfile']);

So far, pretty straightforward. But, depending on whether we're talking something like a CIDRAM module, or talking a completely external script, the problems to contend with are slightly different.

If we're talking an external script, the main problems are that (1) that particular part of the codebase only gets executed when it's needed (i.e., when something gets blocked), meaning that those particular variables most likely won't be available for normal, non-blocked requests, and (2) the loader destructs/unsets CIDRAM from PHP's memory after execution completes, in order to free up that memory for whatever comes after.

If we're talking a CIDRAM module, the main problem is that that particular part of the codebase executes after CIDRAM has already finished executing all modules.

For a CIDRAM module, we could resolve that problem pretty easily by just duplicating the necessary parts of that code into the module and then modifying it per whatever the needs of that module are (i.e., pull from the configuration array, append the vault variable, maybe parse it through the TimeFormat closure if necessary, etc).

For an external script, the easiest way to resolve problem No. 2 would be to use the CIDRAM API (basically, a modified loader that doesn't destruct/unset anything, doesn't call the output generator, and provides a basic OOP mechanism to leverage the code that CIDRAM uses internally for the front-end IP test page, which.. hasn't been updated in quite a while, which is why is doesn't currently mention any reference to auxiliary rules at all.. I'll get that sorted out at some point, when there's time). If the external script calls the CIDRAM API instead the standard loader, all of CIDRAM's internal code (functions, closures, variables, etc) will remain available to the calling script. The easiest way to resolve problem No. 1 would most likely be similar to the solution for doing it with a module: Just duplicating the necessary parts of the code and modifying it per the needs of the external script.

Something like (as a super simplistic example of calling the CIDRAM API and duplicating/modifying):

<?php
require 'api.php';

$Latest_Logfile_Path = $CIDRAM['Vault'] . $CIDRAM['TimeFormat']($CIDRAM['Now'], $CIDRAM['Config']['general']['logfile']);

[Maikuolan]

I should mention, too: The last update to the API was prior to the first v2 release, so I don't yet have separate APIs for v1 and v2 (and haven't extensively tested the API against the latest CIDRAM versions yet). I'll most likely release distinct v1 and v2 variants of the API, the next time I get around to updating it (due to the different PHP version requirements between v1 and v2 and to account for whatever other differences that might be pertinent to the API between v1 and v2).

(Meaning that.. and the relevance to your question here.. although it should work properly, I can't 100% guarantee that it'll work properly, so as usual.. if you use it, and encounter any problems, let me know).

[Maikuolan]

But you forgot to remove the same rows in AUX page that I mentioned earlier.

I'll take a look at that tomorrow (ready to call it a night, I think).

Done. (Added some alt keys in there too, to make it easier to quickly add multiple new conditions in succession, without needing to chase after the button with the mouse when it gets pushed down). :P

Anyway, issue has been resolved now, so I'll mark it accordingly and close.