Cidram: The future of CIDRAM CLI-mode

[Maikuolan]

Just wondering.. Do any of you ever use CIDRAM's CLI mode, or know anyone that ever uses it? I thought I should ask, because I almost never use it myself, and most of what it can do, can also be done via the CIDRAM front-end nowadays anyhow.

Currently, CIDRAM's CLI-mode provides the ability to check whether (a single) IP address or CIDR is blocked by the currently installed/active signature files (this can also be checked via the front-end, which can also allow several thousand IPs or CIDRs to be checked simultaneously, against the installed/active modules, auxiliary rules, and with a custom specified user agent), provides the ability to generate a set of CIDRs from an IP address (the front-end can also do this), and provides a facility to verify, validate, and (optionally) fix broken signature files (which is the one major thing that CIDRAM's CLI-mode provides which isn't currently possible via the front-end, but also doesn't function quite as well as it possibly could anyhow).

While there's nothing wrong with CIDRAM's CLI-mode, and keeping it wouldn't be a problem, I also figure that there's not much point in effectively keeping alive an entire separate interface in the project if nobody really actually uses it, and if nobody really uses it, then cutting it might be an easy, low risk way to reduce excess fat from the project, increasing maintainability, reducing future technical debt, etc.

Seeing as cutting it from the project would be a backwards-incompatible change (in the same sense as would be the case when removing features, functionality, changing the fundamentals of an API, etc), if this happened, it wouldn't be happening until the next major version (v2), which I haven't set a definitive schedule for anyway (and so, wouldn't affect any v1 releases).

Regardless of whether it gets cut, the facility to verify, validate, and (optionally) fix broken signature files is something that could (and probably should) be added to the front-end anyhow, so.. cutting CLI-mode wouldn't necessary mean that signature files couldn't be verified/validated/fixed anymore.

Worth mentioning also, that CLI-mode, whether kept or cut, either way, should have zero bearing on CIDRAM's normal, core functionality regardless.

If anybody actually uses it though, I'm quite happy to keep it, as there's no particular reason to remove it anyway, other than the aforementioned points about unused/unnecessary features. For that matter, if anyone thinks that it should be kept regardless, or thinks that they might use it in the future, I'd be happy to take that as valid reasons to keep it.

So, I invite opinion from others:

:+1: :-1: Should CIDRAM's CLI-mode be deprecated, and then removed when v2 is eventually released?
:+1: :-1: Should the ability to verify, validate, and fix signature files (which currently exists in CLI-mode) be implemented to the front-end?

(If possible, when sharing your opinion, also let me know whether or not you use CIDRAM's CLI-mode and the CIDRAM front-end respectively, for better context to your opinion).

I consider this issue to be low priority, and seeing as there's currently no particular, definitive deadline for v2, I'm not going to put any deadline on this issue for now either.

[mikeruss1]

dont use it
there is value in the ability to fix signature files
Mike

[ghost]

Although I'm a lover of CLI's and API's, I haven't had any reason to use the CIDRAM CLI mode so far.

As CIDRAM is primary a web application I'd say it's natural workflow with the GUI approach, so IMHO it would be better to either port over the last bits from the CLI to the GUI and drop the CLI, or as an alternative (for signature fixes) create a separate lightweight CLI option that is only for those bits not in the GUI (like the signature fixes) and drop the general CLI as it stands today. Makes sense?

[DanielRuf]

I would not drop it but move it out as separate project and keep it as is (as small phar / single file class).

I am mainly a CLI user and manage clients through SSH.

[Maikuolan]

Thanks for the feedback; Much appreciated (and makes perfect sense). :-)

Sounds like moving it out from the main package, and to its own, separate project/package is probably the best solution then, I think. That would mean too, that it would continue to be available for those that might need/want it, but also resolve potential maintainability problems by not having it deeply embedded into the main package.

So, as a to-do:

• Porting over the remaining CLI functionality, which isn't yet available at the front-end (signature file validator/fixer), to the front-end.
• Creating a separate project/package for the CLI functionality (preferably as a small phar or single class file).
• Removing the currently deeply embedded CLI stuff from the main package (after the two above-mentioned things are done, and not prior to v2).

[Dibbyo456]

because I almost never use it myself

Me neither. I did know it existed but never bothered to try it. 😐

[Maikuolan]

All good, and no problem. :-)

TBH, at this point, kind of happy that I've got an excuse to remove it from the main package. One less thing to maintain for the eventual v2 release in the future.

[737simpilot]

I feel like an idiot here for not monitoring the issues section all the time as I should be seen as how I use CIDRAM. So for that I apologize.

I always use the front-end. Never have used the CLI, and quite frankly, I don't like CLI stuff or even Linux terminal. LOL I really should learn Linux more, but for me it's a hard nut to crack. Even installing CentOS into VMware and trying to learn how to maintain my own VPS. I have a ton of bookmarks just to know what exact commands I need just to do basic things like installing PHP and Apache, etc. It amazes me I was even able to install webmin. Just a shared account holder here at the moment.

So with that, yeah, just a front-end user. Never messed with the CLI.

Question: Would in fact removing the CLI reduce any possible hack vector? I thought that and so I disabled the CLI in CIDRAM from being used. At least I think I disabled the CLI. Been a while since I looked at that section. LOL

[Maikuolan]

Question: Would in fact removing the CLI reduce any possible hack vector?

Not AFAICT. Shouldn't technically be possible to access any of the CLI-mode code via a webserver anyhow, due to that it's intended only to be used via CLI/terminal/etc, so, to use it as a hack vector, you'd need to either (1) find a way to trick CIDRAM into thinking that a request, originating from a webserver, was actually originating from CLI/terminal/etc (most likely candidate would be via a cron action or vulnerable daemon, but still.. not something I consider likely to happen anyway), or (2) have an already vulnerable CLI/terminal/SSH/shell/etc set up somewhere (which, at that point, would pose a far bigger threat than anything that even a worst possible quality CLI-mode code scenario could ever pose).

But, albeit not a security risk AFAICT, if you don't intend to ever use CIDRAM's CLI-mode, I would still recommend disabling it, just on the off chance that the particular PHP installation that CIDRAM is running on is somehow unable to distinguish properly between a webserver request and a CLI/terminal request, or on the off chance that there might be daemons, unit testing, or something else inadvertently requiring in CIDRAM at some point, potentially causing some unforeseen conflict between CIDRAM and themselves.

The main benefit of removing CLI-mode from the main package entirely, is increased maintainability for the package, and decreased complexity for the package, which should make it a little easier to move things forward in the future, for if and when we develop some other plans in the future, for alternative directions to the take the package and such (less considerations to keep in mind, when there are less things in the package). Basically, it makes sense to not keep things in the main package that almost nobody actually really uses ever.

[737simpilot]

Okay, thanks. That makes sense then. I just disabled it out of an abundance of caution just in case and since I don't use it disabling it was a no brainier.

Trimming the fat so to speak is always a good thing, and as they say, less is more. IMO anyway. Though, it would be prudent to keep it as a separate option for those who will use it. Also, I'm of the opinion that if you have all kinds of code and features, etc that you're not really using, then that should be gotten rid of to lessen a possible hack vector. Like Wordpress plug-ins for example. I actually have several plug-ins now that I use, but my firewall keeps sending me emails saying that these particular plug-ins have not been updated in x amount of years and should be gotten rid of. Now I'd like to do that, but they still function and do what I need, but then I have to look around for current alternatives. It's on the bucket list. I have other things to do at the moment. :D

[Maikuolan]

Task 1/3 completed: The front-end now has a working signature file fixer (982c35763c04a0b0a65b6f9327620328e32cd085). :-)

[737simpilot]

What exactly does a signature file fixer do other than the obvious? I mean, how does it fix a signature? Does it use CRC or something? LOL I have no idea how this works.

[Maikuolan]

What exactly does a signature file fixer do other than the obvious? I mean, how does it fix a signature? Does it use CRC or something? LOL I have no idea how this works.

The front-end signature file fixer does generate hashes before and after the process, but not during; No CRCs nor hashes of any kind are used for, or during the process. :-)

The hashes produced before and after the process, are displayed just prior to the generated output, and actually serve just as a visual aid for the end-user, to quickly determine the size and severity of the changes produced. The "number of changes" is also stated on the page, after the process, but simply stating the number of changes, doesn't necessarily indicate how big, or how significant those changes were (which is why I thought to include the hashes; it isn't perfect, but it helps in that regard somewhat).

As for using the front-end signature file fixer: Probably not super useful, unless you're writing or redistributing your own, custom signature files, but basically it just looks for ways to improve signature files, looks for invalid data, etc and attempts to generate output based on what it finds where possible.

In more detail:

• Standardise the line endings used by the input or signature file.
• Then, for each line in the input or signature file, if the line (or connected group of lines) looks like it(/they) might be a comment(/s), or might be purely informational (e.g., line[/s] starts with a hash, or adopts known, commonly used comment notation; like # This., or /** This. */, for example), or seems to be a known, valid tag permitted by the CIDRAM signature file format (e.g., Expires: xxxx.xx.xx, Tag: %SectionName%, Origin: XX, etc), leave the line(/s) alone (i.e., should remain as is, unchanged, in the output).
• If the line(/s) appear to be YAML (as per permitted by the CIDRAM signature file format), use the YAML class included with CIDRAM to process, then reconstruct the YAML data, to normalise it, and to remove anything that could be invalid or non-compliant per the class.
• On the assumption that all remaining lines are intended to be IP addresses, CIDRs, or otherwise valid signatures, all remaining lines are then ran through the aggregator, separated into multiple blocks of data to be aggregated in multiple, separate aggregation operations, as many times as necessary; each block separated wherever otherwise ignored lines exist, wherever there appears to be several sequential linebreaks, and wherever there might be different parameters used for adjacent signatures (to ensure that we don't accidentally change the block reasons cited for various different signatures during the aggregation process). This thus allows the dataset to be expressed in the smallest possible way, without any loss of information or instructions, and also strips out anything that isn't recognised as a valid IP address, valid CIDR, valid signature, valid/recognised tag, permitted block of YAML data, recognised comment, or comment block.
• After aggregation has finished, for any CIDRs that aren't accompanied by a parameter, a default Deny Generic is appended, to ensure that we have a valid signature (i.e., in that a valid signature, needs not only the CIDR, but also a parameter, to instruct CIDRAM what it actually needs to do when requests from that CIDR are encountered).

There are some differences between the CLI-mode validator/fixer and the front-end signature file fixer. The CLI-mode version also checked things like line length, and would provide a warning if lines were considered "too long". The front-end signature file fixer just gets on with the job of attempting to fix the input or signature file being checked, without producing any warnings, so it doesn't bother to check things like line length. The CLI-mode fixer would automatically save the changed data to an actual file, whereas the front-end signature file fixer simply provides the changed data as output to the page, without automatically saving it (meaning that end-users would still be responsible for deciding whether to accept or reject the changed data, and whether to apply it to their signature files). The CLI-mode signature file fixer was incapable of distinguishing between individual signature sections within a signature file, effectively treating a whole signature file as a single, big signature section (which IMO, is wrong behaviour). The front-end signature file fixer, on the other hand, has no trouble distinguishing between separate signature sections. The CLI-mode version also makes the distinction between a "fixer" and a "validator", whereas the front-end version simply refers to itself as a "fixer", but is more something halfway between the two, in the sense that it produces changes automatically like the CLI-mode fixer, doesn't produce the same warnings or detailed instructions that the CLI-mode validator produces, but unlike the CLI-mode fixer, doesn't automatically implement its produced changes to files. While perhaps adding an extra step for end-users, it also provides the opportunity to reject changes, in the event that it accidentally gets it "wrong" at some point (e.g., end-users could use a diff tool to compare the original data with the changed data, and based on that diff, decide whether to accept or reject changes; automatically implemented changes, on the other hand, take away this decision from the end-user).

In short: Although I've now effectively ported all CLI-mode functionality to the front-end.. currently, the front-end version of the signature file fixer is actually much better, and much higher quality than the CLI-mode version IMO. :-)

[Maikuolan]

v2.0.0 released. :-)

I haven't started on the separate CLI package yet, but I'll get onto that soon. Otherwise, everything else covered by this issue has been sorted out now.

[Dibbyo456]

I'm not seeing any updates in front-end.

[Maikuolan]

Won't see v2 appear at the front-end updates page just yet, because v1 and v2 are on separate branches of the repository, and the front-end updates page currently targets whichever branch the current major version belongs to. This is because there are a few backwards-incompatible changes between v1 and v2, and I need to figure out how to safely automate the process of updating from v1 to v2, without causing any problems or breakage (v2.0.0 is now stable, and works fine, but the loader.php has been slightly modified, and the updater isn't currently able to modify files outside the vault; some configuration directives have also been renamed, but the updater doesn't currently touch things like configuration settings when updating; those are some of the things I need to figure out how to automate without breaking anything). I've got some ideas for this that I'd like to try out, when there's time. Probably either this week or early next week.

[Maikuolan]

Your opinions invited: Better to keep it super simple, as per the original (now removed since v2) CIDRAM CLI-mode? Or would something a little more evolved and polished be preferred (like how phpMussel's CLI-mode works, for example)?

[Dibbyo456]

I never used CLI and I never intend to so my vote - NOPE.

[Maikuolan]

Temporarily removing "Work-In-Progress" tag and adding "Planning" tag, pending further discussion, planning, etc, and pending completion of some other, higher priorities at the moment.

[Maikuolan]

First commit for a new, CIDRAM >= v2 CLI mode (https://github.com/CIDRAM/CIDRAM-Extras/commit/8a680b4a1b450f1470e30c9e1fdf99155214e183):

CIDRAM CLI mode.

This is useful if you want to use CLI or terminal to check whether an IP address is blocked by CIDRAM. CIDRAM v1 and earlier had an integrated CLI mode feature available, but this was removed since CIDRAM v2 and onward. You can reintroduce CLI mode into CIDRAM by using this separate, downloadable cli.php file.

All you need to do is copy the file to the base directory of your CIDRAM installation (the same directory containing the loader.php file) and then run it directly from your CLI or terminal.

---

Last modified: 5 December 2019 (2019.12.05).

The file (for those that want it) can be obtained from here: https://github.com/CIDRAM/CIDRAM-Extras/tree/master/cli

Note: This won't be reintegrated as a core feature of CIDRAM! It will remain as a separate, downloadable file, which you can manually add to your installation, if that's something you want to do.

Seeing as I don't consider this to be a particularly high priority, not too much work has been done on it yet. Thus far, it's able to check whether IP addresses are blocked by CIDRAM, and it's able to calculate the CIDRs of IP addresses. I haven't yet reintroduced signature fixing to it, and haven't yet added anything else to it.

What's there thus far seems to 100% work properly AFAICT, but I haven't yet added any security safeguards to it or any checks to ensure that it's accessed via the correct endpoint (i.e., ensuring that it's accessed via CLI or terminal, and not via webserver requests), so for the moment, I would recommend using it only for development environments or local installations, and not for production environments or live websites.

[Maikuolan]

The signature file fixer has been reintroduced; The IP aggregator has been integrated; Necessary safeguards, as well as some other features, have also now been added (https://github.com/CIDRAM/CIDRAM-Extras/commit/67985a5bd725a7872d20d5b798f18e0788c68a2c).

With that, everything has now been done. :-)

Marking as implemented and closing.