Cidram: reCaptcha not kicking in some cases.

[Dibbyo456]

I set recaptcha->usermode = 1 so, CIDRAM supposed to offer captcha to every signature. But some cases captcha not kicking in. I'm not sure if it's a bug or I am doing anything wrong. Anyway I used Hostspot Sheild VPN to produce the behavior.

reCaptcha Settings

reCaptcha not kicking in

reCaptcha kicked in

Cloud service is the reason signature of why the ip blocked in both cases, but why recaptcha shows in one & not the other?

[Maikuolan]

but why recaptcha shows in one & not the other?

It's buried quite deep in the documentation, but an explanation can be found in the final paragraph of __7.2.1__

Note: A reCAPTCHA instance will ONLY be offered to the user if reCAPTCHA is enabled (either with "usemode" as 1, or "usemode" as 2 with "enabled" as true), and if exactly ONE signature has been triggered (no more, no less; if multiple signatures are triggered, a reCAPTCHA instance will NOT be offered).

So, in this case, the reason that reCAPTCHA kicks in for the second example, but not the first, is that in the second example, 1 signature is triggered, but in the first, 2 signatures are triggered.

The main reason for this is to provide users with a way to effectively disable reCAPTCHA for specific signatures, if they want to enable reCAPTCHA for all signatures, except for some very specific signatures (i.e., if there's some particularly bad entity that they want to thoroughly block, they could double-up on the signatures targeting that entity, effectively disabling reCAPTCHA for that entity).

Of course, in this case.. The first example shown in nothing particularly insidious compared to the second; Rather, the two signatures being triggered ("EGIHosting" and "Alentus Corp") is simply due to that both of these providers are blocked by the main signature files, and they happen to have some interlacing CIDRs, which means that when they were most recently updated, we ended up with signatures blocking the same ranges belonging to each of the two providers. I certainly wouldn't call it a false positive, nor particularly problematic that reCAPTCHA wasn't shown for that request, but that said, in the spirit fairness, if we're showing reCAPTCHA for one of these, it would make sense to also show it for the other, seeing as both requests are of much the same nature.

Not really a bug (correct behaviour as per the documentation and design), but it is nonetheless something that could probably be improved upon.

I'll keep this issue open for now and have a think on it; I'll try to plan for some way to improve on this a bit.

[Dibbyo456]

I see. I read the reCaptcha section could not find any information there, so I thought it was a bug or something. Thank you for explaining it. 👍

Suggestion:
1) One more reCaptcha usemode -> *4* (Show captcha even it was found on multiple database) would be nice, because sometimes real users can be blocked too.
2) The recaptcha was showing align-left, kind of looking bad, so I made it center, looks nice now. :) Consider having it on the center by default.
3) Consider adding Invisible reCaptcha? (Google's new powered by AI captcha solution) It's makes real users less irritate.

[Maikuolan]

Definitely; These would all be good ideas, I think. :+1:

About to get some sleep for the night, but I'll see to making a start on some of these tomorrow.

[Maikuolan]

Alignment issue solved. Other issues in progress.

[Dibbyo456]

@Maikuolan please check message on ShoutMeLoud forum, there is a serious problem, needs your attention.

[dev-101]

I don't actively use CIDRAM, because I have yet to modify/integrate it within a custom setup, which has a specific workflow and provides it's own captcha... effectively, I need CIDRAM just for detection and to trigger outside call, and have to find time to study it's recent considerable changes and all.

However, according to my stats (note: excluding requests which fall into spam (e.g. via keywords or UAs), actively listed on SFS last 3 months, sql injections and other malicious behaviors, which I instantly block):

1. Around 3.3% (weekly average) of total blocked requests are actually regular visitors "hiding" or using common cloud/vpn/bad reputation signatures we commonly block (cidram or zbblock). If I don't count multiple consequtive requests from (stupid) bots/scripts, then the % is actually even higher! (maybe 10-15%).

2. They are, in fact, not spammers/bots/scrapers at all, just ordinary "readers"

IMHO providing a method for them to pass, even in case of multiple signature triggers (as found in cidram) is a good thing.

While, occasionally, it will let pass a real human spammer, in most cases, it will be worth the risk.

[Maikuolan]

IMHO providing a method for them to pass, even in case of multiple signature triggers (as found in cidram) is a good thing.

Working on that now. :+1:

[Maikuolan]

@dev-101 @Dibbyo456 I've finished making the changes necessary for allowing a reCAPTCHA instance to be offered for up to an amount of signatures as per defined by the configuration, as opposed to per a strict limit of one (as is currently the case in the codebase). Changes currently working correctly on my local copy and will commit a little later today. A quick question first though: Should such an option be provided only for "blocked" IPs, or should it also be extended to "banned" IPs? ("Banned" IPs being IPs that exceed the limit currently defined by configuration for how many times they can be "blocked", after which, they should continue to be unconditionally blocked for future requests. Extending it to banned IPs would effectively provide a way for them to unban themselves, which I see as a potential big problem in some cases, but I figure it's worth asking what you think about this first anyhow).

[dev-101]

Banned IPs should definitely stay banned in my opinion, assuming that they are auto removed after a predefined period of time (sorry, haven't tracked all the recent changes, I guess they are?).

Thanks :beers:

[Dibbyo456]

Banned IP should stay banned.
This option should be available to blocked ip.

[Maikuolan]

Awesome; Thanks for the replies. I'll get the current changes committed shortly. :-)

Edit: Done.

assuming that they are auto removed after a predefined period of time

Yep. :-)

[Dibbyo456]

@Maikuolan Hi, I just updated 1.3.0-DEV+17334504 does the new use mode available in this version?

[Maikuolan]

There should be a new directive appearing in the configuration now, "signature_limit". If you set it a reasonably high number, it should allow reCAPTCHA to always be available for all blocked requests. Probably won't need an additional mode for reCAPTCHA with this new directive in place, I'm thinking.

As for the invisible reCAPTCHA: TBH, I'm not entirely sure (yet). You may be thinking, "but if I coded it, how could I not know whether it's available?!" -> On that note; I recently modified how the reCAPTCHA API is being called by CIDRAM as to make use of what the reCAPTCHA documentation describes as "__explicit rendering__", and according to the documentation, doing it that way should work correctly for both the standard v2 reCAPTCHA and the invisible reCAPTCHA (assuming that the correct mode is selected right at the beginning, when registering the keys for it via the reCAPTCHA dashboard in the first place), and so, in theory, it should now be available. However, I haven't been able to actually get it working correctly yet for any of my test copies, and at least, until I can get it working properly on some of my test copies, I'm not confident enough to say that it's definitely available (thus why I'm not entirely sure yet; the theory is good, but in practice, no success yet on my end).

I'm thinking it might be a problem with domain certificates for my test copies or something like that. The documentation does also mention that the invisible reCAPTCHA requires HTTPS (which of course, in turn, would require certificates). Next step in my plan (when I've got some more spare time for it, probably beginning of this coming week) was to double-check over my certificates to make sure that everything is in order, in case that's what's stopping it.

[caxanga334]

@Maikuolan I tried to use Invisible reCAPTCHA, changed sitekey and secret but I still received the V2 reCAPTCHA.

[Dibbyo456]

@Maikuolan @caxanga334 I'm having the same problem.

[Maikuolan]

Hm.. Must be a code issue then, rather than an environment issue. Thanks for testing it out. I guess I've got some more reading to do.

[DanielRuf]

The v2 and v3 have different API endpoints. Are you sure that it was the v2 script and not the v3 but displayed due too many reCaptcha submissions?

[Dibbyo456]

Are you sure that it was the v2 script and not the v3 but displayed due too many reCaptcha submissions?

@DanielRuf There is NO v3 in reCaptcha, at least not in my knowledge. But sometimes Invisible reCaptcha does show v2 recaptcha, but I'm pretty sure it wasn't the case.

[DanielRuf]

The new invisible reCaptcha is v3. Also if your ip / device has not a great reputation they will show the old reCaptcha as fallback.

[DanielRuf]

I'll check that later.

[dev-101]

Not sure if Google calls it officially v3 or not, that's not so important.

But, new keys must be used, and onload method must be updated (line 38 @ recaptcha.php).

[Maikuolan]

Got some help from a programming group I chat with sometimes. Figured out why it's not rendering, and it's nothing to do with SSL at all. Actually, it was just a missing parameter in the rendering code, so, reasonably easy to fix, thankfully. :P

Still having some binding issues though, and the discovery of this parameter which determines whether V2 or invisible is used would imply that we'd likely need an additional configuration directive to specify whether we're using V2 or invisible (in order for CIDRAM to know whether or not to include this parameter). so, not quite ready to release an update for it yet, but I'm still working on it, and hopefully should have something working for it reasonably soon.

[Maikuolan]

Done. :+1:

When you get the chance, let me know how it goes. :-)

[Dibbyo456]

@Maikuolan holy moly! page auto completes the captcha, then refresh? wow! did not see that coming.. Awesome.! 👍

[Maikuolan]

Yep. I figure, if we're going to use an "invisible" reCAPTCHA, it may be well be automated as well, in order to be truly invisible for users (but hopefully still keeping out bots by way of still running it all through the reCAPTCHA API). :-)

[Maikuolan]

Marking as "implemented" and closing.

If any problems emerge, let me know and we can reopen this issue. :-)