Cidram: Country blocking not working in some cases.

[Dibbyo456]

See below.

[Maikuolan]

Hi,

I notice you've closed this issue. Does the problem still persist or has it been resolved?

Cheers.

[Dibbyo456]

@Maikuolan Hi dear, today I faced a very weird problem in CIDRAM.
Yesterday someone asked how to block certain country to access my site in GulshanKumar Forum. It's a community built by the moderator of shoutmeloud.com. And as you can probably guess I suggested him CIDRAM.

He was not able to configure it first, but luckily I walked him through. When I was walking him through for testing purpose I blocked United States & Romania in my CIDRAM config (both ipv4 & ipv6). But surprisingly I could still visit my website using VPN's which IP belongs to US & Romania, that's when I opened this issue. But after few mins, when I was checking my logs, I could see some of the request were blocked because of US ip. But in the same time I could still visit websites using US VPN or check my websites in pingdom which ip belongs to US.

So, what's just happening?

[DanielRuf]

Did you clear all caches? Is REMOTE_ADDR used for checking and which IP was your VPN service using?

Were your blocklists uptodate?

Keep in mind that IPv4 addresses often change their owner and DNS functions and services are often not very accurate.

[Dibbyo456]

@DanielRuf

Did you clear all caches?

Yes.

Is REMOTE_ADDR used for checking

Yes.

which IP was your VPN service using?

US ip. Hotspot Shield VPN

The user i suggested him CIDRAM, also faced the same problem!

[RealSuprim]

Hello guys,

It was me who was trying to block India. On my log file it says it has blocked 10,000 entries but when I try to browse website using vpn or try any other speedtest using India as test centre the website was working from there. Also on my google analytics I can still see a lot of users from India. I also asked one of my friend who lives in India and he was also able to browse the site, I don't know what's happening.

[ghost]

@hekasi @Dibbyo456 Can you tell the some exact IP-addresses where the connection wasn't blocked?

The country lists source is MaxMind's free Geolite data, and updated once each month.
The current version at time of posting is 2018.65.364, is that the one you have installed?

[RealSuprim]

Currently My friend is not at home so I can't tell the ip of his. Yes I installed 2018.65.634 version

[RealSuprim]

Yeah this is his ip 2405:204:8104:632d:99aa:bcbc:bb6e:eaa7

[ghost]

2405:200::/29 is in the IPv6 file and should be blocked.

@hekasi You installed the IPv6 version as well? (IPv4 and IPv6 are separate files)

[RealSuprim]

@macmathan Yes I installed both file.

[ghost]

@hekasi And file is activated in CIDRAM?
(If you look at the configuration page under signatures->ipv6:, do you see it in IN_cidram_ipv6.inc listed there? Does it say Component active on the update page?)

[Dibbyo456]

I guess it does. Why else his logs shows hundered of requests blocked due to the ip belongs to India?

[RealSuprim]

yes it is there

[ghost]

@hekasi If you paste it in the IP-test function of CIDRAM, what does it report then?

I tried on a local install matching the latest signature set, and updated CIDRAM (fresh dev version) and it said a block would occur;

IP Address | Blocked
2405:204:8104:632d:99aa:bcbc:bb6e:eaa7 | Yes – No access allowed from India ("IN_cidram_ipv6.inc-IPv6", L607:F0, [IN])

[RealSuprim]

@macmathan weird it says blocked. But he says he can access the site?

[ghost]

Cache issue?
Either website cache or client cache?

[Dibbyo456]

@macmathan could it be any cache issue of CIDRAM?

[Dibbyo456]

@macmathan I tried different browsers, ingocnito mode. Same result.

[ghost]

Any server cache involved?
Is CIDRAM called before anything else?

[RealSuprim]

I deleted my website cache and he used incognito mode

[ghost]

Proxy cache?
I'm running out of ideas. It should work as the CIDR is correct and the IP-test report a block.
I think we have involve @Maikuolan on this to figure out what is going on.

[Dibbyo456]

I don’t have any proxy services like Cloudflare or similar.
The only firewall I have installed is CIDRAM.

[RealSuprim]

I also don't use any proxy service, I have wordfence installed that's it.

[Dibbyo456]

Disable Wordfence.

[RealSuprim]

Okay what security plugin to use?

[Dibbyo456]

I meant try disable Wordfence & see if the issue still exists.

Btw.. I don’t use any security plugins. Because all of them sucks. I just movied my wp-login.php to a different location

[RealSuprim]

Oh okay

[DanielRuf]

I just movied my wp-login.php to a different location

That does not help much. Security by obscurity is never secure.

Try NinjaFirewall.

[RealSuprim]

I might give NinjaFirewall a go, Wordfence makes website really slow.

But for now I really need to block India. Is there any alternative method? I tried with .htaccess way but it made my website so slow I was unable to access my site until traffic went down.

[DanielRuf]

Filling .htaccess with IP ranges is not recommended. Some hosting providers have the Apache GeoIP module enabled. This could be an alternative.

[RealSuprim]

Thanks @DanielRuf I managed to block the website using csf_Deny from driectadmin panel is it a safe way to do?

Thanks

[Maikuolan]

Sorry about the delays in replying. Busy day today. About to head to work but will reply properly when I've got a moment (should be within the next few hours).

[Maikuolan]

Quick question: On the website where we're seeing this problem occur, is "Endurance Page Cache" currently installed? We __encountered an issue recently__ on another website where it was discovered that this particular plugin isn't compatible with CIDRAM, and it is also sometimes installed on WordPress websites by various hosting providers without authorisation from website owners. When installed, unexpected results may sometimes occur (non-blocked traffic seeing the blocked message and visa versa).

[Dibbyo456]

@Maikuolan I don't know about his, but my website does not have any firewall, because my server is completely managed by me. The problem also occurring to me.

I'm getting some BingBot false/positive, how do I report?

[Dibbyo456]

@Maikuolan This IP's are being blocked, even they're valid BingBots.

131.253.27.138 »    1 (5.26%)
131.253.27.82 » 1 (5.26%)
131.253.25.171 »    1 (5.26%)
131.253.25.194 »    1 (5.26%)
131.253.25.156 »    1 (5.26%)
131.253.27.149 »    1 (5.26%)
131.253.25.184 »    1 (5.26%)
131.253.27.29 » 1 (5.26%)
131.253.25.173 »    1 (5.26%)
131.253.25.158 »    1 (5.26%)
131.253.27.37 » 1 (5.26%)
131.253.25.157 »    1 (5.26%)
131.253.27.129 »    1 (5.26%)
131.253.27.47 » 1 (5.26%)
131.253.25.196 »


msnbot-131-253-27-138.search.msn.com »  1 (5.26%)
msnbot-131-253-27-82.search.msn.com »   1 (5.26%)
msnbot-131-253-25-171.search.msn.com »  1 (5.26%)
msnbot-131-253-25-194.search.msn.com »  1 (5.26%)
msnbot-131-253-25-156.search.msn.com »  1 (5.26%)
msnbot-131-253-27-149.search.msn.com »  1 (5.26%)
msnbot-131-253-25-184.search.msn.com »  1 (5.26%)
msnbot-131-253-27-29.search.msn.com »   1 (5.26%)
msnbot-131-253-25-173.search.msn.com »  1 (5.26%)
msnbot-131-253-25-158.search.msn.com »  1 (5.26%)
msnbot-131-253-27-37.search.msn.com »   1 (5.26%)
msnbot-131-253-25-157.search.msn.com »  1 (5.26%)
msnbot-131-253-27-129.search.msn.com »  1 (5.26%)
msnbot-131-253-27-47.search.msn.com »   1 (5.26%)
msnbot-131-253-25-196.search.msn.com »

[DanielRuf]

I'm getting some BingBot false/positive, how do I report?

Please create a new issue to make it easier for us and others to follow the single issues.

[Maikuolan]

@Maikuolan This IP's are being blocked, even they're valid BingBots.

The BingBot problem might be related to __this__. I encountered problems with both Bing and Yandex being blocked briefly over a 1 day period at the beginning of February this year, but I wasn't able to replicate the problem at the time, and the problem also seemed to disappear from the affected website of its own accord, without my making any changes at all. Being unable to replicate the problem and being that I wasn't seeing it anymore, I put it down to just being a quirk of the affected server or possibly some external networking issue.

Happy to keep looking into it, but a bit short on ideas for replicating the problem at this point too. :-/

[DanielRuf]

As these are IPv4 addresses they might change their owner very often and quickly.

[dev-101]

This particular range is well-known bingbot's that hasn't been changed in years.

[Maikuolan]

^ Both above comments are true. In any case though, current implementation in CIDRAM for verifying search engines such as BingBot (as well as Googlebot, YandexBot, Baidu and others) does follow their own guidelines (forward/reverse lookup from IP to hostname, hostname back to original IP), which is generally preferable. Plus, for the most part, for most websites, it seems to work correctly. But then we have cases like these, where it doesn't, for reasons as yet unknown.

The ideal solution, if we can find a way to do so, would be to actually get to the bottom of why it's not working sometimes, rather than simply building workarounds that could potentially become a permanent feature of CIDRAM if these bottoms aren't ultimately resolved, or at least better understood. If we end up needing to do that anyway, then so be it, but I'd rather gain a better understanding of the problem first, if I could. Not currently sure how to go about doing that though.

[Maikuolan]

(In terms of things like Unit Tests: Though the specific code for search engine verification isn't currently checked by TravisCI or PHPUnit, my general feeling, due to that the problem seems to affect only specific websites, at sporadic times, even when those websites are using the same CIDRAM codebase as other websites where the problem isn't occurring, has been that it might be some server environmental thing, or the way that CIDRAM interacts with DNS/IP/network lookups within specific environments or something like that. I'm just speculating at the moment though).

[Maikuolan]

Also, for what it's worth: I'm not sure whether or not I'd already mentioned it before, but I've tested Macmathan's India blocklists both on two different websites that I control, as well as on my local testing environment and my main dev copy of CIDRAM, and it seems to work perfectly fine on all them currently (i.e., I haven't yet been able to replicate the problem originally mentioned by this issue).

[Dibbyo456]

I haven't yet been able to replicate the problem originally mentioned by this issue

I would say, keep aside the issue for now, I would try to give you more information later.

Let's focus on Search Engines. Because that is the biggest concern right now.

[Maikuolan]

I would say, keep aside the issue for now, I would try to give you more information later.

But, Search Engines is biggest concern right now.

Definitely, and much appreciated. :-)

[Maikuolan]

..More speculation of course, but I wonder if it might be some problem with reading specific files (signature files, cache file, etc) due disk I/O bottleneck? Like.. Too much going on in the background on the server, so calls to read specific files just outright fail for specific requests and that sort of thing.

[Dibbyo456]

@Maikuolan Something weird is going on..

IP - 131.253.27.138

Check this - https://whatismyipaddress.com/ip/131.253.27.138

And try the IP on this - https://www.bing.com/toolbox/verify-bingbot

Verdict for IP address 131.253.27.138:
No - this IP address is NOT a verified Bingbot IP address.

What? hostname saying valid bingbot, but bingbot own tools saying not legit bot!

[Maikuolan]

That's.. Definitely interesting.

Screenshots, for the benefit of others:

The first verification tool (shows as BingBot):

BingBot's own verification tool (shows as NOT BingBot):

And for safe measure, checking manually via PHP in CLI on my own system (also shows as BingBot):

[Dibbyo456]

Also CIDRAM blocking it for the reason of Cloud service ("Azure", L20786:F0, [US])

But @Maikuolan I thought, CIDRAM checks the IP against their hostname, right? If so, the IP should pass, because it's having a valid msn hostname? Why it's not passing?

[Maikuolan]

Regarding speculation about possible disk I/O bottlenecks: Currently in the code, if the "ReadFile" closure can't read the file (CIDRAM's internal wrapper for reading local files), it returns "false" (which could also be considered equivalent to successfully reading an empty file in any cases where the wrapper is used by something that doesn't check the type). Also, in the "CheckFactors" closure (this closure is the main powerhouse and most important part of CIDRAM's ability to fetch signatures and compare an IP address against them), if a signature file is deemed to be "empty" (i.e., attempting to read the signature file fails or results in no data), it will automatically and silently skip ahead to the next signature file in the queue:

        if (!isset($CIDRAM['FileCache'][$Files[$FileIndex]])) {
            $CIDRAM['FileCache'][$Files[$FileIndex]] = $CIDRAM['ReadFile']($CIDRAM['Vault'] . $Files[$FileIndex]);
        }
        if (!$Files[$FileIndex] = $CIDRAM['FileCache'][$Files[$FileIndex]]) {
            continue;
        }

So in theory, if somehow a signature file couldn't be read properly, that would effectively cause whatever IPs the signature file is meant to blocking to no longer be blocked for the given request instance.

[Maikuolan]

@Maikuolan Also CIDRAM blocking it for the reason of Cloud service ("Azure", L20786:F0, [US])

But @Maikuolan I thought, CIDRAM checks the IP against their hostname, right? If so, the IP should pass, because it's having a valid msn hostname? Why it's not passing?

That particular issue, of it showing as being blocked due to the Azure signatures when checking it via the IP test page, I can understand. The search engine verification code currently is implemented by the output generator, which the IP test page doesn't use. That, I'll need to fix, I think; and it should be reasonably easy to fix (..consider it now on the to-do list; I'll get that sorted prior to the next release). Still.. If actual users are being blocked, that'd be a separate issue to the issue of the IP as showing as being blocked by Azure when checking it via the IP test page.

[dev-101]

if a signature file is deemed to be "empty" (i.e., attempting to read the signature file fails or results in no data), it will automatically and silently skip ahead to the next signature file in the queue

This may be considered as a vulnerability of a sort, if you overload the server with too many requests it can't handle (for example) you may get in (at least for one page). Not sure if this should be default behavior, it has both advantage and disadvantage.

[dev-101]

btw. there's no need to always use mentions (@) as anyone already participating here will get notified anyway. Also, if you click follow (watch) on top, you'll always get notifications ;)

[ghost]

I'm not in a place to check this up (on mobile), but can the bingbot issue be that Microsoft has messed up their DNS again?

I.e. dual hostnames, or an update not yet propagated to all servers? It wouldn't be the first time, but it doesn't have to be the reason either. Just a thought thrown out.

DNS failures are a tricky thing under PHP. The internal functions are not 100% reliable (although they work fine in like 99,9%), and creating workarounds are often dependant on the server environment, so they are also not a 100% sure bet.

I wouldn't be too surprised if we can attribute temporary failures that we can not put the finger on to this internal unreliability.

[Dibbyo456]

Some more..

msnbot-199-30-24-185.search.msn.com » | 1 (0.29%)
msnbot-199-30-25-161.search.msn.com » | 1 (0.29%)
msnbot-199-30-24-166.search.msn.com »

[Maikuolan]

Reopening #50. We have some minor progress there, and I think it might be the cause of the search engine verification problems we're encountering.

At this point, we should keep this issue (#54) on-topic (i.e., stick to the original problem concerning country blocking not working sometimes) and use #50 for discussion regarding search engine verification.

[Maikuolan]

BTW, do we know specifically which other IPs are getting through (from the VPNs and so on)?

[Maikuolan]

Hi @RealSuprim @Dibbyo456 ,

When you two have a spare moment, could you let me know whether this issue persists, and if so, which IPs are still getting through? Cheers. :-)

[Dibbyo456]

You can close the issue for now.
If I trigger the behaviour again, I will reopen it.

[Maikuolan]

Ok, sounds good. :-)