Cidram: Problems with CDN

[Dibbyo456]

Hello,

Today I started using BunnyCDN for my websites, but unfortunately now I'm having problems.
BunnyCDN can't pull files due to CIDRAM.

Possible solutions?

[DanielRuf]

Please provide some more debugging information.

[Dibbyo456]

This might help.

https://support.bunnycdn.com/hc/en-us/articles/115003578911-How-to-detect-when-BunnyCDN-PoP-servers-are-accessing-your-backend

https://bunnycdn.com/api/system/edgeserverlist

[Dibbyo456]

@DanielRuf I can't seem to find any particular debug other than this.

ID: 3562
Script Version: CIDRAM v1.5.0
Date/Time: Wed, 21 Mar 2018 19:59:23 +0530
IP Address: 139.59.27.59
Referrer: https://static.xxxxxxxxxxxxxxxxx.com/favicon.ico
Signatures Count: 1
Signatures Reference: 139.59.0.0/16
Why Blocked: Cloud service ("Digital Ocean, Inc", L3929:F0, [US])!
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
Reconstructed URI: https://static.xxxxxxxxxxxxxxxxx.com/favicon.ico
reCAPTCHA State: Failed!

[Dibbyo456]

According to the support page,

CDN-ServerId HTTP Header

The most basic method of detecting our edge server is to look for the CDN-ServerId header in your backend. Our servers will always send this header with all the requests and the value will include the internal server ID.

I think I should create a custom function, which will check CDN-ServerId HTTP Header.
If CDN-ServerId exists in the HTTP Header. Then request coming from them.

But I have no idea, how to create a custom function like that, help? @DanielRuf @Maikuolan

[Maikuolan]

Interesting that the log entry explicitly states that the reCAPTCHA failed. That would imply that they (or something) attempted to complete it (but didn't get through). Not something that a CDN should normally do, AFAIK. o.0

Anyhow.. As for a working solution:

Edge Server IP List API

We also provide an API that automatically returns the list of IPs that are currently used by the BunnyCDN network. However, due to the design of our system, these can change relatively frequently, so we highly suggest to automate this to prevent connectivity problems. To get access to this API call, please open a support ticket.

Depending on how accessible this API is, whether they'd allow it to be exposed publicly (e.g., via a publicly accessible module), and on what format the data is provided as, I'm thinking that there might be a viable solution in that (like an optional module that checks their API, caches results, compares the IP of inbound requests against their list of APIs, and lets the request through if it's a match). I can't see anything about this in their public documentation though, so I'd probably need to open support ticket there to find out about it (as per suggested on that page). I'll do that shortly.

In the meantime, until they get back to us, a temporary solution - assuming, of course, that they're operating solely from Digital Ocean, and not through anyone else - could be to add "Digital Ocean, Inc" to the ignore.dat file. That should let them through for now (in theory).

[Dibbyo456]

@Maikuolan But don't you think checking if CDN-ServerId exists in HTTP Headers, would better? less stress?

[Dibbyo456]

whether they'd allow it to be exposed publicly

It's public. - https://bunnycdn.com/api/system/edgeserverlist

[Maikuolan]

I think I should create a custom function, which will check CDN-ServerId HTTP Header.
If CDN-ServerId exists in the HTTP Header. Then request coming from them.

But I have no idea, how to create a custom function like that, help? @DanielRuf @Maikuolan

It should work, as far as ensuring that they don't get blocked and that they'll be let in, but my concern about relying on HTTP headers as a permanent solution is that HTTP headers can be easily forged, and if anyone that we want to keep blocked catches on that they can let in by citing a specific HTTP header, there's not much reason why they wouldn't do that eventually.

[Maikuolan]

It's public. - https://bunnycdn.com/api/system/edgeserverlist

Cheers. :-)

That looks pretty easy to work with actually. Shouldn't be too difficult to code up a quick module for them using that data, I think.

[Dibbyo456]

We also need a caching mechanism if we are going to use the API. Right?

We are not going to hit their API, every single time.

[Maikuolan]

We also need a caching mechanism if we are going to use the API. Right?

We are not going to hit their API, every single time.

Yep, correct. We'll definitely be caching the lookup results. Always good to cache API lookup results. Not doing so is a pretty surefire way to get blacklisted by whoever's providing the API in most cases.

[Maikuolan]

(About to head to work for the night, but I'll reply again within the next few hours).

[Maikuolan]

Seems a viable solution. I'll start working on a new module a little later today and reply here again when it becomes available. :-)

[Maikuolan]

Module has been written and committed. :-)

You should see "BunnyCDN compatibility module" now listed on the updates page. If you install and activate that (you'll also need to update to the latest signatures and bypasses, due to that there were some small changes needed there in order to get the new module working properly), BunnyCDN won't be blocked anymore.

[Dibbyo456]

That’s awesome. 😘
May I ask what’s the Cache TTL?

[Maikuolan]

I've set the TTL to 4 days (or 345600 seconds).

    $CIDRAM['Cache']['API']['BunnyCDN'] = [
        'Data' => $CIDRAM['Request']('https://bunnycdn.com/api/system/edgeserverlist') ?: '',
        'Time' => $CIDRAM['Now'] + 345600
    ];

Although they don't seem to mention anywhere in their documentation which TTL they would prefer, I'm thinking that 4 days is a pretty fair and reasonable balance between updating frequently enough to prevent too many false positives, and caching for long enough to prevent anyone on their side getting upset about excessive server requests and so on.

[Dibbyo456]

Yup.. 4 days seems good to me.

[soumsps]

I am not able to install this BunnyCDN Module because I am using PHP 7.0. Is there any workaround or could you make it compatible with PHP 7 also?

In the present situation it says " Component not installed (requires PHP 5.4.0)! "

[Maikuolan]

Hm.. Sounds like something has definitely gone wrong there. The module should be compatible with PHP7 already. I'll take a look at it now.

[Maikuolan]

Having checked over the responsible code again, I suspect the problem might be because of the "VersionCompare" function incorrectly reporting that your PHP version is older than the minimum required version. To be able to test whether my suspicions are correct, I'll need to know exactly what your PHP installation reports as its version.

When you've got a spare moment, could you test the code included below using your PHP installation, and let me know what you see? Thanks. :-)

<?php
echo PHP_VERSION;

[soumsps]

On doing <?php echo PHP_VERSION;
I got 7.0.25-0ubuntu0.16.04.1

I have also cross checked with CIDRAM Home Front page for system info:

System information:
CIDRAM version used: 1.4.1
Branch latest stable: 1.4.1 – Latest stable: 1.4.1 – Latest unstable: 1.5.0
PHP version used: 7.0.25-0ubuntu0.16.04.1
Branch latest stable: 7.0.28 – Latest stable: 7.2.3 – Latest unstable: -
SAPI used: apache2handler
Operating system used: Linux xxxx.xxxxxxxxxxx.com 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64

I am on Cloud hosting with VestaCP (0.9.8) installed.
VestaCP cannot have multiple php versions installed.
Normally in Cpanel we get a php version selector. (I have tried on NameCheap Shared Hosting and found CIDRAM was working properly)

Again to note there are many modules that are failing to update or install, Infact i am not able to update to ver 1.5.0 just because of "(requires PHP 5.4.0)! "

Here is a complete Updates page screen shot here

[Maikuolan]

Cheers. Suspicions have been confirmed. I'll begin working on a fix now.

[Maikuolan]

Fix committed to the codebase. :-)

Because the problem, in this case, is preventing you from updating normally, you'll need to manually add the fix to your copy of CIDRAM, so that you'll be able to resume updating normally. After you've manually added the fix, you should (if all goes well) be able to update everything else normally using the update page, as well as install modules and so on.

Note: Only users experiencing this exact problem should need to update manually, and should only need to do so just one time. Users not experiencing this exact problem should just continue using the normal update methods that they already use.

How to manually add the fix:

1. From the file manager, locate "frontend_functions.php". From the menu directly to the right of the file, click "edit", and "OK".
2. You should now see a textbox, containing the contents of that file. Search for this one of code (Ctrl+F, in most browsers, will allow you to search more quickly):

preg_match('~^v?([0-9]+)\.([0-9]+)\.([0-9]+)(RC[0-9]{1,2}|-[0-9a-z_+\\/]+)?$~i', $Ver, $Matches) ?:

1. When you've found that line of code, replace it with this (the actual change is just adding one small dot, but it fixes the problem; be careful that this is the only change you make here though, because changing the wrong thing could break your installation):

preg_match('~^v?([0-9]+)\.([0-9]+)\.([0-9]+)(RC[0-9]{1,2}|-[.0-9a-z_+\\/]+)?$~i', $Ver, $Matches) ?:

1. Click the "edit" button to confirm the changes and save the file. Return to the update page, and click "Update all". It should be working normally again. :-)

[soumsps]

Thanks a lot, after performing above fix now my version of CIDRAM is working as expected.

[Maikuolan]

Awesome. :-)

[DanielRuf]

Is the native version_compare or a semver module of Symfony not used? Makes more sense than such a regex imho.

[Maikuolan]

The regex part actually just normalises version numbers, due to that I've sometimes seen non-normalised version numbers before in certain custom PHP builds (e.g., "7", instead of "7.0.0", or "5.6.33-im.a.special.service.provider-and-add+weird+crap+to-my+php_build_version-numbers.123", instead of just "5.6.33"; thankfully not too common though), which can badly mess up the results of version_compare when used directly (due to that it relies on "PHP-standardized" version number strings), and then splits the versions into distinct, progressively optional parts (major, minor, patch, build) - At least, in theory (the problem mentioned by soumsps is a case where it failed to split the versions into distinct parts, because the dot in the build part of the version number wasn't included in the regex responsible for doing the splitting; fixed now though).

So, using version_compare directly (without normalising anything), "7" and "7.0.0", for example, would be considered different, and therefore not equal versions (one being higher or lower than the other, depending on which order they're supplied as parameters to the function). Using our own though, these two version numbers would be treated as equal (due to being normalised).

A somewhat weaker argument, but also: It's useful to be able to tweak how versions are normalised if, at some point in the future, other devs join the project to build their own components, modules and so on, and start using weird, distinctly non-Semver-like version numbers for their work (which, I would strongly advise them against doing, especially seeing as we're already sticking to SemVer for the project itself, and SemVer-like versions for most peripheral components, but it's easy enough to be prepared for it anyway, just in case).

After the versions have been normalised and split into parts though, the actual comparison is just done using simple, logical operators ($this === $that, $this > $that, etc). Always open to possible improvements though, of course. :-)

I haven't seen Symfony's srmver module before, so I can't comment much on that. I'll check it out though.

[Maikuolan]

A possible counter-argument to my own argument above: PHP's native version_compare should be able to compare an infinite number of version parts (i.e., 1.1.1.1.1.1.1.1 and 1.1.1 should be equally easy to compare and so on), which isn't possible with the "VersionCompare" function I came up with (due to that it would be limited to major, minor, patch, and build parts), so any future devs wanting to implement non-SemVer-like versioning schemas would be somewhat limited in terms of how far they could take it (assuming that nothing had changed by that point in time). Not too likely to happen though, so not too likely to be a problem in the future, I think.

[DanielRuf]

https://github.com/composer/semver

I guess I had a bundle in my mind. But there is something that Symfony uses and is one of their numerous components.

[Maikuolan]

Cool. I'll take a closer look at that tonight.