Wp-rocket: Htaccess corrupted on update

[webtrainingwheels]

A few customers report that on occasion, typically when updating, the htaccess file is regenerated with only with our rules and nothing else, creating 404 errors.
It doesn't seem to happen every time and seems almost impossible to reproduce, yet we have several reports now.
Is there anything we can do to make sure that at least the default WP rules are placed back into the file?
HS tags: htaccess, update fail, error 404
Intercom report: https://app.intercom.io/a/apps/llnz7wr8/inbox/inbox/46787/conversations/15589780670

[blogtutor]

This just happened again on one of my clients' sites, updating from 3.0.5.1 to 3.1.

The .htaccess file was left with only the WP Rocket 3.1 directives, stripping out other redirects and the standard WordPress .htaccess code.

As a result, every page on her site was a 404 error, except her homepage, resulting in downtime of about two hours until it was reported to us and fixed. :(

I know this is tough to reproduce, but we really need a solution here. Maybe every time WP Rocket makes edits to the .htaccess file, it can make a local backup, and then compare the new/old files, and then take action if the code outside of the WP Rocket code has changed?

[ecotechie]

Same problem here. Today is the fourth time this happens to one of the sites I maintain. Same as above, I updated to 3.1.1 and all directives were removed except the WP Rocket and WordPress ones. Seems like a problem with the code updating the file from previous versions.

[blogtutor]

@ecotechie Are you sure the WordPress directives weren't removed? I think Cloudways may have already added that back in by the time you got to it. In the past when I've encountered this issue, only the WP Rocket code was there - everything else was gone.

[ecotechie]

@blogtutor that may have been the case as there were many fingers on that file when the site stopped loading correctly.

[blogtutor]

Ack! This just happened again to another client. Last night she updated to WP Rocket 3.1.2 - and her .htaccess file ended up with _only_ the WP Rocket directives, and everything else stripped out.

Her site was down (except for her home page) for over 12 hours because of this...and I've also had to pull a copy of her .htaccess file from a backup in order to restore the rest of the directives that were in the file. (Re-saving the permalink settings gets the site back up, but we had other stuff in the file that was lost when WP Rocket stripped everything out.)

@Tabrisrp & @webtrainingwheels, is there any way to get this bumped up on the priority list?

[blogtutor]

OMG. Yet another client just ran into this issue! It's a good thing we've got backups of their .htaccess files...

This used to only happen very rarely... this latest update seems to have a much higher rate of occurrence. :(

[Tabrisrp]

Are they all hosted on the same hosting provider?

The only function that write to the htaccess is this one https://github.com/wp-media/wp-rocket/blob/master/inc/functions/htaccess.php#L13

And the part that is doing a removing of content is this $ftmp = preg_replace( '/# BEGIN WP Rocket(.*)# END WP Rocket/isU', '', $ftmp );

For some reason, it might be that this regex is not working as expected, removing all content instead of only the content between the WP Rocket markers.

a good test to determine if it's indeed the issue would be to check for the regex matching against one of the htaccess affected, on a site like https://www.phpliveregex.com/ for example.

[blogtutor]

HI @Tabrisrp - thanks for jumping in on this.

Most (maybe all) of the sites where we've encountered this are hosted on Cloudways, either Vultr or Linode servers.

And yep, it sure does sound like that preg_replace function is stripping everything out.

Here's one of the .htaccess files that was corrupted (I had to rename it to upload here.):
htaccess.txt

I tinkered with the phpliveregex.com site, and it seems to work fine on that .htaccess file:
https://www.phpliveregex.com/p/p1o
(gotta switch to preg_replace, but then it works)

[Tabrisrp]

Yes indeed it works correctly for this htaccess file. The issue might be later in function.

Are you able to reproduce the issue everytime if you update/rollback between versions, or is it happening randomly?

[blogtutor]

Thanks @Tabrisrp.

We haven't attempted to reproduce the issue on any sites (since these are live sites, after all, and we just wanted to get them working again). If it happens again on another site, though, we can give that a quick try.

At this point my hunch is that it's either (1) related to a specific server setting, and/or (2) the method in which the update is run. (Batch updates page vs. plugins page -- and if other updates are run at the same time). I just emailed the two most recent clients to see if they remember exactly how they ran the updates.

Also, I just emailed some more sample .htaccess files to Vasilis, per his request. Though there's not too much customization stuff in them, and they're very similar to many other clients who haven't had issues (such as http->https redirects).

[blogtutor]

Heard back from one client about her upgrade process:

I did the updates from the Dashboard> Updates. I did update several plugins at once, they were: Wp Rocket, SEO by Yoast and Social Warfare (I remember these exactly). Also Social warfare pro had update but it did not work. So out of 4 plugins, 3 updated and WP Rocket messed up.

[Tabrisrp]

Currently I'm leaning toward server related issue as, AFAIK, all the cases we had for this were on Cloudways.

The resulting htaccess does contain the WP Rocket rules, but the rest is stripped out, correct?

[blogtutor]

All of the recent cases have definitely been on Cloudways. The ones that happened weeks/months ago I don't remember for sure, but odds are good they were also Cloudways.

And yep, the resulting.htaccess contains the new WP Rocket rules (with the newest version number), and that's it.

[blogtutor]

This just happened again to another client. Same exact situation. She's on Cloudways hosting. She updated to 3.1.3 and ended up with an .htaccess file that had only the WP Rocket stuff -- everything else was stripped out. Attached are before/after .htaccess files.

She said: "I was on the 'All Plug-ins' page. I updated one plug-in first (I can't remember which) after checking the first box on the UpDraft pop-up. Then I updated the WP Rocket plug-in, but first unchecked the box on the pop-up so I didn't have to wait for additional backup to run."

(So it sounds like she ran plugin updates one-at-a-time, and everything updated correctly, so it seemed.)

htaccess_original.txt
htaccess_broken-by-update.txt

[blogtutor]

@Tabrisrp The problem seems to be getting worse. Over the holiday weekend this happened again on (at least) two more client sites when they updated WP Rocket. Both are on Cloudways.

Not sure if this info helps, but: One is running:

• Apache/2.4.10 (64-bit #1 SMP Debian 3.16.51-3+deb8u1)
• PHP 7.0.31-1~dotdeb+8.1
• MySQL 5.6.41.

The other site was running:

• Apache/2.4.10 ( 64-bit #1 SMP)
• PHP 7.2.9-1+0 20180901081207.4+jessie~1.gbpdaac35
• MySQL 5.6.41

We are now desperate for a solution and/or workaround here... how can I help get this resolved?

At the very least, can WP Rocket save a backup of the .htaccess file locally, before making the edits? Then at least restoring will be much quicker/easier.

Thanks!

[Tabrisrp]

I'm talking with Cloudways about this too, but they are not able to reproduce this on their side for now. I believe it would be really helpful if you also contact them, and provide them the list of websites where it's happening, because the only recent reports of this issue on our side are coming from you.

[blogtutor]

Thanks Remy. I will reach out to them -- with whom have you been talking? I'll try to connect to the same person.

(It also just happened on another client's site.)

Hey, might it be a file permissions issue? On Cloudways, the master user on the server is something like master_xndjwemw -- but WordPress runs under its own user (whatever the "application" username is). I run into hassles a lot when the WordPress user creates files (such as when installing a plugin from the repo) that are then owned by the application user name -- and then when I use FTP, logged in with the master_ account, the master_ account doesn't have sufficient permissions to edit those files. Maybe that has something to do with all this?

[blogtutor]

Unfortunately, this continues to be a significant problem; it happened again on another clients' site yesterday, and again this morning on a different site.

Both Cloudways support and I have not been able to replicate the problem with any consistency.

We need to find some fix, or at least a workaround, for this. This problem takes down sites completely (except for the homepage). Many of my clients are very high-traffic sites and downtime costs them hundreds of dollars an hour in lost revenue.

At this point, since we can't isolate the root cause of the problem, can you please at least add some error-checking so that in case it happens, there's an automatic solution in place?

[blogtutor]

@Tabrisrp - This happened yet again on another client's site this morning - she updated WP Rocket and boom, everything else in her .htaccess file was gone. Same exact issue, though I hadn't seen this site affected before. It's also on Cloudways.

At this point, since we can't isolate the root cause of the problem, can you please at least add some error-checking so that in case it happens, there's an automatic solution in place?

[vmanthos]

A customer has a site on Cloudways. I've updated WP Rocket from 3.1.4 to 3.2.2 without an issue.

However, when I opened the _htaccess_ file using Notepad++ I saw this:

Above the NULs there were WP Rocket's htaccess rules.

I removed the NULs, saved it to a new file and diff checked the two _htaccess_ files using Meld:

The original _htaccess_ can be downloaded from this link:
https://mega.nz/#!SpVUnS5S!42C5hVJlEXhL5EssO6QtDtKIc2Z_cho2UiRlykIDlK0

This may or may not be related to this particular issue. However, it's very strange and that's why I'm reporting it here.

Related ticket:
https://secure.helpscout.net/conversation/711583715/86840?folderId=2135277

[blogtutor]

Thanks for adding this! For what it's worth, I haven't seen this particular issue before on any of our sites that have had update issues with the rest of .htaccess getting stripped out.

[webtrainingwheels]

Possible case: https://secure.helpscout.net/conversation/717560494/88121?folderId=273761

[blogtutor]

Thank you!! 😃

[blogtutor]

Just had another client on Cloudways with this issue. It's been happening less often, but clearly it's still a problem.

😢

[blogtutor]

Yep, just happened again with a different site. Client updated, and boom, the .htaccess file lost everything except the (new) WP Rocket code.

[blogtutor]

...and it's still happening. Another client on Cloudways updated from v3.3.3 to v3.3.3.1 ... same exact problem.

@webtrainingwheels any other ideas?

[blogtutor]

@webtrainingwheels @Tabrisrp Can you please reopen this ticket? It's definitely not solved, unfortunately.

[blogtutor]

Just happened again on another site. Also hosted on Cloudways. Updated 3.3.2 to 3.3.3.1.

Interestingly, this site was updated with Staging Pilot -- which updates on a staging server first and then pushes the updated plugin files to the live site. I think this is the first time a Staging Pilot update has run into the issue.

[blogtutor]

...and just happened again, on another site that was updated via Staging Pilot.

This was the third site to have this issue on the most recent 3.3.3.1 update.

[ecotechie]

Hi @Tabrisrp seems like the commit that closed this issue is only checking for the standard WordPress copy in the .htaccesss file and not any of the custom code that gets stripped. Is this correct? If so, it won't solve this issue. Also I just had a .htaccess file stripped on another site and it took all the directives with it except the WP Rocket ones.

[vmanthos]

A case which seems similar:
https://secure.helpscout.net/conversation/977417384/126315/

This was on Dreamhost.

[blogtutor]

This is still a problem. We just updated a client from 3.4.4 to 3.5.1. Checked the site after the update and all seemed fine. A couple of hours later she emailed us because only her homepage was working. Sure enough, the .htaccess file had been stripped of everything except the WP Rocket code.

Happened on another site last week as well.

[goddas]

I've just had a similar issue that might be related to this. My client contacted me with the website being down and only the homepage working. I've checked the htaccess file and it was missing about bottom-500 of the 1000-something lines. So the first 500 or so lines contain the WP Rocket code and some redirects, while the other half containing more redirects and wordpress directives was missing. The website is hosted on cloudways/digital ocean server

[mifrero]

Related ticket: https://secure.helpscout.net/conversation/1192263109/172159/

[ecotechie]

Similar issue with another site. .htaccess was 636 lines, but the update stripped all but the first 134. Left behind the new WP Rocket directives and some Redirect 301 lines. This seems kinda new, since it used to strip ALL directives except the new WP Rocket ones, but this left some lines behind, like what happened to @goddas above.

Happy to email the files if wanted....

[webtrainingwheels]

https://secure.helpscout.net/conversation/1297510459/198847?folderId=377611
It's happened to this customer for the last 2 updates. Hosted on 02switch.
It's never reproducible after the first update attempt. Rolling back and updating again will be successful.
What's happening from our side only in the very first update routine for each version?

[vmanthos]

@blogtutor Reported this is still occurring for them:
https://secure.helpscout.net/conversation/1302688547/200372?folderId=3326173

[vmanthos]

Possibly related ticket: https://secure.helpscout.net/conversation/1319380832/205461/
The site is hosted on OVH.

[WIP] I'm investigating a similar case that took place on OVH: https://secure.helpscout.net/conversation/1302165472/200249/#thread-3776342425

Please find the customer's comment and my findings in the note here: https://secure.helpscout.net/conversation/1319380832/205461/#thread-3804061230

[vmanthos]

Update: I heard back from one of the customers (ticket).

I had prevented WP Rocket from writing to the htaccess file on their site. Despite that, the file was corrupted after a few weeks.
Other plugins were writing in that file.

Most likely this was related to the host, OVH in this case.
If a process takes too long, it may timeout, and that can corrupt the file.

[DahmaniAdame]

Possibly related - https://secure.helpscout.net/conversation/1335422175/211113/

[vmanthos]

Possibly related: https://secure.helpscout.net/conversation/1366892923/223063/

[NataliaDrause]

happened for customer again when updated to 3.8: https://secure.helpscout.net/conversation/1368024563/223391

[blogtutor]

This is continuing to happen for our clients on Cloudways servers. I cannot emphasize enough how problematic this is! It takes the entire site (save the homepage) offline, and leads to an extremely panicked site owner, often requiring emergency response/support at off-hours.

PLEASE PLEASE PLEASE FIX THIS! I BEG YOU.

[blogtutor]

Also, have you actually talked with Cloudways about this? Clearly there's something unique about their setup -- even it is "just" nginx in front of apache... Their frontline support isn't going to be helpful, but perhaps you could work with one of their senior techs or developers?

I have emailed with a couple of Customer Success Managers and Associate Vice Presidents there -- happy to send along their contact info if that would be helpful.

[vmanthos]

Related ticket: https://secure.helpscout.net/conversation/1366892923/223063/
Host: O2switch

[vmanthos]

Related ticket: https://secure.helpscout.net/conversation/1374886981/225451?folderId=3326173
Host: Hetzner.

[vmanthos]

Related ticket: https://secure.helpscout.net/conversation/1389568405/229477?folderId=2135277
Host: Dreamhost

[NataliaDrause]

Related ticket: https://secure.helpscout.net/conversation/1391871951/230082?folderId=3864740
Host: WPopt, which uses Litespeed web server

[NataliaDrause]

Related ticket: https://secure.helpscout.net/conversation/1392031396/230125?folderId=3864740
Host: OVH

[vmanthos]

Potentially related ticket: https://secure.helpscout.net/conversation/1401738049/232928/

[vmanthos]

Related ticket: https://secure.helpscout.net/conversation/1404506358/233660?folderId=2135277

Host: Variomedia AG

[DahmaniAdame]

Related ticket - https://secure.helpscout.net/conversation/1429056639/240402/
Bluehost - PHP 7.2.34

[Tabrisrp]

New approach to this issue:

Scope a solution ✅

Use insert_with_markers() to handle the writing of our rules in flush_rocket_htaccess(). This function is from WP core, and is a more advanced version of what we were doing (including some locking on the file).

The new function would look something like this:

function flush_rocket_htaccess( $remove_rules = false ) { // phpcs:ignore WordPress.NamingConventions.PrefixAllGlobals
    global $is_apache;

    /**
     * Filters disabling of WP Rocket htaccess rules
     *
     * @since 3.2.5
     * @author Remy Perona
     *
     * @param bool $disable True to disable, false otherwise.
     */
    if ( ! $is_apache || ( apply_filters( 'rocket_disable_htaccess', false ) && ! $remove_rules ) ) {
        return false;
    }

    if ( ! function_exists( 'get_home_path' ) ) {
        require_once ABSPATH . 'wp-admin/includes/file.php';
    }

    if ( ! function_exists( 'insert_with_markers' ) ) {
        require_once ABSPATH . 'wp-admin/includes/misc.php';
    }

    $htaccess_file = get_home_path() . '.htaccess';
    $insertion     = '';

    if ( ! $remove_rules ) {
        $insertion = get_rocket_htaccess_marker();
    }

    // Update the .htaccess file.
    return insert_with_markers( $htaccess_file, 'WP Rocket', $insertion );
}

This change was tested as part of some R&D, and is working well on a local installation.

This also should be done as part of our refactor of the htaccess related code (#2759), and unit/integration tests added for it.

Estimate the effort ✅

For this specific part, effort is [S]

[NataliaDrause]

Likely related: https://secure.helpscout.net/conversation/1443506061/244478/

[vmanthos]

Probably related: https://secure.helpscout.net/conversation/1453543452/247525