Wp-rocket: Regenerate Specific CPCSS when default CPCSS is regenerated

[arunbasillal]

With WP Rocket 3.6, we will be introducing "Specific CPCSS" for post types. Customers will be able to generate a CPCSS for a specific page.

As per the current design, we haven't account for its automatic regeneration. However we need to regenerate it automatically, otherwise customers will have to manually remember which posts had specific CPCSS and then go through each of them to regenerate it.

When to do it?

• When customer manually chooses Regenerate Critical Path CSS from the admin bar or WP Rocket dashboard.
• Theme is changed
• Any other time we regenerate CPCSS automatically.

Ideas:

• One obvious way is to store the post ID's in the database
• We could "read" the ID's from the \wp-content\cache\critical-css\1\posts folder instead and thus not have to deal with managing the database entry.

Note from Remy:

It might increase the generation time exponentially when you click on regenerate or switch theme. Even more with CPCSS for mobile coming, which will double the amount of generation needed

[crystinutzaa]

Reproduce the issue ✅
The CPCSS is not regenerated for specific pages while recreating the CPCSS.

Identify the root cause ✅
The root cause is the initial implementation which only generates and deletes the specific CPCSS on Ajax call.

Scope a solution ✅
@Tabrisrp , @hellofromtonya we need to discuss a solution.
At this point, we don't have any database entries which will save the specific posts which have CPCSS. We have to save these specific posts in wp-rocket configuration options while generating & deleting it so we don't have to access each time the folders of the specific CPCSS to read it (will be faster instead of reading the file / folder structure and much more accurate).

Also I would go with another Background_Process only for this generation which will run in parallel with the "main" one and to display a new notification related only to Specific CPCSS.

What do you think? Will this do it?

Effort ✅
Effort [M] - this seems a larger task. If you think I am overestimating it please let me know.

[arunbasillal]

Also I would go with another Background_Process only for this generation which will run in parallel with the "main" one and to display a new notification related only to Specific CPCSS.

Can we avoid the new notice? The customer clicked to regenerate CPCSS and ideally everything that we tell them should be in the same notice. (cc: @GeekPress)

[Tabrisrp]

I think the simplest approach would be:

• Save a post meta when we generate the CPCSS for a post
• On upgrade to the version including this change, scan the CPCSS posts directory, and save a post meta for all the already generated posts
• When starting a global regeneration, do a DB query to get all posts with the post meta

This is all happening in the admin context, so relying on the database is ok and all the tools to do that are already there.

I agree with Arun we can keep everything in the same notice.

[Tabrisrp]

I think this task is underestimated, and should be broken down into sub-tasks. The dev team will have a look at it together to discuss it.

[hellofromtonya]

@Tabrisrp Let's put this one on our agenda for tomorrow's meeting.

[hellofromtonya]

@GeekPress @arunbasillal The effort for this one is increased. This is now an epic and will need sub-tasks. Why?

When we first scoped this solution, we were planning to use post meta data. Since then, we've reconsidered our approach to look at other impacts and benefits. Instead of using post meta, we'll create a custom db table.

What are the benefits to this approach?

• Freshness: track freshness to determine _when_ a regeneration is needed.
  This field saves requests on the host's server and our CPCSS API. Only (re)generate what's needed.

• Faster SQL Queries: pull only what needs to be generated or regenerated all types of content including taxonomies.
  Why? Less JOINs to query different types of content.

• Faster execution. How?
  How? We can pre-map each by capturing its path, URL, and job ID in a single db record.
  Why is this good? Currently all of the CPCSS paths are captured together in one transient.

• Lower risk of async operations causing issues.
  Why? We work in one transient with asynchronous processes (background worker and JS async). With one transient, there's a higher risk of one process changing the transient while another is doing something else to it.

• Future: this approach will allow us to innovate in the future.
  For example, on web hosting that has more resources available, we could spin up more background workers for parallel processing. Using a database table helps us to robustly and reliably interact between these async processes and the data.

• Reuse in Preload
  We are planning this same custom db table for Preload for the same reasons. As we design this, much of the db handler and management will be reused.

Next Steps

👉 The team is working together to break down this epic into smaller sub-tasks and estimating the effort for each.

[GeekPress]

I move this to a major version :)

[GeekPress]

@hellofromtonya Can you please schedule the split of this new epic card? It's going to be required to schedule the sub-tasks development. Thanks in advance :)

[hellofromtonya]

@GeekPress The engineering team and I discussed this one today. We'll work on this one on the side. Then next Monday - Grooming Day - we will add the details and effort for the epic and subtasks.

[GeekPress]

@hellofromtonya I'm adding the grooming tag as it need to be split in sub-tasks.

[hellofromtonya]

@GeekPress In discussions with the team, we believe the best long term strategy for this epic is to leverage our custom db table and handler strategy. This new module will be leveraged not only for CPCSS but also for Preload and other caching processes within Rocket.

Why this approach?

👉 Query Performance.

Querying in the post meta table is very inefficient. We can significantly speed up processing and optimize queries.

👉 Mapping Performance.

We can do the pre-work for mapping the relationship between URL and filesystem/cache pathing.

👉 Regeneration Performance.

We can set "freshness" indicator and heuristics to let us know if/when we should regenerate. This makes Rocket more deterministic and deliberate in how and when it does its work.

[hellofromtonya]

Update:

This epic is dependent upon the new Database Module we will create as part of the new Preload Module. This week, @crystinutzaa is doing a prototype of the Database Module and then grooming the Preload. Once done, then we can groom this epic too.

[crystinutzaa]

This is dependant on module-db to be finished before starting on it.

Database module

• Use the prototype created in https://github.com/wp-media/module-db and add a new table creation (wpr_critical_path).

  [S] Schema - use Innodb tables at schema creation

  • Id - unique identifier
  • Post_id -
  • Cpcss_file - (with path included)
  • Is_mobile - bool - to indicate if the CPCSS is for mobiles
  • Date_upd - date updated field to track the modifications to the specific CPCSS

/inc/Engine/CriticalPath/Database which will handle the database processing

- [S] Controller.php
        save( $post_id, is_mobile) - adds / updates a Specific CPCSS field
        delete( post_id, is_mobile ) - deletes the specific CPCSS for a post id
        get( $post_id, is_mobile ) - gets the specific CPCSS for a post id
        get_all_specific_cpccs( $p, $n ) - gets all posts_id for which the CPCSS needs to be regenerated
- [S] Model.php
        Add function `exists()` to check in the database if the CPCSS exists for the specified post id
- [S] - ServiceProvider.php
        Provides the CPCSS DB Controller and the CPCSS  DB Subscriber

- [M] Subscriber.php - 
        Generate - Hook in RestWP generate() method and save the data in the DB if the CPCSS generation succeeded
        Delete - Hook in RestWP delete() method and delete the data in the DB in the CPCSS deletion succeeded
        Hook into `rocket_cpcss_items`  to add to transient rocket_cpcss_generation_pending all the data from the DB table formatted to match the current required data type

[jazir5]

Why do things just get caught in limbo here? This was marked "grooming in progress" on jul 23, that was 7 months ago.

[GeekPress]

@jazir5 As you can notice we have today 445 opened issues and we have to prioritize. As the feature is working properly and it's not a blocker, we don't plan to work on it until to have a lot of time.

[camilamadronero]

Customer Interim Suggestion:

Availability to rebuild page specific CPCSS as a product bulk action, here: https://i.imgur.com/LmZkTsa.png

Related ticket: https://secure.helpscout.net/conversation/1428382126/240182?folderId=2675957