Magento2: Also Varnish Cache invalidates most of the pages when saving one product

[andidhouse]

Hi there,

first we tried the magento2 internal FPC and realized that on one event (a backend product save or an incoming order) clears the complete FPC cache. The feedback on this from magento side is that it is normal - ok for us as Magento recommends Varnish.

Now we set up varnish according to the magento varnish guide and filled up the cache completely by crawling the site - so fine so good. Very fast.

Then we saved just one product which is online in one category. And now ca. half of the site is uncached again?

I dont´t know if this is normal but i do not think so. We now from other ecommerce systems that there is an option on which event which parts of the site are flushed. We are lacking a documentation here at all for magento.

For us (and i think most shops) the following is interesting:

• what exactly is uncached on which event with the standard magento varnish implementation
• how can we achieve to change this for:
  -- just clear the product cache of the product effected (backend save or incoming order with this article)
  -- clear the product cache of the effected product and the category cache(s) belonging to the product
  -- just clear any cache if the "availability" of the product changes (not the stock)

Is there a documentation somewhere where we can find these points?

Many thanks!

[moloughlin]

I've been looking into this as well. When saving a model I expect only pages with the corresponding instance to be invalidated however I found that pages with any instance have been invalidated.

Traced this back to what appears to be a problem in the InvalidateVarnishObserver (module-cache-invalidate):

    public function execute(\Magento\Framework\Event\Observer $observer)
    {
        if ($this->config->getType() == \Magento\PageCache\Model\Config::VARNISH && $this->config->isEnabled()) {
            $object = $observer->getEvent()->getObject();
            if ($object instanceof \Magento\Framework\DataObject\IdentityInterface) {
                $tags = [];
                $pattern = "((^|,)%s(,|$))";
                foreach ($object->getIdentities() as $tag) {
                    $tags[] = sprintf($pattern, preg_replace("~_\\d+$~", '', $tag));
                    $tags[] = sprintf($pattern, $tag);
                }
                $this->purgeCache->sendPurgeRequest(implode('|', array_unique($tags)));
            }
        }
    }

In the case of product saves, this will trigger a purge request for any object with X-Magento-Tag matching ((^|,)catalog_product(,|$))|((^|,)catalog_product_<id>(,|$)).

Note, the following line is causing the purge to target all catalog_product objects:
$tags[] = sprintf($pattern, preg_replace("~_\\d+$~", '', $tag));

I'm not sure what the rationale is for this, I've tested without the above and all other product pages will remain cached as expected. I'm eager to hear any reasoning for sticking with the default behavior, but on a site with frequent content changes this has a big performance impact.

[moloughlin]

Quick update: Just noticed the following commit which removes the line that was causing the problem: https://github.com/magento/magento2/commit/2754427ac2fa9d025272b5502b55a271f7e415b4

An update to 2.1 should resolve this issue.

[choukalos]

This is addressed in todays 2.1.1 patch.

On a product update (any field) the cache should be cleared for the product page and any associated category pages.

[rbostan]

Same error with 2.1.4 & Redis. Saving a single product invalidates also 'catalog_product' and 'FPC', not just product related pages or specific 'catalog_product_'.

For example, a product saved which is in 'Category 1', it means cache will clear product view page and 'Category 1' listing, search result etc. But it clears not related categories and cms pages.

[drk1]

Hi, any news regarding the comment from @choukalos in Aug 30, 2016?
(I'm using Turpentine 0.7.10 on Mage CE)

[alakhdeveloper]

When we save any of the product the FPC cache get invalid. That affect the website speed. That is the big one problem in magento2

[medspec]

I still have this issue on 2.2.5. can we reopen this?

[nicknailers]

same thing here with 2.2.5 seems to also happen after a upgrade to 2.2.6 CE using varnish 5 however was doing it under varnish 4 too. I did double check to make sure the line from the patch posted by @moloughlin wasn't in those version. Had to disable the "flush cache" method temporarily or we are losing the FPC cache on our whole live website which is bad with more than 100,000 products.

[snoroozi]

I'm also facing this on 2.2.5 - does anyone have a solution. It's maddening.

[snoroozi]

I wonder if this has anything to do with having root category set to anchor, in that it ends up clearing all?

[snoroozi]

I see in our case it is happening right after price index finishes running after a product save.

[oefterdal]

We have the same issue in 2.3.1

[agata-maksymiuk]

There is a cron job called every minute.
<job name="indexer_update_all_views" instance="Magento\Indexer\Cron\UpdateMview" method="execute"><schedule>* * * * *</schedule>

public function afterUpdateMview(\Magento\Indexer\Model\Processor $subject)
this method dispatches event clean_cache_after_reindex

and there is observer that cleans Varnish cache for all categories and products.
<event name="clean_cache_after_reindex"> <observer name="flush_varnish_pagecache" instance="Magento\CacheInvalidate\Observer\InvalidateVarnishObserver"/> </event>

I disabled Magento cron and it stopped to clear Varnish cache every minute.
I also put a log in \Magento\CacheInvalidate\Observer\InvalidateVarnishObserver::execute method.

[snoroozi]

@agata-maksymiuk I imagine the side effect is that you will not be showing up to date information, such as out of stock items still displaying in stock.

[agata-maksymiuk]

Yep this is not a solution but an example.

[dpdesignthatfits]

Yep this is not a solution but an example.

Were you able to find a proper solution to this?

[nicknailsit]

had to disable the cronjobs too and make a full cache warm at 2am can't believe this is still a problem more than a year later. Are we the only one who consider this a serious bug? Is magento made for large store or is it becoming woocommerce (nothing against it but had problems with large amount of products too on that platform).

[GuiltyNL]

I wonder if this has anything to do with having root category set to anchor, in that it ends up clearing all?

@snoroozi I'm testing this at the moment and I have disabled the anchor of the root category. For us it didn't solve this issue.

For us it seems to have to do something with (custom) attributes that are set / changed on save. It clears the whole cache. Futhermore our cache warmer does not recognize it as a complete flush and only adds that product and the category pages of that product to warm up. Leaving us behind with and empty cache. Until we manually do a magento cache:flush.

We have the same behaviour while saving a category, also that causes to flush the complete cache.

Both when the indexer_update_all_views cron kicks in.

[GuiltyNL]

@choukalos I'm wondering why this ticket has been closed?

[GuiltyNL]

The issue is related to this one:

https://github.com/magento/magento2/issues/5561

I've been busy trying to debug this and when a category is added to the Top Menu (The option in categories 'Include in menu') it will be added to the X-Magento-Tags of every product.

When Magento asks for a purge and a category is involved, Varnish will flush all the products, because every product has all the category tags of the categories that are included in the top menu. And that causes the purge of the remaining cache of all categories. Leaving you with a complete empty cache.

Another issue is emphasizing this. When changing a product attribute of product, Magento clears the correspondeing categories of that product too. And because of the Top Menu caching issue, something simple like changing the Color from Blue to Green of a product will result in a complete purge of the cache.

[gibbotronic]

Why is this bug closed???? There is no mention as to why it was closed??

This still seems to be an issue - see https://github.com/magento/magento2/issues/25670 that we are facing.

[Spideramn]

Is it possible that these lines are causing the issue?
https://github.com/magento/magento2/blob/f9a88849152344df337d5cc24e0b6b3cef228a82/app/code/Magento/Catalog/Model/Product.php#L2365-L2367

It seems like when you save a product from within the front-end area the complete product cache is invalidated? Or is this a completely different issue?

When I save a product while emulating a different areaCode I see a different purge request in my varnishlog:

Request 99416 is a "normal" product save. Request 99418 is the same save, but now using the emulateAreaCode method, emulating the "global" area. As you can see, the "cat_p" tag is only present when saving from the front-end area.

I'm not sure what the purpose is of clearing all varnish product caches when saving a single product from within the front-end area.

[GuiltyNL]

Why is this bug closed???? There is no mention as to why it was closed??

This still seems to be an issue - see #25670 that we are facing.

The whole FPC is a joke in Magento. We are having FPC issues since 2017 and we have spent a lot of hours of our own and paid hours to solve the issues with limited success.

The cache is being flush the whole time our cache warmer is not able to keep up.

[sinepen]

I've been having the same problem with varnish cache being invalidated or purged on every small product update, which has been really annoying lately.
I recently figured out that it helps a lot to set index management to "update on schedule" instead of "update on save" though.

I'm not into the technical part of M2, but if the index is being refreshed on product save and the event "clean_cache_after_reindex" is triggered every time the product index is reindexed, then I guess this kind of makes sense (?).