Magento2: Product does not get updated after save in backend when called in fronted during the save

[andidhouse]

Preconditions (*)

1. Magento 2.4-develop

Steps to reproduce (*)

1. Go to backend product page
2. Save product
3. While the product is saving (2-4 Seconds load time) go to the frontend product url and call the product.

Expected result (*)

1. The product gets updated after save with the new information

Actual result (*)

1. The product has the old information not the new ones saved

---

It seems that when you call a product in the frontend while saving this product it is cached with the old information and not with the new updated infos.

[victor-v-rad]

it is cached with the old information
Yes and seems it is a normal behavior.

[robfico]

I'm seeing the same behavior. Magento 2.1.1 CE developer mode with varnish configured. After saving a product change like price, name, etc... The front-end page is not being updated unless we restart Varnish manually. My understanding is the front-end page should update automatically - from the dev docs:

http://devdocs.magento.com/guides/v2.0/config-guide/varnish/use-varnish-cache.html#use-varnish-magento-purge
"Maintaining a store (for example, adding or editing categories, prices, products, and promotional pricing rules).
Varnish is purged automatically when you perform any of these tasks."

[robfico]

As a follow-up, if I add this setting to the store:
php bin/magento setup:config:set --http-cache-hosts=127.0.0.1
then it seems to work as intended, where the cache gets cleared when a product is edited. Is the above setting required for Varnish to operate properly in Magento?

[magento-engcom-team]

@andidhouse, thank you for your report.
We've created internal ticket(s) MAGETWO-81759 to track progress on the issue.

[GuiltyNL]

@magento-engcom-team I've reproduced this also. Sometimes the cache is being purged succesfully after saving the product several times in a row.

And sometimes the purge randomly is succesful after a single save.

[GuiltyNL]

@robfico Are you sure that setting up the cache host was a permanent solution for the issue in your case? Because our cache host correct and cleaning the cache from the backend works well, but still we have issues with purging the cache after a product update.

[magento-engcom-team]

@andidhouse, thank you for your report.

Unfortunately, we are archiving this ticket now as it did not get much attention from both Magento Community and Core developers for an extended period. This is done in an effort to create a quality, community-driven backlog which will allow us to allocate the required attention more easily.

You may learn more about this initiative following this link.

Please feel free to comment or reopen the ticket if you think it should be reviewed once more. Thank you for collaboration.

[GuiltyNL]

'Funny' that this bug is being ignored. It seems that nobody has a clue, but while this bug persists, Varnish is unusable as it serves wrong and outdated information to visitors.

It is a bit silly to serve a PDP page with 'In Stock' while a product has been 'Sold out' in the mean while.

Or when a discount promotion ends, to serve a page to a customer where the discount promotion is still mentioned, while it has expired.

@andidhouse do you still have Varnish in front?

it is cached with the old information
Yes and seems it is a normal behavior.

Well it is abnormal behaviour, isn't it? Nobody wants to show old and outdated information to visitors.

[dverkade]

@magento-engcom-team I'm reopening this issue. The issue has been acknowledged. The fact that the core team or community has not picked it up yet is not a valid reason for closing this issue. It's still an issue which needs to be addressed. It needs to be open until is has been resolved.

[GuiltyNL]

@magento-engcom-team Why can Magento recommend using Varnish while it serves outdated data to visitors? Are there Magento shops running Varnish in front? Because it is unworkable?!

[GuiltyNL]

As a follow-up, if I add this setting to the store:
php bin/magento setup:config:set --http-cache-hosts=127.0.0.1
then it seems to work as intended, where the cache gets cleared when a product is edited. Is the above setting required for Varnish to operate properly in Magento?

Is someone able to answer this question, see above.

[snoroozi]

I'm chiming in on this. I see the same behavior.

[japalekhin]

I'm chiming in as well. Saving changes should clear the cache for that "object". I've also cleared the cache via SSH. I even tried reindexing and pesky HTML still persists.

[agata-maksymiuk]

The same issue here. Clearing all varnish cache both from command line and admin panel works fine. But it does not work for tags - single product or category.

I've found an error in var/log/system.log (why it's not exception.log? seems like "happy debugging suckers") A valid response status line was not found in the provided string

The error is thrown in
\Zend\Http\Response::parseStatusLine
containing this
$regex = '/^HTTP\/(?P<version>1\.[01]) (?P<status>\d{3})(?:[ ]+(?P<reason>.*))?$/';

As you can see there is hardcoded HTTP/1.0 or HTTP/1.1 protocol.
Check your HTTP protocol - we're using 2.0 now and that is was the cause of this error.

[agata-maksymiuk]

Also - if you use basic auth - make sure your Varnish IP is excluded from basic auth.

[IanFiretoys]

As a follow-up, if I add this setting to the store:
php bin/magento setup:config:set --http-cache-hosts=127.0.0.1
then it seems to work as intended, where the cache gets cleared when a product is edited. Is the above setting required for Varnish to operate properly in Magento?

heads up to anyone trying this, if you don't set up the correct http_X_Forwarded_for headers all of your orders will appear to be coming from 127.0.0.1 once this config has been set.

[stevetnt]

Chiming in here on 2.2.8. I have this process when I work called multi-tasking and being efficient. It doesn't really work well with Magento.

Is this ever going to be fixed? Dating back to 2016, it doesn't seem likely.

Has anyone found a solution to this?

[bkvargyas]

We are seeing similar issues with Magento 2.3.2, product pages do not update on save. We have to flush the magento cache to clear the varnish cache. Even though we do see a "purge" command in the log file of varnish during product save, it doesn't work.

[snoroozi]

@bkvargyas does flushing Magento cache from backend work, or do you have to purge varnish manually.

[bkvargyas]

@magento-engcom-team - We still have an issue with cache being flushed in Mag2.3.2. Product pages are not being updated on save in varnish cache, while debug logs show otherwise.

So, flushing magento cache does purge the varnish cache (full purge). -- ONLY after we added the following lines to our default.vcl code:

# If all Tags should be purged clear
        # ban everything to catch assets as well
        if (req.http.X-Magento-Tags-Pattern == ".*") {
          ban("req.url ~ .*");
        }

However, when we update a product page, and save it, we do see an update to varnish cache by showing the following debug info -- However, Product ID 6900 is not flushed

• << Request >> 1278167
• Begin req 1278166 rxreq
• Timestamp Start: 1567024228.300653 0.000000 0.000000
• Timestamp Req: 1567024228.300653 0.000000 0.000000
• VCL_use boot
• ReqStart 127.0.0.1 59220 a0
• ReqMethod PURGE
• ReqURL /
• ReqProtocol HTTP/1.1
• ReqHeader X-Magento-Tags-Pattern: ((^|,)cat_p_6900(,|$))
• ReqHeader Host: 127.0.0.1
• ReqHeader X-Forwarded-For: 127.0.0.1
• VCL_call RECV
• VCL_acl MATCH purge "127.0.0.1"
• VCL_return synth
• VCL_call HASH
• VCL_return lookup
• Timestamp Process: 1567024228.300684 0.000031 0.000031
• RespHeader Date: Wed, 28 Aug 2019 20:30:28 GMT
• RespHeader Server: Varnish
• RespHeader X-Varnish: 1278167
• RespProtocol HTTP/1.1
• RespStatus 200
• RespReason OK
• RespReason Purged
• VCL_call SYNTH
• RespHeader Content-Type: text/html; charset=utf-8
• RespHeader Retry-After: 5
• VCL_return deliver
• RespHeader Content-Length: 242
• Storage malloc Transient
• Filters
• RespHeader Accept-Ranges: bytes
• RespHeader Connection: keep-alive
• Timestamp Resp: 1567024228.300720 0.000067 0.000036
• ReqAcct 85 0 85 220 242 462
• End

Only until we do a full cache flush, does the product page actually update.

Just as a test, we tried the latest version of Varnish from V5 to V6, and still the same results.

This is affecting our ability to show accurate In/Out of stock on product pages, as well as other items that may have been updated. The indexes are set to update on save.

Beyond that, we're not sure where to go from here.

Brian

For Reference, Here is our Default.VCL file:

vcl 4.0;

import std;
# The minimal Varnish version is 4.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

acl purge {
    "127.0.0.1";
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
    # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
    # has been added to the response in your backend server config. This is used, for example, by the
    # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }

    # If all Tags should be purged clear
    # ban everything to catch assets as well
    if (req.http.X-Magento-Tags-Pattern == ".*") {
      ban("req.url ~ .*");
    }
    return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove Google gclid parameters to minimize the cache objects
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

    # static files are always cacheable. remove SSL flag and cookie
        if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
        unset req.http.Https;
        unset req.http.X-Forwarded-Proto;
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure HTTP users don't see SSL warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    /* {{ design_exceptions_code }} */
}

sub vcl_backend_response {
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # validate if we need to cache it and prevent from setting cookie
    # images, css and js are cacheable by default so we have to remove cookie also
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
        if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
            set beresp.http.Pragma = "no-cache";
            set beresp.http.Expires = "-1";
            set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
            set beresp.grace = 1m;
        }
    }

   # If page is not cacheable then bypass Varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
        beresp.http.Surrogate-control ~ "no-store" ||
        (!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

[bkvargyas]

So we've been researching this, and we believe it may be related to a module we're running called Unirgy Dropship, and the fact that it's not clearing the stock information in the cache when it's updated. I'll post more details as I find them. We have several developers looking at this to see if we can find the source of the issue.

[stevetnt]

We are not using that module and have the same exact issue.

On Fri, Sep 13, 2019 at 5:00 PM bkvargyas [e-mail] wrote:

So we've been researching this, and we believe it may be related to a
module we're running called Unirgy Dropship, and the fact that it's not
clearing the stock information in the cache when it's updated. I'll post
more details as I find them. We have several developers looking at this to
see if we can find the source of the issue.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/magento/magento2/issues/5980?email_source=notifications&email_token=AMTDWW643R4LBPOMMW7L7Y3QJP5NBA5CNFSM4CLR36H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6WFVJI#issuecomment-531389093,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMTDWW2D5IOCVV6JX7GELGDQJP5NBANCNFSM4CLR36HQ
.

[IanFiretoys]

we are also not using that module and have this issue as well.

[m2-assistant[bot]]

Hi @engcom-Charlie. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

• [ ] 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[stevetnt]

I would like to chime in here and mention that after we switched servers to AWS, and possibly due to Varnish cache setup -- our pages can be refreshed while product is saving without issues. Product always gets updated now and have not had this issue once.

My advice to a lot of people here would be to check and make sure Varnish and/or your caching setup is working properly.

Have a great day! Hope that helps!

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Charlie
Thank you for verifying the issue. Based on the provided information internal tickets MC-30047 were created

Issue Available: @engcom-Charlie, _You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself._

[adarshkhatri]

In My case, product data were saved as store 1 and store 0 in the database.
Basically data were saved in store 0 but store front was getting data from store 1.

Upon deleting store 1 related items helped. But I wonder, why some products have saved with store 1 data? I have got a single store only.

For ex:
catalog_product_entity_text table has 2 rows for a product for a specific attribute.

[engcom-Charlie]

Hi @andidhouse, I'm not able to reproduce on the latest 2.4-develop with varnish cache by following steps:

1. Update _name_ or _price_ or _description_ or _short description_ or _stock status_ on the backend.
2. Go to PDP.

AR: All changes applied successfully.