Magento2: Varnish never invalidates static assets unless restarted

[pantaoran]

Preconditions

1. Magento 2.1.2 CE
2. PHP-fpm 7.0.14
3. Varnish 4.0.2
4. Debian Jessie docker containers running on Ubuntu 14.04.5 host

Steps to reproduce

1. Configure Varnish according to the documentation and make sure that the cache is hit (use developer mode)
2. Make sure that flushing all caches correctly sends the wildcard purge request to Varnish, instructing Varnish to purge all cache tags
3. Load a cached page with an image from the media folder (pub/media) in the browser
4. Replace the image in the media folder by a different one on the server filesystem
5. Flush all caches (either in admin or on terminal)
6. Load the same page again in the browser

Expected result

1. The page should not be cached and the replaced image should be loaded

Actual result

1. The html page was correctly invalidated and rebuilt by PHP but the old image is still served from the Varnish cache. Varnish did not invalidate static assets when caches were flushed (I verified that the webserver backend never got a request for it, so I conclude that Varnish must be serving it from cache). In fact, the only way that I found to invalidate the cached image was to manually restart Varnish.

Besides the actual issue there is also an architecture question that I want to raise here. I'm by no means a Varnish expert, but I have read the following short blog article that makes a lot of sense to me: https://ma.ttias.be/stop-caching-static-files-in-varnish/

He argues that static assets like JS/CSS/Images should not be cached by Varnish since they only consume memory but do not really cause load on the webserver if not cached. Also, they give wrong ideas about the cache hit rate in the Varnish statistics.

Is there a good reason for why the exported Magento2 varnish config makes it cache such static assets? If not, I want to suggest changing that.

[craigcarnell]

This could be a bug that I am experiencing with Varnish and M2

[giacmir]

Same here with Magento 2.1.2 EE

[PieterCappelle]

Can confirm this in 2.1.3. Have same issues. Always need to purge the Varnish cache from the command line. Very irritating.

[giacmir]

As a temporary workaround you can find in the vcl file the line where it caches the static assets and change with return(pass);. The static assets are now served directly by the webserver instead of varnish.

[PieterCappelle]

@giacmir Don't think that is a good idea :-)

[craigcarnell]

I'd rather have the bug fixed, but I appreciate any help with a workaround for now

[pantaoran]

This workaround goes in the direction of my last question about why static assets are cached in the first place.

@PieterCappelle what makes you say that that wouldn't be a good idea?

[craigcarnell]

Any updates on this Magento? This is a major bug for any production site using Varnish

[kokoc]

Magento tries to propagate static resource as far as possible in web. Built-in php adapter and installations without Magento_PageCache module store static resouces in browser cache. Thus described case is a bug and Varnish shouldn't handle static content (content must go from webserver to browser directly and cached there). However, this doesn't solve your issue and even make it worse.

Usually, such cases are solved by content versioning. Magento is not an exception. We adding suffixes to filename in case when file with the same name already exists. It allows to deliver fresh files to the end user. For your case, I would recommend to change the picture name and notify productRepository with new picture's path.

We will put appropriate tasks in backlog in order to remove inconsistencies between varnish and php page cache

[naydav]

Internal ticket MAGETWO-33342 is in our issue backlog.
Thanks

[mbahar]

Try to add ;

varnishadm "ban req.url ~ "/" "

to your cache:clean task, to manually purge the static file cache.

works for me.

[alinalexandru]

If you are using a public CDN service, you will have the same problem. The solution is to change the change the image with one with different filename.

[andimov]

issue MAGETWO-33342 was fixed and delivered to mainline.
Thanks!