Woocommerce: Geolocation based prices and Caching

[lterys]

Hello, Apparently some issolated product pages are displaying the product price before tax, at least in Google Chrome. Had to reload several times for the price to update.

Anyone has some insight in what could be happening? A conflict with W3 Total Cache maybe? Then why its only in some product from time to time, apparently at random?

Thank you

[mikejolley]

I imagine caching could be an issue here. If its cached for a certain customer in location X, this may be cached and be shown to a customer in location Y.

You may need an exclusion rule.

I don't think we can use ajax here like we do with the cart widget because it would require a request per product, so the only other option would be to inc/dec prices with javascript.

@claudiosmweb @barrykooij Ideas?

[claudiosanches]

It seems to be related to cache. Maybe some rule to never clear the cache.

[lterys]

It was probably a cache issue. Disabled it a couple of days and I couldn't reproduce the problem again, so yeah it's nothing caused by Woocommerce. Thank you..!

[daigo75]

If I may chime in, I've been working with _per country_ and _per user_ and _per currency_ pricing for two years now (you might have come across my multi-currency/prices/tax plugins a couple of times) and I can confirm what @mikejolley wrote: caching is the root cause of the issue. The reason is that caching systems (plugins and services) tend to use only the page URL as the "key" for each cached page. That is, they store one copy of _http://mysite.com/my-product_, and serve the cached copy to everyone.

Since the caching system works by not loading WordPress plugins at all, no geolocation resolution is performed (WooCommerce is not even started), and the static cached copy is served until it expires. As a result, everybody will see the same data, regardless of their location.

The solution is to add an exception to the caching system, so that the catalogue and the product pages are not cached.

[milque]

I can vouch for the quality of @daigo75 his plugin. We've been using it for about 6 months now and haven't looked back. Top notch.

[mikejolley]

@claudiosmweb @justinshreve @davidlenehan Any further ideas here? #wontfix or maybe look into appending something to the query string to prevent caching different geolocations?

[davidlenehan]

Appending something to the query string seems like it would work, if the query string had something like my-product.php?currency=usd&country=us. I assume you would not want to change the prices shown if the user was to manually edit this to something different? I guess these params could just be ignored in code.

In the past some caching systems would not cache anything with a querystring so the better solution was: example.com/us/usd/my-product.php

Adding unnecessary params in the URL is kind of ugly though. I wonder would some kind of hash just look better: my-product.php?v=1234 where v is uniquely generated based on the currency/country_code combination, but really means nothing to the user.

I assume this is also a problem for URLs like `example.com/cart/? do they mix up people shopping carts?

[mikejolley]

@davidlenehan We always recommend excluding cart/checkout from cache plugins as they always need to be dynamic.

For the catalog as a whole though, advising turning off cache everywhere is kind of bad :)

[davidlenehan]

Make sense :)

[justinshreve]

"I don't think we can use ajax here like we do with the cart widget because it would require a request per product"

Could we add a new AJAX batch method that allows us to pass in many product IDs and get a list of products back?

[mikejolley]

@justinshreve Do you think that would be better performance wise vs ajax geolocation and redirect?

[justinshreve]

I can't say off hand but I don't think it would be a huge performance burden to do it - though obviously it does require an additional HTTP request :). It does keep URLs cleaner though, so I wanted to throw it out there.

[mikejolley]

@justinshreve It could be part of our cart fragments request. But I imagine it would be very difficult to reliably target all products displayed on the entire page.

[daigo75]

Yet fairly necessary, as caching logic is flawed. The idea behind caching is "don't bother rendering that page, I know what the content is". With geolocation, you DON'T know what the content is, as the page content changes dynamically.

The assumption behind all caching systems, i.e. one URL = one content, falls apart miserably. I've added Geolocation features to WooCommerce in 2013, I've been through it dozens of times.

On Tue, Jun 16, 2015 at 8:10 AM -0700, "Mike Jolley" [email protected] wrote:
@davidlenehan We always recommend excluding cart/checkout from cache plugins as they always need to be dynamic.

For the catalog as a whole though, advising turning off cache everywhere is kind of bad :)

---

Reply to this email directly or view it on GitHub:
https://github.com/woothemes/woocommerce/issues/7939#issuecomment-112465139

[mikejolley]

Little experiment branch here https://github.com/woothemes/woocommerce/tree/geolocate-cache

This will;

• add a 'geolocate via ajax' default customer location option (not fully hooked in yet, but in place)
• geolocate the user on visit via ajax and set the value in session storage
• if outside base, redirect to same URL with location=XXX appended to the query string
• use default location for taxes, unless on cart/checkout, or the 'location' query string is present

This essentially makes URLs without 'location' show base taxes + prices, and URLs with location show correct taxes for visitors. Tested with WP Super Cache which cached the location vs regular endpoints independently.

[justinshreve]

The branch/approach makes sense to me.

I did a quick code review (I realize this is an experiment branch but wanted to flag them):

I noticed both the if and else if conditions are the same: https://github.com/woothemes/woocommerce/compare/geolocate-cache#diff-76251725cd8bd5faac3338eb4906b6b2R78

JS coding standards - strict equality checks: https://github.com/woothemes/woocommerce/compare/geolocate-cache#diff-76251725cd8bd5faac3338eb4906b6b2R46 or https://github.com/woothemes/woocommerce/compare/geolocate-cache#diff-76251725cd8bd5faac3338eb4906b6b2R63 for example

JS coding standards - spacing: https://github.com/woothemes/woocommerce/compare/geolocate-cache#diff-76251725cd8bd5faac3338eb4906b6b2R87 there should be a space before added_to_cart but not after the callback

PHP coding standards - yoda: https://github.com/woothemes/woocommerce/compare/geolocate-cache#diff-9ed886896d7600b13d6a12a7d0ccefc7R316 'base' can be switched to a yoda condition

[mikejolley]

@justinshreve patched those

[justinshreve]

:+1:
Also, gave the branch a test spin and things worked as expected.

[barrykooij]

Tested and working great here :+1:

[shawaj]

Is there a known issue with WooCommerce GeoLocation and CloudProxy (https://sucuri.net/website-firewall/) ?

We seem to be running into an issue with this caching whereby every single person sees the same query string at the end of their domain name, and the prices are not geolocated properly.

CloudProxy routes all data through Sucuri's network before hitting our webserver, and then re-writes the IP address to be that of the end user. Is this what would be causing an issue?

Thanks

[daigo75]

@shawaj that may be the case. I encountered similar issue with my own geolocation mechanism some years ago (which is not the one from WC core). If the proxy rewrite passes Sucuri's server(s) IP address to WooCommerce, for any reason, then the visitor will appear as being the same all the time.

It all depends on how customer's IP address is passed through, unfortunately the headers are not very standardised. This article on Sucuri might help configuring the proxy to prevent the issue: Same IP for All Users.

[shawaj]

@daigo75 - Currently it is using HTTP_X_FORWARDED_FOR which I have set through the Sucuri Plugin which it recommends using from the link you posted.

In the Sucuri plugin backend, it correctly resolves my client IP.

However I am not sure if WooCommerce geolocation is pulling in the same IP? And it would appear not as I am getting prices including VAT regardless of where I access the shop from.

Is there a better header to use in order to get CloudProxy to play nicely with WooCommerce?

[daigo75]

As far as I know, WooCommerce uses HTTP_X_FORWARDED_FOR if X-Real-IP is empty (see WC_Geolocation::get_ip_address()), therefore the IP address it finds should be the correct one. Perhaps someone from the WC team can shed some light on this (as I mentioned, I don't use the WC geolocation myself, having implemented my own logic years ago). :smile:

[shawaj]

@daigo75 - Would X-Real-IP show the forwarding server's address (i.e. Sucuri) instead of the original client's IP in a standard configuration?

And is X-Real-IP different to HTTP_X_REAL_IP?

[shawaj]

Just in case this issue occurs to anyone else:

Looks like I had a combination issue - it was a misconfiguration in CloudProxy AND corrupted GeoIP.dat and GeoIPv6.dat making it difficult to find the issue!

I now have everything working with CloudProxy / Sucuri WAF using the Sucuri Wordpress Plugin with Reverse Proxy Support enabled, IP Address Discover enabled and Main IP HTTP Header set to HTTP_X_REAL_IP

GeoIP.dat and GeoIPv6.dat are located in wp-content/uploads folder and I just deleted and then force-downloaded them again.

[shawaj]

Actually it's now apparently still not working properly, according to sucuri, due to the 307 redirect caused by the geo location.

Apparently a 307 redirect specifically causes no caching?

[worldsdream]

It's good that you fixed the issue with caching, but the url looks not user friendly. People that visit a page with such url think that something is misconfigured or don't trust it completely. Isn't there a way for you to fix that in feature version? So we only see mydomain.com/product/myproduct instead of mydomain.com/product/myproduct-298102892 ??

[mikejolley]

If you want geolocation, you just need to accept a URL needs to be unique.

[tarranjones]

@worldsdream @mikejolley A possible clean URL solution would be to set v as a session cookie and have the cache take this into consideration when caching and fetching pages.

[daigo75]

@dtjones that is precisely what we did with all our plugins. Instead of relying on unique URLs, we rely in unique cache keys. That way, there is no need for a redirect, nor JavaScript or Ajax calls. The caching system simply creates and retrieves a copy of each page depending on each combination of the criteria (in our case, currency, country and tax settings).

It works perfectly, as long as the caching system supports dynamic caching. Unfortunately, some of them do not, while others, such as CloudFlare, require a paid plan (quite expensive too). With these system, the solution with the cookies doesn't work, the only way to serve the correct content is to use a unique URL.
Still, it was an acceptable solution for us to simple flag as "unsupported" the caching systems that don't allow dynamic caching. Our opinion is that the caching systems should be configured to serve the site (i.e. adapt to the site's needs), not the other way round.

[worldsdream]

The caching systems should adapt to the site, not the other way around. The cache plugins should update to support dynamic caching. I do know that wp super cache has it, zencache has it, wp rocket has it in beta version right now and will be released next stable version. W3tc does not have dynamic. Woocommerce could give us an option to enable it if we have plugins cache that accept dynamic caching till all caching plugins support dynamic caching.

[daigo75]

@worldsdream that's exactly my position. Our solution supports ZenCache/Comet Cache, Super Cache, WP Rocket and Nginx. W3TC is too rigid right now, not supporting dynamic caching, but it should be easy enough todd support for it, of the author(s) want.

[raamdev]

I'm one of the developers for Comet Cache (previously ZenCache / Quick Cache) and I just reviewed the discussion here regarding caching and query strings. Note that Comet Cache disables caching for URLs with query strings by default, because the assumption is that if a page URL contains a query string, the content is likely intended to be dynamic in a way that might not be compatible with caching.

By default, both Comet Cache and W3TC disable caching for URLs with a query string (aka GET Requests):

WP Super Cache, however, enables caching of query strings by default (which I would argue is a bad default as it makes it easier to unintentionally cache something that is not cache-compatible):

In any case, plugins that _do_ append a query string and which are designed to have those query strings cached should update any documentation to suggest making sure that caching for query strings (aka GET Requests) is enabled in whatever caching plugin that site owner is using.

WP Rewrite API: The proper way to support caching plugins

Since caching plugins (should) assume that URLs with query strings are _not_ cache-compatible (as the presence of a query string usually indicates something dynamic should occur on a page), the _best_ way for plugin developers to add support for caching plugins is to make use of the WP Rewrite API to make WordPress aware of custom query string variables using functions like add_rewrite_tag (which is often used in conjunction with add_rewrite_rule()).

Using the WP Rewrite API, a geo location plugin for example might append /example-page/geo-north-america/, or a currency plugin might append /example-page/usd/. The plugin author would then use the WP Rewrite API to look for the presence of appropriate variable when the page is loaded.

Since all WordPress caching plugins use the page URL itself as a way of identifying the cache file for a specific page, pages that use the WP Rewrite API to make WordPress aware of custom query string variables would automatically be fully cache-compatible. For example, caching plugins would simply create a cache file for example-page/usd/ or /example-page/geo-north-america/.

[daigo75]

@raamdev It's an interesting approach, but I still think that it should be up to the caching plugins to be more flexible. Comet Cache, Super Cache, WP Rocket, Nginx and, I'm sure, other solutions can support dynamic caching as they should, therefore other plugins should follow suit. Sticking to the idea that a caching plugin only uses the URL as a cache key seems backwards, like the "_tail wagging the dog_".

[raamdev]

@daigo75 One of the most important jobs for a caching plugin (especially a WordPress caching plugin, where content is often very dynamic) is to make sure that an out-of-date cache file _never_ be shown to a visitor. If the content of a page changes, the caching plugin needs to make sure that the old cache file is deleted and a new cache of the page is generated.

For that reason, caching plugins should always err on the side of caution and not cache a page that _might_ not be cache-compatible. GET Requests are the most common way to dynamically modify the content of a page, to pass in some unique variable that then modifies the output of the page in some way. For that reason, caching plugins should _not_ cache pages with query strings by default. It's as simple as that. That doesn't mean _all_ query strings are not cache-compatible, but the point is that a caching plugin has no way of determining which query strings can be cached and which cannot, so it does what it should do: err on the side of caution and not cache the page.

As I've already noted, Comet Cache, W3TC, and WP Super Cache are already very flexible in this regard: they all give you the option to cache URLs with query strings. In the case of Comet Cache and W3TC, the option is to leave that disabled by default. If a site owner wants to enable that option, that's up to them.

[daigo75]

@raamdev I agree, caching plugins should adopt a careful logic. My point was simply that any issue caused by how caching works ("risky" defaults, rigid logic, etc) should be fixed by correcting the caching logic, rather than by using workarounds to "trick" the caching plugins into serving different content. That's the reason why I like Comet Cache: it's straightforward, it has sensible defaults and it's very flexible. :+1:

[raamdev]

@daigo75 writes...

rather than by using workarounds to "trick" the caching plugins into serving different content.

I don't believe using the WP Rewrite API is "tricking the caching plugins". It's simply using an existing WordPress feature that was specifically designed for the problem we're discussing: passing dynamic query string variables in a way that is WordPress-specific (i.e., in a way that other WordPress plugins, such as caching plugins, can make themselves compatible with).

That's the reason why I like Comet Cache: it's straightforward, it has sensible defaults and it's very flexible.

Thank you. :-)

[Dev49net]

Thank you for your input @raamdev and @daigo75. As a developer of the multi currency plugin myself, I am probably going to be on the same page as @daigo75.

Having currency as a part of URL is a no-go. Many users of my plugin specifically do not want their users to be aware that currency has been selected for them. Many of them don't use currency switcher at all - they just use geolocation and present the customer with a store in currency related to their location - the user is not even aware that this is a multi-currency store and the base currency is different than the one they are browsing in. Therefore, changing the URL to example-product/usd is not an option. What I do in my plugin is basically the same as WooCommerce Ajax-based geolocation. I append ?c=XXXXXX to the end of URL, which, as we all know, is far from perfect - not only because it looks bad, but because it needs additional cache plugin configuration, as @raamdev has mentioned. The c parameter in my case is a hash of various parameters related to currency, so that customer has no idea this is related to currency, but it is and it differs for every currency.

I am looking into the dynamic cache subject you guys have discussed earlier, but since W3TC does not support it, the GET parameter is still necessary, at least as a fallback. W3TC is very popular and most common among my clients and it cannot be ignored.

The inability to execute any PHP action before page is displayed, makes our hands almost completely tied. If only it was possible to have one hook right before the cached HTML is loaded, it would be perfect. But since all this is happening directly in .htaccess, things are quite complicated - especially with so many cache plugins out there.

Cheers,
Dan

[daigo75]

@raamdev That boils down to the same thing: it's a way to create "virtual" URLs that look different, so that the cache key will be different as well. I think it makes more sense to allow adding extra data to the cache key instead. Having dealt with over four thousand multi-everything sites, when developing my products, I came to that conclusion.

Although, as @Dev49net pointed out, W3TC is a popular caching solution, at the moment it's simply too rigid. Passing a GET argument does indeed allow the page to show the correct data, but it bypasses the cache altogether. For this reason, I often help my customers moving to a different solution (i.e. I'm not ignoring W3TC, I'm replacing it), and, so far, they are all happy with the results. :smile:

[raamdev]

@daigo75 @Dev49net Thank you both for your input here. :smile: A big part of building WordPress plugins is making sure that they play nicely with other plugins. Caching plugins play a unique role in that they are generally loaded before any other WordPress plugin.

Comet Cache makes it possible for other plugins (and themes) to 'hook into' the caching process early on using a Dynamic Version Salt, however I'm wondering if an easy way to improve support for plugins that use cache-compatible query strings would be to maintain a compatibility list inside the cache plugin, so that for example if @Dev49net's plugin was active Comet Cache could automatically detect that and allow caching for URLs that contained only the ?c=XXXXXX query string.

Anyway, I think this is getting to be off-topic for this specific GitHub issue so I'm going to fork this discussion over to https://github.com/websharks/comet-cache/issues/707.

[shawaj]

@worldsdream @mikejolley @dtjones +1 on the suggestion of having clean URLs for caching. Perhaps a fourth setting could be added to the list "Geo location with clean URL caching support" or something. The redirect for geolocation is currently showing down our site significantly and the URL looks ugly

[monecchi]

My website's server uses nginx with fastcgi-cache enabled. I've noticed that the ordinary Geolocation option was not working, so I updated WooCommerce to get the new enhancements of the option Geolocation with cache support, but it seems to add an ugly ?v= parameter to all url requests.

Is this the final solution for the issue or is it just a workaround?

How am I supposed to deal with the ?v= url parameter when it comes to SEO? Wouldn't search engines index all those dynamic ?v= url parameters from my website's pages?

[daigo75]

@monecchi Ciao Adriano. :smile:. That's a workaround to "trick" caching plugins and systems into serving the correct content to the users. As discussed in detail in this thread, caching system are fundamentally flawed, out of the box. They assume that one URL = one content, when that is might not necessarily the case (geolocation-specific content is a typical example). Based on this assumption, WooCommerce does what it can: it creates unique URLs, using the _?v=234567890_ argument, so that the caching systems can let WC do the geolocation and then serve the correct content.

Technically, it's the caching system that should be able to serve different content from a single URL, and this is the solution I've been advocating for two years now. The plugins I wrote rely on such assumption, and I wrote an article that explains how a caching system, like Nginx, can be configured to support them.
The article doesn't cover the geolocation included in WooCommerce, because my solutions use their own geolocation logic (I wrote it over a year earlier than the WC one, and I decided to keep it), but, if you are familiar with Nginx, it can be useful to put you on the right track to configure it.

[galapogos01]

Any progress in this area? The query parameter workaround is not great as it defeats the purpose of a cache and will cause sites with a large number of products to require much more server resources.

[daigo75]

@galapogos01 Have you had a look at the article I posted earlier? The solutions explained in it, which don't use a query parameter, are generic. They can be adapted to a "plain" WooCommerce system too.

[galapogos01]

Are you suggesting the core fix for this issue is to use third party software downloaded from an untrusted source? No thanks.

I am advocating for a fix in core that allows WooCommerce to function correctly with a caching plugin, which is basically a mandatory requirement to run a customer facing eCommerce site using WooCommerce.

[daigo75]

@galapogos01 I wasn't suggesting to download anything. I simply suggested to look at some ways to solve the issue caused by caching, which is what the article (which I wrote, and I'm definitely not an "untrusted source") is about. :smile:

From my perspective, it's the caching that has to adapt to the site, not the way round. There are several ways to do so, depending on the specific setup.

Still, if you would rather wait until a solution is added to WooCommerce core, that's fine too.

[galapogos01]

The article you posted includes links to download enhancement plugins for various caching engines. The plugins are attachments to the freshdesk article and not code that is managed via the Wordpress Plugin system. They are also not relevant to how core does geo and require your plugin to be installed for them to work.

Please keep the bug on topic and stop trying to promote your plugins.

[daigo75]

I wasn't trying to promote my plugins. I was trying to show an alternative solution to the caching issue, which is something I've been working on for quite a while. Anyway, that's clearly not what you're looking for, so I will leave you to it.