Ublock: bad Fanboy server cache policy (NOT a uBO or browser bug)

[joey04]

For at least the past month my one 3p filter list, Fanboy Ultimate, has not been updating properly in Pale Moon. It's held steady at the same used filter count even though it states it's been auto-updating. And when I force an update it's no different. I've also started fresh with a new .sqlite file and importing my settings, but it's still the same behavior. And viewing the list within uBO shows the list is not changing. (Most importantly, uBO still functions fine in every other respect, as the unmodified list rules still block/allow properly along with my custom dynamics and statics.)

Also importantly, I've not seen any problems on Firefox or GChrome using the same exact settings. Fanboy Ultimate auto-updates properly on both of those.

I'm currently on the latest uBO 1.12.3rc0, but it's the same problem as 1.10.0 (which I used until yesterday). I'm also on latest Pale Moon 27.3

I've attached screenshots of the Logger during a forced update today and the resulting list view of timestamp from over a month ago. Note that the Logger is practically identical to what I saw in Firefox successfully forcing an update today. (The only difference being a few specific invalid rules, but the xhr to fanboy server is the same.)

I also attached my settings file, which can be imported to any of the 3 browsers. (Note I removed my custom dynamic and static rules but those shouldn't matter since I have same ones in all 3 browsers.)

uBO settings.txt

[gorhill]

Anything reported at the browser console?

[gorhill]

Just tried Palemoon 27.2.1 and uBO 1.12.3rc0 and all worked fine here. You will have to investigate on your side -- I can't investigate what I can't reproduce.

[joey04]

@gorhill thanks for trying out my settings and the suggestion of browser console.

I figured out the problem: the Fanboy server has a bad cache policy, telling the browser to cache the list until 2028!!! So, in theory, this problem could've persisted for 11 more years! Thus no uBO problems whatsoever.

So I removed my Pale Moon cache and did a forced update of all lists. It updated Fanboy Ultimate successfully with exactly the same filter count as Firefox. Here's the about:cache screenshot:

Pale Moon is my primary browser and the only one I preserve the cache. I've confirmed this is the exact same behavior in the Firefox cache as well, but it's cleared when I exit the browser so the list will update fine in future sessions.

For completeness, here's the detailed Fanboy Ultimate cache entry corresponding to the stale March 30 screenshots in my original post:

Note that Pale Moon v27 uses essentially the same cache scheme as the current Firefox v53, so these screenshots are exactly the same in Firefox about:cache.

[gorhill]

Thanks for investigating.

I was leaning toward a disk-write issue on your side preventing uBO from writing to storage the just-downloaded list, I did not think of cache policy (I know browser cache will prevent remote server abuse, but that is always a short term thing, like an hour, not days, months, let alone more than a decade in the current case).

Now here is what I see.

If subscribing to https://www.fanboy.co.nz/enhancedstats.txt (or any Fanboy lists hosted on fanboy.co.nz) with Adblock Plus, the server does not set an expiration date, meaning the list will be properly updated.

If subscribing to the same list with uBlock Origin, the server set an expiration date of more than a decade in the future, meaning that the list will never be updated for uBO users (unless the list is removed from their cache).

In short, this potentially makes uBO look bad because severely obsolete filter lists, while this potentially makes ABP looks good because the filter lists are properly kept to date.

[gorhill]

I use to add a random query parameter to the URLs of assets pulled from remote servers to be sure the cache was bypassed, but I removed this when I refactored asset management as I had observed that not doing so made uBO more considerate to remote servers -- it's ok to pull from browser cache when purging/update with less than an hour delay. But I never expected that sort of behavior from a remote server. This forces me to bring back that random query parameter, I don't see how else to prevent uBO from being at the mercy of remote servers with such caching policies.

[gorhill]

Fixed with 22d74421e37c6743702e783d81018450516091bc.

[stanlane]

https://github.com/gorhill/uBlock/issues/2623