snip
Cookieless Cookie Fake Test: The test page mentioned by Pants uses the IP address for session tracking and the user agent but not ETags.
This test should use ETags, instead.
http://lucb1e.com/rp/cookielesscookies/
Results:
browser.cache.memory.enable TRUE & same IP - I was tracked.
browser.cache.memory.enable FALSE & same IP - I was not tracked.
browser.cache.memory.enable TRUE & different IP - I was not tracked.
This "test" could be fake! We need a more reliable ETag test.
Noscript. NoScript's ABE (Application Boundaries Enforcer) allows stripping etags.
Secret Agent generates spoof ETags headers with random values, preventing ETags being misused for tracking.
Run a local proxy as Privoxy that can block ETag headers.
Tor Browser 7.0.2 (vanilla)
TB with stored text > tracked by number of visits but not by stored text
TB with stored text + new identity > not tracked
Installed Modify Header Value (HTTP Headers) on FF ESR 52.0
In HTTP Headers options I created a new entry with:
URL *
Header Name If-None-Match
Header Value *
Then I picked Add.
The test passed.
I get 2 Visits repeatedly if I pick Remove instead of Add.
@Atavic This way every other Website breaks its layout/design.
You can whitelist sites, deactivate your filtering choice or launch a second browser with this WebExtension on AMO, part of these projects by Andy Portmen.
The HTTP Entity Tag is used to reduce the server bandwidth when a resource was previously stored in the cache. Abusing the ETag values allows browser's tracking: the ETag value is unique as an MD5 or SHA1 hash. You can either strip the hash value from every If-None-Match header value with Privoxy or spoof it with Secret Agent Add-on: its spoofing feature adds a random hash to outgoing requests, while its whitelist feature allows you to specify trusted hosts. For Privoxy issues there's an active forum.
LOL!
Hmmm .. where did zymase's comment go? .. Yup, this is testing ETags zymase, there is no JS, no cookies, no dom etc etc
That's why I had removed my comment. Thanks for not including in your brief recall of my shame the fact I love chocolate and mini-skirts!
Yeah, I had commented too quickly, suggesting the LocalStorage to be an explanation of the cache issue detailed by this thread. I then took the time to read extensively, and at this time I've installed The _Secret Agent_ add-on (from developer's page mentioned above by Atavic, when unavailable on AMO).
Hey! I'm a secret agent man as sung by Austin Powers; yeah! -- Seriously, the add-on is most interesting, testing it right now as I said. One thing is sure, the cache issue revealed, exposed, applied with the _ETag Test PoC_ is obsolete with _Secret Agent_.
After some time testing the _Secret Agent_ add-on I'm finally keeping it but will be using only its _eTags & Caching_ feature because it seems to handle the eTag intrusion correctly by spoofing it.
_Secret Agent_ has several other features but too many sites get problematic when these features are enabled. Of course the add-on has a whitelist which is flawless but I've had to whitelist domains, sites only because of the features I then removed. But whitelisting a site will then stop filtering its eTags as well.
In other words, only eTag spoofing and no sites (up to now) need to be whitelisted
_Secret Agent_ then but only for eTag spoofing, which it does well, and which resolves the very concern of this thread, as I see it.
One last thing: Secret Agent is displayed with a toolbar button on its dedicated toolbar, positioned right under the urlbar and reserving the screen width for only its toolbar button followed by description of the add-on's status (what it's doing, is it on or off). Removing the toolbar button is possible, can then be positioned anywhere and the toolbar itself can be hidden with right-click on the tabs toolbar then uncheck _Secret Agent toolbar_. Of course Secret Agent toolbar button alone will not be followed by description of status as when it is on its toolbar, but hovering it provides the information and right-clicking it opens a drop-down menu for whitelisting the site, disabling/enabling Stealth Mode and access to Options.
Done this way, seems interesting to me.
I don't like the idea of needing another addon (and why isn't it in AMO anyway? Not that I would use it if it were).
As I and the author understand it, cache in general is the problem: "[...] _cache tracking issues_ [...]" and "[...] _I currently have no straightforward answer since cache tracking is virtually undetectable,_ [...]" .
I don't understand, why is cache not cleared if Firefox is closed?
This tracking seems to be disabled if browser.cache.memory.enable is set to false (https://github.com/ghacksuserjs/ghacks-user.js/issues/179#issuecomment-317534877).
I would prefer deleting the cache if Firefox is closed. This seems not to work? (https://github.com/ghacksuserjs/ghacks-user.js/issues/179#issuecomment-317547307) What's the status on that?
@Thorin-Oakenpants Thanks for your work then. This is indeed intriguing. I think this must be local indeed, because if I use a second profile, the tracking-cookie isn't there. I wouldn't bother with ETags, because the cache is the generic issue here and who knows that other techniques are used using cache. The real issue is, as you pointed out in your initial post, is that the cache is not being cleared -- if it's set to be cleared -- when FF is closed. I'd personally would want the cache to be cleaned when FF is closed (expected behavior), or I would disable cache altogether (not the preferred solution).
Another fingerprinting test, and this one is the toughest I know : Cross-browser fingerprinting test 2.0
Of course it's always possible to add a uBlockO filter to block _Cross-browser fingerprinting test 2.0_, such as /evercookie.js$important but that solves a test, an effect and not the cause. Moreover I doubt a site using evercookies would name their script evercookie.js.
@zymase
You're right and I just deleted my comment after a short time while you were answering.
Gulty ! That will not happen again.
@2glops it happens to everyone! If deleting a comment is authorized it's for a good reason.
So, we can use medicine to block evercookie.js (namely) but we cannot cure the fingerprinting disease :)
About ETags and performace...
To get rid of ETags, I have set:
user_pref("browser.cache.memory.enable", false);
user_pref("browser.cache.memory.capacity", 0);
No ETags, but I don't see any performance issues too.
^^Is any of those master switch and I don't need the other one?
As I say, I don't see any performance issues. Why don't you try it?
That is OK and understandable. But I have asked why don't YOU try it. I haven't implied anything further, at least not to include it into public version.
you can try this one: https://addons.mozilla.org/en-US/firefox/addon/header-editor/
I have my own WE to do these kinds of things but I'm not gonna release it anytime soon. (or maybe ever)
You can't remove If-None-Match though because it's one of the headers that gets added after WEs had their hands on them. But removing the ETag Response Header should work.
For me too...pleaseeee :)
The code doesn't use .toLowerCase to compare headers but I think existing headers are now always lowercase, meaning you need use lowercase when you want to remove (=use an empty value for it) a certain header. BUT if you want to add one you should probably use the correct casing, fe. if you always want to add a Referer Header, name it 'Referer' not 'referer' or you might make yourself pretty unique because nobody else sends a lowercase referer header!
Anyway, I think this should do:

you can do some testing on http://httpbin.org/ if needed.
Thank you @earthlng, this seems to get rid of etags with memory cache enabled.
Nice
Test at http://lucb1e.com/rp/cookielesscookies/ also passes in desired way.
If-None-Match sends the Etag if one was received but since you're now removing it there's nothing to send. And you can still profit from caching because of the Last-Modified header. That could also theoretically be used to fingerprint by sending unique dates for each user and request but I don't think anyone does that (yet?!). If you want to remove that too you might as well just disable caching again.
Shame I used a shit extension to test it.
comments like that are part of why I'll probably never release an extension again. Just because people don't understand how it works and what it does, doesn't make it shit. Just sayin'
It didn't work because FF adds the If-None-Match header after changes made by WEs. Not knowing that doesn't make you an idiot but ranting about how shit it is makes you kind of an asshole. It even clearly states
Modify Header Value can add, modify or remove an HTTP-request-header for all requests on a desired website or url.
obviously if you try to modify a response header with that extension it won't work.
There's a big difference between "it doesn't work - this is shit" and "I can't get this or that to work - can you help, please", wouldn't you agree?
Re: last-modified - you could use a custom function to add a random number of seconds or minutes but since I doubt that this FP technique is commonly used, it would make you stand out more on most websites than just sending back the original value.
Sure I am. :)
Why aren't the Dephormation and Secret Agent add ons published on Addons.mozilla.org?
I don't subcribe to the view that there can only be one marketplace for computer software.
^From dephormation.org.uk FAQ
RAS has a Send Spoofed If-None-Match headers option
https://github.com/dillbyrne/random-agent-spoofer/wiki/Headers-Tab-Usage#standard-options
Send Spoofed If-None-Match headers- ... I would like to improve on this so we eventually send nothing but have not been able to get a better solution thus far.
Well, I just gave you that solution. You can of course "spoof" the If-None-Match header indirectly by changing the incoming Etag header but AFAIK you can't tamper with the outgoing If-None-Match header directly. This is at least the case for Webextensions, it might have been different with legacy addons.
And ...
Random Agent Spoofer
Note: This addon is no longer being maintained as it uses the legacy addon sdk. I will not be porting it to web extentions due to certain incompatibilities with the web extensions API. It will stop working in firefox 57
@Thorin-Oakenpants I read that entry some time ago but after skimming the source code I couldn't find anything suspicious. In fact, the source was surprisingly simpler than I expected. Also, some of the buggy behaviour is explained in the very test page, below.
I wouldn't go as far as to call it a fake test. I'd just say it's kinda buggy, unreliable. Probably in part due to its simplicity, and the fact that it seems like it wasn't updated in four years.
Then again, I am by no means an expert on the subject.
I just skimmed the source for a second time and it seems the IP and user agent are used as seeds to first generate a unique ETag for each user, but using a pseudo-random number instead should produce mostly the same results in that context. It shouldn't be relevant to the tracking mechanism.
Something else must be in play. It would be nice to find out what that is, exactly. Not just out of curiosity, it could be useful. For starters we could test from different browsers, to see if it is a Firefox-only thing or not.
Of course, summoning some gurus here would hopefully save us the hassle of testing............. starts drawing the summoning circle just in case
Here are my latest test results using the cookieless cookies test page at http://lucb1e.com/rp/cookielesscookies/ (note: read the whole page... there is good information within).
On Firefox v58.0beta10 (tweaked), it always shows 2 visits, and does not retain the text. Refreshing the page does not increment the counter, and the store string is not maintained. I'm not sure why it always shows 2 visits, but the counter not incrementing, and the string not being stored, are both encouraging. Anyone have similar results and ideas on how to interpret them?
On Firefox Klar/Focus (Android) v2.5, the counter increments. Clearing the cache one time has no effect on the next time the page is loaded. However, loading the page one more time (or clearing the cache again) reduces the counter to 2 and removes the stored string. Again, I'm interested in your feedback on interpreting these results.
What seems to be the problem here guys? That it always says "Number of visits: 2" ?
It's because his sessions folder is stacked full with previous sessions because it never cleans up.
Clearing the cache one time has no effect on the next time the page is loaded.
that's because it takes a short while to clear the cache
@Thorin-Oakenpants
do I stay on the inside or outside of the circle?
On the outside, unless you don't mind lending some blood for the ritual.
EDIT: Actually, I really hope you don't mind lending some blood. The quality of the guru in question may depend on that. Just imagine how awesome it would be if Steve Gibson appeared. Or Kevin Mitnick.
Wow I see a lot of confusion going on here lol. I tried to explain what I found in the code and why it's theoretically acceptable to make ETags out of remote IP + user agent, but the explanation kinda got buried down there, among the many test results and different claims.
I should've been clearer from the start. Regardless, I still can't explain what's going on with the test once the client changes IP.
@Thorin-Oakenpants It this typo?
Option1: don't use disk or memory cache
and should be:
Option1: don't use disk and memory cache
@Thorin-Oakenpants Hi! I don't think you meant to @ me? I'm not really sure what you're talking about :)
Ok, so after spending quite a bit of time testing Header Editor, Secret Agent, and, to a lesser extent (since it only deals with 3rd-party ETags, so it doesn't seem to have any effect here), Privacy Possum, I'm at a loss. I've read this whole thread and about half the comments so far at ghacks, and I'm not sure how to test this. I set up a rule in Header Editor and tested it on lucb1e (PoC?), and the only way I've been able to get that site to "forget" me is to either clear cache (thanks to @earthlng for their comment about it taking a while, as I was also struggling to figure out why sometimes this worked and sometimes it didn't) or with Secret Agent*.
After reading everything so far, it seems the lucb1e site doesn't work properly, and it sounded like Pants was going to remove it as a test site, but it's still listed in section 4.2.4 of the wiki. As for the other site listed there, I have no idea how to use that. I also tried the site @Atavic posted above, but it doesn't seem to do anything, and the iframe doesn't seem to load properly:

@Thorin-Oakenpants - I pointed out to you elsewhere that you had misspelled JonDonym on the Appendix A of the wiki, but I see you've spelled it the same (JoDonym) everywhere else as well, including the ghacks article comments. Not important enough to go tracking them all down to fix them, but just wanted to mention it again in case you didn't realize how widespread it was. 馃槈 And of course, thanks for yet again digging into this stuff.
*As far as Secret Agent is concerned, I have no issue with it not being on AMO, especially considering how Mozilla handles (dictates) certain things with it, such as what's allowed now regarding XUL addons. That said, while the addon certainly seems to work well for this, and has lots of features, there's a _lot_ of room for improvement with it that I see. Unfortunately, it doesn't seem to be under active development (and I seriously doubt that will change with the death of XUL addons) and the dev doesn't seem open to input, so I'd just as soon figure out an alternative method for dealing with ETags.
There is no alternative. Either don't use any cache (disk or memory) or clear it constantly (which still leaves x minutes of overlap). _Or block the header._
This is what I meant by an alternative, i.e. an alternative to Secret Agent. The problem is, I can't figure out if Header Editor is working the way it's supposed to, since the PoC uses the IP to seed the tag, and therefore isn't really testing ETags specifically. I just followed Kane's instructions and either it's not working or I'm not understanding it. I loaded the page, checked the header info, saw no If-None-Match line in the request headers, reloaded the page, the line was there, cleared the cache, waited, reloaded the page, the line was gone (due to clearing cache), reloaded again and it was back.
and therefore isn't really testing ETags specifically
If I interpreted the source correctly, it is. The fact that the PoC doesn't hold when you change your IP or user-agent is a side effect of how the former generates the unique ETag to track you. As long as you don't change your IP or user-agent, you can still use it for the purpose of checking if your browser is sending If-None-Match headers or not.
I would suggest you to make sure nothing in your setup is changing your user-agent. Also, make sure to reload the page to see updated results. From the PoC:
To demonstrate how this works without having to use Javascript, I had to find a piece of information that's relatively unique to you besides this ETag. The image is loaded after the page is loaded, but only the image contains the ETag. How can I display up to date info on the page? Turns out I can't really do that without dynamically updating the page, which requires javascript, which I wanted to avoid to show that it can be done without.
This chicken and egg problem introduces a few bugs:
- All information you see was from your previous pageload. Press F5 to see updated data.
- When you visit a page where you don't have an ETag (like incognito mode), your session will be emptied. Again, this is only visible when you reload the page.
I did not see a simple solution to these issues. Sure some things can be done, but nothing that other websites would use, and I wanted to keep the code as simple and as close to reality as possible.
Note that these bugs normally don't exist when you really want to track someone because then you don't intend to show users that they are being tracked.
Not changing my IP or user agent, and I'm refreshing the page over and over trying to figure this out. And I guess I'm more confused than I thought, since I was understanding the issue to be that, since it uses your IP to generate the ETag, if the IP _doesn't_ change it causes the same ETag to be generated again, making it appear as though it was never blocked, when in fact it's just replaced by an identical one.
@Thorin-Oakenpants we're removing the response etag because that prevents the browser from storing the etag, which in turn prevents it from sending the if-none-match header in the future (which can be used to track us). I just re-tested the PoC and had the same results as you.
To avoid further confusion, we should forget about the PoC for a moment. We don't need it to test this.
Now, if you ask me why the etag still shows in the response headers (when the Header Editor rule is supposed to be removing it), my guess is that whatever hook Header Editor uses for modifying the response header kicks in after the Dev Tools load that info.
As for what is going on with the PoC... well, I say F it. It probably has to do with the other cache headers or something.
Not sure what to tell you, it sure works for me. You might need to reload a couple of times to get the 304 status code. Other than that, maybe use the proper casing in the Header Editor rule for the etag header? As in ETag instead of etag? I think it shouldn't matter but.. who knows.
Also keep in mind that whatever image you use to test needs to have a decent time to live, otherwise the server will refresh the image in your cache more often and that could mess with this test.
Javascript could mess with that too, make sure it's disabled for that site for the purpose of testing.
Weird. Both ETag and etag seem to work on my end. Sounds like it is unrelated. Maybe it's just one of those weird timing things that probably happen on the server side. Who knows.
Wait a while and try again with etag if you're curious. I don't know lol
are you using the user.js basically as is?
I'm using mostly minimal overrides on this profile. I enabled most of the Personal section, enabled disk cache, and relaxed the shoulder-surfing and related stuff I don't need to worry about.
@claustromaniac's method worked for me. Glad that's finally sorted out, been working on it on and off for a few days now. @Thorin-Oakenpants, I had the opposite problem as you at first: the tag wasn't present even after disabling Header Editor and reloading the page again. I wonder if your problem is related but in reverse. Maybe try disabling the rule, shift+F5, F5, enable the rule, shift+F5, F5. That's basically how I got it to _show_ the tag then discard it again, and it seems to be working now. I guess there's some interference somewhere in there with all the back and forth and reloading and cache clearing.
I'm suspecting it might be an issue with the extension. Two of my oldest rules disappeared the last time I restarted the browser (just because).
Maybe it doesn't have anything to do with the casing, but rather with the way the extension saves the settings.. maybe something relevant changed in one of the updates in the time frame between the moment you created that rule and today (that you edited it, or created a new one).
I created three rules: etag, ETag, and eTaG. Only the first worked. It also worked with all of them enabled, so maybe a bunch of different rules (ETAG, etag, ETag, Etag, ...) could be made, or maybe it could be done with regex (something like [Ee|Tt|Aa|Gg]). Or would that possibly make it stand out and be more unique? Also, are you testing with the ghacks link from claustromaniac's post? I just assumed you were, but if not that could explain the difference.
Not a typo. The first one ("etag") is the one I was using all along, but couldn't get it working (or verify that it was working). What I was saying worked from claustro's post was verifying it with the ghacks site. I'm not sure if the difference was just using a normal site vs the "flawed" test/PoC, or if the time I tried following Kane's instructions it just happened to not work. So does that mean you didn't try with the ghacks site? If that's the case, try that one with "etag."
OMG! before you guys all go nuts I'll step in here :)
The problem looks like it's caused by a recent change in FF, not the extension. Not too long ago all headers were exposed by FF in exactly the casing sent by the server. Then they changed it to all lowercase for a while and now they apparently changed it back to original casing (IDK when exactly).
@vertigo220 what version of FF are you using?
But the extension code has also changed since the last time I looked at it, and frankly is still less than ideal. For example certain headers can exist multiple times and the extension only mods the 1st one.
For ETag it probably shouldn't matter but you never know. fe. IDK what FF does when a server sends 2 ETag headers.
The best solution that doesn't rely on FF keeping its current way of providing the headers nor any potential future changes in the extension itself, is to use a custom function. That should also work with all FF versions (ESR52.x included) and all current, former and future releases of the extension.
And it bypasses the less than ideal implementation of the extension.
So, change your rule from Normal to Custom Function and paste this:
for (let a in val) {
if (val[a].name.toLowerCase() === 'etag') { val[a].value = ''; }
}
Don't use this in Chrome though because Chrome preserves empty headers and it might end up sending an empty If-None-Match header, IDK.
You'll know that it works when "Number of visits" never goes higher than 2 @ https://lucb1e.com/rp/cookielesscookies/
Oh, and I forgot to mention that you can't rely on the devtools for headers because it doesn't accurately show changes made by extensions. In this case you'll see that it lists ETag in the response header despite it having been removed but you can deduce that the removal worked when you refresh and the next request header doesn't include If-None-Match. So despite being less than ideal the devtools are still somewhat useful because of the connection between these 2 headers.
Thanks @earthlng! That function is the only way, besides Secret Agent, that I've been able to pass the test at that site. And it's good to know it's more robust. Am I correct in understanding that code doesn't change the case, so etag=xyz will be set to etag="", ETag=xyz will be set to ETag="", and so on?
Also, I'm using Waterfox 56.2.0x64.
An OT question: could you or someone else tell me how I would make the following redirect work:
^http(s?)://www.amazon.com/gp/buy/spc/(.*) > https://smile.amazon.com/gp/buy/spc/handlers/display.html?
I figured out how to do it with Request Control, but since it seems Header Editor is capable of much more, I'm trying to transfer the rules from RC into it so I can reduce my number of addons. I've tried different variations, and the one I pasted here worked once (I think), but hasn't worked since.
Am I correct in understanding that code doesn't change the case, so etag=xyz will be set to etag="", ETag=xyz will be set to ETag="", and so on?
yes it doesn't change the case.
I'd recommend to stick with RC or Redirector for redirects because it's a lot easier to match precisely the request types that you want/need.
Thanks. I had my doubts, since it doesn't follow the "rules" in the examples from the wiki, but it does work, sort of. Unfortunately, it doesn't work when landing on the page, only when refreshing it, so I guess I'll have to ask on the HE repo.
HE is not great for redirects because I think it matches all request types. That can be pretty terrible actually.
Unfortunately, it doesn't work when landing on the page, only when refreshing it
It's probably not a normal link but uses some JS to load it. Use RC with request type Document and then always middle-click those links to open in a new tab, that should work.
OMG! before you guys all go nuts I'll step in here :)
If that includes me, I'm sorry to say you're too late. Like...years late.
Thanks for the explanation and the solution, mate! BTW did someone translate the HE documentation to english without me finding out? Last time I checked it was only in japanese IIRC. Do you know japanese? :stuck_out_tongue_closed_eyes: (EDIT: NVM. I see that part is translated now)
I forgot to mention that you can't rely on the devtools for headers because it doesn't accurately show changes made by extensions. In this case you'll see that it lists ETag in the response header despite it having been removed but you can deduce that the removal worked when you refresh and the next request header doesn't include If-None-Match.
I pointed this out in my post above and everyone seemed to understand it. If my guess is right the inaccuracies should be in the response headers, especifically. Did you ever happen to investigate that too, by any chance? Just curious.
BTW, one of my HE rules that disappeared from my list today just reappeared a while back. LOL! The other one is still MIA.
Clastro - have you looked inside the HE storage?
And what should I be looking for in there, exactly? I'll probably wipe it and rewrite the few rules that I use, anyway. It's not much of a hassle.
I'm still wondering why we got different results with the casing in our tests earlier. I only had issues with the lowercase etag rule when I tried the PoC, but both etag and ETag seemed to work for me everywhere else.. Then again, I didn't test much. I didn't stay around here for too long.
I'm not using groups either. I just had 5 rules and nothing really complicated. I wouldn't mind digging into the issue in other circumstances - and I probably will, if it turns out to be a recurring issue - but I'm pretty sure I'll invest less time rewriting them for the time being.
Plus, I get to save _you_ the hassle of helping me with that! How nice is that? You sound tired enough as it is, :jeans:
@earthlng said:
It's probably not a normal link but uses some JS to load it. Use RC with request type Document and then always middle-click those links to open in a new tab, that should work.
It is JS, which is what gave me a hard time originally, but I already have it working in RC. I was just hoping HE could do the same thing so I could eliminate RC (nothing against it, just wanted to slim things down a bit).
@claustromaniac - Once you rebuild them, you should be sure to export them in case it happens again.
@claustromaniac
I pointed this out in my post above and everyone seemed to understand it.
yes I saw that, just wanted to re-confirm. And I'm not so sure everyone seemed to understand it :)
If my guess is right the inaccuracies should be in the response headers, especifically. Did you ever happen to investigate that too, by any chance? Just curious.
No I did not. As far as I can tell the inaccuracies are all over the place
I'm not so sure everyone seemed to understand it :)
That's ok. I admit I often come short when I try to explain stuff to others anyway.
No I did not. As far as I can tell the inaccuracies are all over the place
I've never seen inaccuracies in the request headers but I'll try to keep that in mind in case I ever do. I better stop trusting the devtools on that. Thanks again.
BTW, here's another, unrelated cache tracking technique you may or may not be aware of which, AFAICT, can't be defeated except by periodically clearing the cache. Thought you might find it interesting.
posted in #436
Done.
Most helpful comment
The code doesn't use .toLowerCase to compare headers but I think existing headers are now always lowercase, meaning you need use lowercase when you want to remove (=use an empty value for it) a certain header. BUT if you want to add one you should probably use the correct casing, fe. if you always want to add a Referer Header, name it 'Referer' not 'referer' or you might make yourself pretty unique because nobody else sends a lowercase referer header!
Anyway, I think this should do:
you can do some testing on http://httpbin.org/ if needed.