Since I've not been able to re-open bug #581 I create a new one, but it is still the same problem.
Not only the bug hasn't been fixed, but I would say it's even worse now than before.
If you have a look at bug's description (in my 1st post of #581 thread), you can see that bug (ie. lack of cleansing of storage relative to the trainline.fr domain) used to take place "only" when closing Firefox without closing first the tab opened on the domain, nor switching this tab to another domain.
Now, with the "fix", local data of Trainline.fr are NEVER cleansed !
Whatever the way you close tab or Firefox.... Once you logged into Trainline, you are never logged out except if you choose to explicitely log out.
And nevertheless, CAD's pop-up and the logs show that cleansing of cookies take place.
This means that, at least with this website, local data are NEVER cleansed.... while
At least, before the "fix", the bug was active only in some cases. Now, it is ALWAYS active.
It's not a fix, it's a regression....
And I'm sorry to say that it makes your extension 100% useless....
Hopefully, there is not the same issue with "Forget me not" extension, to which I switched while the bug wasn't fixed yet....
I'm sad, because I expected to go back with CAD after fixing...
Because, "on the paper", CAD is better than "Forget me not" on containers management.
But only "on the paper", since actually, CAD do not work at all.....
I'm going to try and get some debug lines into the extension itself and see what outputs we get. Please be patient as we try and work it out together.
Now I don't understand French, but in an attempt to explore that site in question, it appears to use a third-party site thetrainline.com to handle its logins. I probably already know the answer to this question but for clarity, thetrainline.com is not part of any whitelist/greylist?
Hi,
yes, I suppose that Trainline.fr is sowhere "only" the french portal of some more international website that could be thetrainline.com.
But clearly, there is no rule, neither grey nor white on this domain.
All my rules are all on domains I know and I can be 100% sure they are not linked in any way to Trainline...
ok. one more thing: can I ask you to take a screenshot of the storage Inspector? (Menu Icon / Tools) > Web Developer > Storage Inspector. Expand all the entries shown on the left and screenshot it over to me. Namely the cookies and LocalStorage entries with their entries on the right. Sanitize any personal information if needed.
Sorry for late answer, but due to time lag, I was in bed when when you asked... and I send you answer when YOU are in bed...
In the subdir "Before loging in", you can find images of local storage when I had opened trainline.fr but not yet provided my credentials to log in, and in "While logged in", the same after providing these credentials.
If I switch the Ffx tab to another domain, CAD tells me it clears 5 cookies, but if the I go back to trainline.fr, I'm already logged in, and content of LocalStorage is exactly the same than previously
ScreenCopies.zip
Ahh. Yes your domain uses www. to store its data. It was definitely mentioned in the previous issue. Next version will include www. to all domains that require localstorage cleaning. I guess I missed that when I was trying to get chrome localstorage function working. (they needed the http(s) protocol as well while Firefox only needed the domain and subdomain.)
My apologies there.
I'm not sure to understand what you mean, when you write "Yes your domain uses www. to store its data.".
What I know it that the URL used to log in is https://www.trainline.fr
I don't know (and afaik I cannot know) if the web site stores data only on this domain (www.trainline.fr) or also on other associated domains. On the other hand, thanks to "Cookie manager" extension, I can see it creates cookies not in this domain, but in .trainline.fr domain.
(In particular, I don't know any mean to see if there are also localdata on .trainline.fr domain)
However, what is absolutely necessary is that CAD is able to delete ALL cookies and ALL localdata created when browsing this website.
And more generally, to be able to delete ANY cookie and ANY localdata created when browsing any site, except those associated to a domain specifically white or greylisted.
I don't know if and how this is feasible, but if it's not feasible, it means that CAD is useless...
Sorry to be so "rude"....
To try to be a little bit more "positive", a first approach could be to manage
Will then remain to determine any other domain on which localdata could have been created, without creating a cookie.
For example, in trainline case, cookies could have been created on www.trainline.fr domain. But would it be possible for Website to have created localdata on .trainline.fr domain ? If yes, on which domain would it be possible ?
I do not have any technical skill in this domain, and cannot help you in providing solution idea, but at least I want to be sure we share the same goal : Have CAD able to delete any cookie, or any localdata except those explicitely "protected" by white/grey list.
Hi Kenneth
I ran a test I want to share with you.
Thanks to "Cookie Manager" extension, it is feasible to manually CREATE a new cookie.
My test is the following :
Result : Localdata has been correctly cleansed (probably thanks to this supplementary cookie) since I am no more logged in.
BUT (because there is a major "but") if I replace step 4 by closing FireFox without closing before the tab, nor switched to another domain,
it appears that Localdata has NOT been cleansed by cleansing at Firefox startup, since I am already logged in !
Which is EXACTLY the bug I described in #581 : Absolutely NO CHANGE !
Conclusion of my test : rev 3.2 of CAD
I'm afraid I'm far from being able to use again CAD.....
Since I remembered that CAD used to work in the past, I downloaded the different versions of CAD, to test them and see when it began to dis-work.
It was a little bit complex since old versions of CAD were not signed, and then I had to install ESR version of Firefox, and set a property to allow unsigned modules, but I eventually succeeded.
And the result is that the last version of CAD that worked fine (at least from this standpoint) was 2.2
Starting with 3.0, the bug described in #581 is there...
I would prefer to find a "border" between two "minor" releases, because I guess that there are a lot and a lot of changes between 2.2 and 3.0, but nevertheless, I hope that this information can help you to find bug's origin.
I only started contributing to the project just about a year ago, so I can't say much for v2.
I had left the last major versions of CAD on AMO itself...so it should still be accessible somehow online.
From what GitHub comparison told me, v2 was using Javascipt mainly, and v3 transitioned to using TypeScripts.
In addition, from https://github.com/Cookie-AutoDelete/Cookie-AutoDelete/issues/482#issuecomment-453884635
To everyone else:
The reason why this appears only in 3.0.0 is because of #437. So before 3.X.X, all these errors were silently failing (and probably not cleaning correctly) until I pushed those errors up into a notification. So all this error information will help me improve CAD. Thanks again!
This meant that if any errors occurred while cleaning, it gets ignored so the user doesn't know about any of it.
I've also spotted a minor difference related to this is how the prepareCookieDomain was done.
in 2.2.0, The cookie domain was only trimmed if the dot was the first character. The current version trims both the first dot and end dot if it exists.
In both versions, it then gets passed into getHostname which attempts to parse the domain into a JavaScript URL then extracts the hostname given, removing the www in front of it if it exists.
2.2.0 also had localstorage cleanup but it appears it didn't have a separate localstorage clean or specific cookie keep option (i.e. if the domain wasn't on the list then all cookies and localstorage would be attempted to be removed).
Could you try the builds posted in #713 and let me know?
I haven't touched how the restart of the browser handles it, but I did try to clear more of the localstorage blindly based on the cookies given ( sub.example.com will try to clean localstorage for www.sub.example.com, sub.example.com, and .sub.example.com). Hopefully your case will fall in here.
There's debug mode in this one, so if you're comfortable with using the browser extension console logs, then feel free to try and see what it is doing.
And for cookies/localstorage to be cleaned on browser restart, make sure the option is checked in settings as well.
OK to help, but not understood how to install this build ????
Best to use a fresh profile first. See https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles to assist with this.
Head to about:config and change xpinstall.signatures.required to false. Then you should be able to install by going to about:addons then click on the cog icon and Install Addon from File.
Hi Kenneth,
my question was because I didn't find any xpi file... and didn't know it was possible to install addon directly from zip file...
On the other hand, FYI it's no more feasible to install a non-signed addon with latest version of Firefox, even with xpinstall.signatures.required set to false. To install such addon, you have to install an ESR version, like 68.8esr
Since I run my tests in a VM under Virtualbox, no issue to do that, and thus I eventually installed your "debug" build.
But the result is there, and is negative.
The bug is still there with this build
I'm afraid that it won't be with such a "step by step" approach that you can solve the issue, but only with a global re-exam of what CAD does....
my question was because I didn't find any xpi file... and didn't know it was possible to install addon directly from zip file...
you can rename zip files to xpi and it works the same.
On the other hand, FYI it's no more feasible to install a non-signed addon with latest version of Firefox, even with xpinstall.signatures.required set to false. To install such addon, you have to install an ESR version, like 68.8esr
You can also install Firefox Developer Edition (currently 77.0b2 i believe), and it will install after doing the config above.
Since I run my tests in a VM under Virtualbox, no issue to do that, and thus I eventually installed your "debug" build.
But the result is there, and is negative.
The bug is still there with this build
There's a debug mode to which you can check the console logs. I've made it so it would show almost every thing that the extension does, including which hostnames are being passed onto the removeLocalStorage function from the webextension API.
I'm afraid that it won't be with such a "step by step" approach that you can solve the issue, but only with a global re-exam of what CAD does....
I'm going to try to write a high but detailed cleanup workflow for you to follow along in the next post.
Here's a somewhat detailed high-level overview:
When the background script is loaded, it gets the settings from storage and parses it for use, and adds listener to the folowing:
On every browser start, this extension hooks into browser.runtime.onStartup.
Going into the cookie cleanup operation:
sub.domain.com and .sub.domain.com)www. to the domain, and add it to the array. (e.g. www.sub.domain.com).www. to the domain, and add it to the array (e.g. .www.sub.domain.com)http:// and https:// to each domain returned in that array and adds them to the clean list.I'll reopen this to allow status updates after either your tests from the unsigned build or the next version push shows it is working.
Hi Kenneth,
surprised you are still waiting update of my tests from unsigned build : I already answered 4 days ago, and the result is that this unsigned build DO NOT CHANGE ANYTHING : bug is STILL THERE, exactly THE SAME.
Edit 20 minutes later : I just saw there were a new unsigned build : So I did a new test, with
Cookie-AutoDelete_Dev_20200509_054429_Firefox.zip
and BUG IS STILL THERE
A: is 'clean cookies from open tab' enabled?
B: could I get a copy of the debug log? You can save the output into a file and either email it or upload it here as it will be a long one.
A : Yes, of course...
Nevertheless, this shouldn't impact since there is no open tab on trainline.fr at startup, when cleansing is not correctly done
Find attached my settings
B : Find attached two debug logs
B1 : Scenario 1 : Open FFx, open Trainline.fr, log into trainline.fr, switch tab to another domain, wait for CAD cleansing, get copy of Debug log
B2 : Scenario 2 : Open FFx, open Trainline.fr, log into trainline.fr, close Firefox, re-open Firefox (that open on about:blank, and NOT on last opened tabs), wait for CAD cleansing, get copy of Debug log
In scenario 1, cleansing is OK, since if I open again a tab on www.trainline.fr, I am NOT logged in
While in scenarion 2, cleasing is NOT OK, since if I open again a tab on www.trainline.fr, I am ALREADY logged in.
Finally solved the root issue - this time :)
Please confirm that it is fixed on your end as I had used the html5 localstorage test site that was mentioned in previous issues to test it out.
Cookie-AutoDelete_Dev_20200511_200027_Chrome.zip
Cookie-AutoDelete_Dev_20200511_200027_Firefox.zip
I also fixed a chrome storeId bug, so maybe try these build instead, not that it should affect firefox much I believe.
Cookie-AutoDelete_Dev_20200511_211217_Chrome.zip
Cookie-AutoDelete_Dev_20200511_211217_Firefox.zip
Yes, it's a lot better !
The bug as described in #581 is gone...
Nevertheless, I still have some issue with greylisting.
Resulting of my tests, I'm afraid it could be a bug in greylisting, but since I'm not sure to have understood exactly how it's supposed to work (doc is not really clear on the topic), maybe it comes from my misunderstanding, and I prefer to submit to you a few use cases in order for you to provide detailed rules able to manage them.
Afterwards (and afterwards only), I'll re-do the tests, and in case there is really a bug, I'll create ticket.
Thanks in advance for your answer.
Use Case 1 : For website *.UC1.com, I would like
Use Case 2 : For website *.UC2.com, I would like
Use case 3 : For website *.UC3.com, I would like
Use case 4 : For website *.UC4.com, I would like
Use case 5 : For website *.UC5.com, I would like
It's mostly likely a bug as only the first matched expression found will be processed. Not sure how he envisioned it at that time.
In other words, either clear cookies on domain change OR browser restart. Not both.
And as for localstorage, there is no 'some' for one domain. It is either clean localstorage for entire domain or not at all. This is a browser API limitation.
This will be for future release then.
It will get complex in the backend, but I guess open a new issue with those usecases anyway.
For Localstorage, it's well known (and up to now accepted) that we cannot "make detail" in what is cleansed : all localstorage of the domain, or nothing.
I understand also (and I accept) that only first matched expression found is processed. Otherwise, it could be very very complicated if two expressions relative to the same domain were contradictory.
But I fully disagree with your conclusion "In other words, either clear cookies on domain change OR browser restart. Not both."
1) Because it would be an unaccceptable limitation of the tool, on its "functionality" standpoint
2) Because this consequence can be avoided quite easily :
An "easy" solution could be to
The only drawback would then to have to create rules twice, once in white, and once in grey, when we want to have the same in the two events.
And to avoid this (quite minor) drawback, why do not invent THREE colours (rather than only two) :
As a result, if I want to have the same rules for a domain in both events, I create only one Grey rule.
And if I want have distinct rules at domain change and at startup, I do NOT create a Grey rule, but one White rule and one Black rule.
Sounds easy to use, and efficient.
No idea about easiness to implement.... It's up to you Kenneth !
PS : If you prefer other colors, you can choose Yellow, Orange and Red : The idea is to have two colors dedicated to the two events (one color per event), and one color being the mix of two others (orange is a mix of yellow and red) available for the two events....
A) Black is already reserved for blacklist i.e. rules that will remove cookies upon creation / when possible. When that will be implemented is probably soon.
B) For backwards compatibility, white will always apply for all rules as it is the one that keeps the cookies from being removed in any scenario (when checked).
C) Rules considered at startup and ignored at tab close and domain changes - for cleaning this applies to checked greylist items. unchecked will be treated as clean on domain change/tab close/browser restart.
D) Clean at domain change/tab close are all other domains/cookies that don't match anything. That will stay as is as part of the main autocleaning feature. Otherwise how will you remove third-party cookies?
If rules are created for domain change/tab close, why not also clean them at browser restart?
Looks like it may have not been interpreted properly when I said In other words, either clear cookies on domain change OR browser restart. Not both.
I may be stuck behind the times for now, but the base features stemmed from the legacy Self-Destructing Cookies extension, so the main logic will stay as is. White checked - keep always. Grey checked - keep until restart. nothing - clean on tab close/domain change.
However, that's not to say that in the future, I may find a way to (instead of returning the first match), get all matched expressions for that container, and then go through the logic for each one. (Blacklist when implemented first), then Whitelist and grey list. falling though all that will be removed on domain change. This would probably make the use cases mentioned above more likely to pass the tests.
Sorry, but I really do not understand what you mean....
You begin explaining that this color is already reserved for....
But I don't care of colors. You can call it black or white, or call it apple or strawberry...
I described you some real use cases. Not only ideas, but real needs.
If something is possible in current situation to obtain these results I describe in my different use cases, please give us precisely what rule(s) to define for these 5 domains, UC1.com to UC5.com....
And if it is not possibe today, explain me why... and you'll understand how to modify CAD !
Actually, I'm afraid that CAD is became very complex (maybe not tool itself but the comprehension we may have of it) because the initial principles have been lost.
Let me remind that in its principles, CAD is supposed to delete all cookies (and local storage) EXCEPT those that are PRESERVED by rules.
Rules are NOT there (at least were not there in the beginning of the tool) to describe what to delete, but at the opposite what to DO NOT delete.
In particular, you wrote : "_Otherwise how will you remove third-party cookies?_"
IMHO you ask the question because you are not in the line to delete _all except_ what has been defined to be protected by rules.
I don't care that the cookie is a third-party cookie or not.
The only question is : Did I write a rule to keep it, or not. Either a "global" rule on its domain, or a specific rule on the cookie itself. If not, it has to be deleted. Whatever it has been created while visiting its domain, or created while visiting another domain than its own (which is the definition of third party cookies : cookies created on domain B while browing domain A)
In the very first versions of the tool, rules were a little bit "limited", since we were only able to say : do not delete anything from this site. And moreover, CAD was not able at all (at this time) to delete local storage, so the question was not even asked.
In following versions, it became possible to be more specific, and to ask for preservation not of all cookies of a given site, but only some of them, precisely designated by their name.
And since in the same time, since CAD began able to delete Local storage, it was also possible to specify by rule that CAD has to DO NOT delete local storage of a given site. But here also, never define which Local storage to delete, but which local storages to DO NOT delete.
According to me, it is when "Grey" list has been introduced that things begin to go wrong.
Not because the idea of grey list was wrong : it is excellent to be able to have a some cookies or local storage that are deleted only at browser restart and not at "domain change". (while, I agree, it doesn't make sense to preserve something at startup that would not be preserved at domain change)
And since rules are there to define "what to preserve" (and NOT "what to delete), to have some cookies (or local storage) that are NOT preserved at startup.
But "Grey" rules have been presented as rules that apply only at startup and not at domain change. And rules began to be a mix of rules to define what to preserve and rules to define what to delete... and nobody can understand any more...
It would have been more clear to define two kinds of rules (and once again, I don't care they are called black, white, apple or strawberry) : the cookie named xxx from domain zzz (or any cookie of domain zzz), a/o the local storage of domain zzz has to be preserved
Grey rules are actually rules that DO NOT apply at startup, rather than the opposite....
I think it is time to go back to fundamentals. Otherwise CAD will look more and more like a mess...
Personnally, I am less and less confident in what CAD evolution can lead to.... if you (or someone else) do not go back to "robust" fundamentals (unfortunately, I am not personnaly skilled to develop this evolution)
Somewhere, you wrote "_....For backwards compatibility...._"
After all our exchanges, I'm less and less convinced you have to maintain this _backward compatibility_
At the opposite, I'm more and more sure it's time to create a fork of CAD, and restart on good basis.
Do you want to run the challenge ?