Https-everywhere: Custom Rules with WebExtensions

[kjorand]

Hi,
According to this site it is now possible to add custom rules to Google Chrome by just editing a few lines of it's code, but doing so we corrupt the app which Chrome doen't seem to appreciate it.
Could it be possible to apply the trick descibed above directly in the code of the extension (no corruption, no disparition on update)?
Then we would just have to edit

C:\Users\****\AppData\Local\Google\Chrome\User Data\Default\Extensions\gcbommkclmclpchllfjekcdonpmejbdp\2017.6.20_0\rules\custom.rulesets

Working a lot with local devices (192.168.1.XXX), some of them having to be served over https, others not ... it would be much easier and much less of a pain in the ass ;-)

Additionnally, some websites like Doodle do work with https everywhere on but not with the block unencrypted content option ...

So thinking a little more long term, it'd be great to be able directly from the menu to:

• remember exclusion of the current site (with an openning field to edit the filter: to choose if the whole domain or the current subdomain or current page ?)
• remember the deactivation of the blocking option site to site just as for the activation of https everywhere itself

Thank you for that great app!
There are a few of them on the market, but that one is currently the only one I trust (even though it's a little boring having to activate/deactivate it every two minutes when switching tabs .... )

[jeremyn]

I'm not entirely sure what you're asking for, but Firefox lets you use complete custom ruleset files as described in https://www.eff.org/https-everywhere/rulesets#testing, by putting the files into a special directory. Chrome has a limited ability to add custom rules using the extension GUI with Add a rule for this site.

With the coming requirement for add-ons to use WebExtensions instead of XUL/XPCOM as for Firefox 57, the existing Firefox code will be replaced with something like the Chrome code. It's not clear yet to me how custom ruleset files will be supported going forward.

[gopherit]

With the coming requirement for add-ons to use WebExtensions instead of XUL/XPCOM as for Firefox 57, the existing Firefox code will be replaced with something like the Chrome code. It's not clear yet to me how custom ruleset files will be supported going forward.

I use the custom rules directory in Firefox for all of my development and testing purposes before I even think of submitting a PR. I'd hate to see that functionality disappear.

I barely use Chrome, but I've found the Chrome add-on to have less functionality than the FF one. It's just "different."

[J0WI]

I'm not sure if this dupes something that is going to change for the Firefox Webextension release.

[ghost]

@J0WI Add the high priority label please.

[jeremyn]

@gopherit I also use the custom rules directory, for HTTPS Everywhere maintenance. I asked @Hainish about plans to add this to the WebExtensions version, and he responded in https://github.com/EFForg/https-everywhere/issues/9958#issuecomment-315202693 and following with a (IMHO) tedious and complex workaround. It involves preprocessing the ruleset on the command line and then including the processed ruleset in a JavaScript snippet in the debug console.

This approach is not a good substitute for the existing functionality, but I don't know how pressed @Hainish is for time on getting the WebExtensions add-on finished for general users. I don't know how to prioritize this critical problem for the twenty or so very important users who are maintainers and regular contributors, vs getting the basic stuff done for the million general users.

@Hainish Can you weigh in here please?

[ghost]

@J0WI I think WebExtensions has some mechanism for migrating the rulesets in HTTPSEverywhereCustomRules folder into the localStorage, but that's not real custom rule support. I also think the migration only happens once.

[Hainish]

@jeremyn

• The Embedded WebExtension version is still able to digest rules in HTTPSEverywhereUserRules, https://github.com/EFForg/https-everywhere/issues/9958#issuecomment-321452120
• The WebExtension version does not have the capability to read files to disk

Do you have a suggested path forward?

[ghost]

My suggestion is to migrate the rules from HTTPSEverywhereCustomRules folder into localStorage. For some reason I thought it is already done.

[mgol]

@Hainish I feel what's deeply needed here - and that applies to both Chrome & Firefox versions - is to have a settings UI with an editable list of rules. At the basic level even a textarea with rules in a custom DSL would be a big improvement to the current state of things. Some extensions already have something like that, see e.g. uBlock Origin's rule list (the "My filters" tab in the add-on's options).

[ghost]

@mgol #11794

[Hainish]

@koops HTTPSEverywhereUserRules are being stored via the storage API (not localStorage) as of #11800, as an array under the key legacy_custom_rulesets.

[Hainish]

This is only after a migration is performed, though, which can only be done in Embedded WebExtensions.

[jeremyn]

@Hainish: @mgol and @koops76 have the solution: an in-extension UI where you can edit rules. The key is that we can insert, modify, and persist rulesets which look like normal (not compacted) rulesets, using some kind of text editor.

[jeremyn]

@Hainish: even an "Import file..." dialog that just wraps the JavaScript console stuff you described in the other issue, would be a significant improvement.

[ghost]

@jeremyn Working on that too.

[Hainish]

@jeremyn I'm confused about the way ruleset maintainers are testing rules. Why not just pull the PR, test locally with --justrun, and merge if it works? Why add HTTPSEverywhereUserRules at all?

I'm not against the "Import file" dialogue but I'd rather have it hidden away and standardize the current ui for adding rulesets.

[ghost]

@Hainish I don't test rules locally at all. I rely on Travis to find broken hosts. I probably shouldn't though.

[gopherit]

Here are my use cases for the HTTPSEverywhereUserRules directory:

1. General ruleset testing as they are being written. Comes in especially handy for testing the <securecookie> tag to see what cookies do or don't respond. If everything looks good, then submit PR.
2. Rough draft rulesets that are a long way from becoming PRs. Usually a quick-and-dirty left wildcard target, but in the final ruleset I'll have the specific hosts enumerated.
3. [Less often used] HTTPS hosts in my internal environment that I will never submit PR since they don't apply publicly.

I understand the priority of WebExtensions and making sure everything is migrated on time to the user population, so I'm not impatient about the custom functionality. I do, however, love the idea of an in-extension UI to add/remove custom rulesets. I'm actually fine with just an import XML file/delete XML file function. Built-in editing would be a luxury.

[ghost]

@gopherit HTTPSEverywhereUserRules folder is going away with WebExtensions transition. We can't bring it on WebExtensions no matter how much we want to.

[gopherit]

I'm aware. I was saying how I currently use custom rules and how I'd like to in the future with a UI that enables import and/or editing of custom rules.

[Hainish]

Thanks for the rundown @gopherit. This sounds reasonable, and I'm in favor of adding this import functionality, so long as it's in the new options dialog and not in the popup. This is a feature relatively few people will use, so let's not make it something people will get confused by if they see it in the popup.

[ghost]

@Hainish Is the options dialog a tab or a popup of some kind, similar to popup.html? I am asking since a[download] attribute doesn't work in popup.html and I will need it for settings export.

[jeremyn]

@Hainish My HTTPS Everywhere development environment is in a graphic-less VM based on https://github.com/jeremyn/https-everywhere-dev-environment. I don't run automated tests locally other than fetch-test.sh.

For reviewing PRs that add new rulesets or overhaul existing ones (example: Export.gov.xml in https://github.com/EFForg/https-everywhere/pull/11784) I might do a combination of this (not always strictly in this order):

• See if the set of domains in both the comments and targets is complete. This can involve Sublist3r or Google searches and is essentially a manual diffing between what I find and what the ruleset has. The goal is to find the union of the two sets of subdomains. I exclude domains for various reasons, especially if I'm using Sublist3r which can find a lot of technical domains like mx or ns1.
• Copy/paste the modified ruleset from the PR (looking at a browser at https://github.com) into a text file in HTTPSEverywhereUserRules/ on my local system in a separate Firefox profile configured for HTTPS Everywhere testing. The profile has HTTPS Everywhere and a few about:config modifications including disabling multiprocessing due to a bug with the developer tools. I usually give the ruleset a slightly custom name, especially if it's modifying an existing ruleset, so I can be sure to distinguish the new ruleset from the existing one.
• Start a Firefox instance for that profile.
• Deactivate HTTPS Everywhere and load up every domain in both comments and targets with both HTTPS and plain HTTP to see what happens. This catches different HTTP/HTTPS content problems, and also failures with the HTTP version (if a domain fails HTTP I often exclude it entirely).
• Activate HTTPS Everywhere with the new ruleset -- for existing rulesets, I'll paste one of the URLs into the address bar and then disable the existing ruleset using the add-on UI without loading the page. This is what I described to you in https://github.com/EFForg/https-everywhere/issues/9958#issuecomment-318840745. -- and test all the targets to see what happens.
• (Often during in-browser testing, I'll test with the developer tools Network tab open so I can look for things like redirects and new subdomains.)
• Iterate the above until I'm satisfied with it.

During the above I'll open a new text file and keep notes, which eventually turns into one of the checklists I regularly use (example: https://github.com/EFForg/https-everywhere/pull/9251#issuecomment-320685200).

Putting rulesets into HTTPSEverywhereUserRules/ is a key part of the process. The extra steps you described involving merge_rulesets.py and the JavaScript console, that's just too much.

This approach has a lot of problems. It's a ton of manual work. However the reality is that I almost all non-trivial rulesets that I do this analysis with have some kind of problem, as you can see from the PRs I review, so this level of testing seems, unfortunately, appropriate.

[Hainish]

An XML editor is highly useful for our ruleset maintainers, thus it's a high priority for me. After https://github.com/EFForg/https-everywhere/issues/12270 this is my next todo.

[ghost]

@Hainish #11794

[J0WI]

I'm a bit scarred that this will blow the extension up. Aren't there already some tools included in browsers?

[Hainish]

I literally mean a textarea to enter XML into. I don't want to include a full code editor, that's for sure.

[ghost]

@Hainish err, why?

[Hainish]

This will increase the size of the extension dramatically for something literally only devs will use.

[Hainish]

It also seems to me, from the workflow @jeremyn describes, that it's not really necessary. You should be able to compose the XML somewhere else and paste it in pretty easily.

[J0WI]

You may have a look at the about:debugging feature of Firefox. I was not able to locate the rulesets, but I think that's the best way to do live editing. I might get improved by upstream bugs.
Firefox uses CodeMirror anyway for its DevTools.

[Hainish]

@J0WI for some reason the Embedded WebExtensions version does not allow you to access background frame globals. This is not the case with the full WebExtensions version.

[Hainish]

Merged in https://github.com/EFForg/https-everywhere/pull/12863