Https-everywhere: WebExtensions timeline/checklist

[Hainish]

1. Wait until a few weeks after Tor Browser 7.0 stable has been released, so that users have had the chance to upgrade to a Tor Browser based on Firefox 52 ESR.
2. Change https://github.com/EFForg/https-everywhere-docker-base/blob/master/Dockerfile to use ESR 52.
3. Merge the webextensions branch into master.
4. Make a new release, which will follow the WebExtensions date-based versioning. In AMO, ensure the versioning is correct, so that users are upgraded only if they are on Firefox >= 52 or Fennec >= 53.
5. Generate a signature for https://www.eff.org/files/https-everywhere-eff-update-2048.rdf which ensures upgrading to the WebExtension version if users are on Firefox >= 52 or Fennec >= 53.
6. Create https://www.eff.org/files/https-everywhere-updates.json and ensure it is signed, so that the WebExtensions version can update itself.
7. Create the 2017.X.X tag, and push it to both tor and github remotes

[Hainish]

cc @J0WI @jeremyn @gloomy-ghost @jsha

[jeremyn]

It would be really nice if you could release another version of HTTPS Everywhere, call it "HTTPS Everywhere (WebExtensions Beta)" and make that available alongside the live version for one or two Firefox release cycles. You could even just release it on the EFF site, with a hidden link, and not AMO. I feel like we'll have a much lower chance of catastrophic failure if even a few of us could run with it for a month or two before it goes out to a million people.

[Hainish]

@jeremyn I think that's a fine idea. I'm currently waiting on Mozilla to document how in WebExtensions you can provide a signature and embed the extension public key. This is supported in XPCOM, but maybe not WebExtensions (yet?) This provides a mechanism above and beyond TLS keys to ensure that extension upgrades are securely deployed.

Regardless, providing a temporary WebExtensions beta to interested users wouldn't necessitate that level of secure signing, and can probably be done pretty easily.

[jeremyn]

Is that process different than the lower level review that Mozilla does for add-ons that aren't distributed through AMO?

[jeremyn]

I mean the "unlisted" process described here.

[Hainish]

Yes, it's a separate thing that allows app developers to embed their own public key into an extension, and then Firefox will only upgrade when the upgrade manifest is signed with this public key. This is only available for self-hosted, not for AMO.

[jeremyn]

Oh okay. Can you point to some discussion on that somewhere? I'd like to know more about it.

[jeremyn]

I guess you mean something like https://addons-server.readthedocs.io/en/latest/topics/api/signing.html .

[Hainish]

@jeremyn you can see this by downloading the self-hosted XPI, then checking the install.rdf file in it (it's also in src/ in the source tree), which lists the public key. The extension then checks for new files and attempts to verify the signature in https://www.eff.org/files/https-everywhere-eff-update-2048.rdf and if it verifies, an upgrade can take place.

The documentation for updateKey is here.

There is more comprehensive documentation of the entire process here

[jeremyn]

Thanks!

[Hainish]

I've created a new tag to note issues with the WebExtensions version for Firefox, just called webextensions-firefox. I've already noted one issue: https://github.com/EFForg/https-everywhere/issues/9965

@jeremyn: here's a link to the testing versions of HTTPS Everywhere: https://www.eff.org/files/https-everywhere-test/index.html

I have several files listed there. The first few (older) files are to test the upgrade path. Feel free to install and test it yourself, and if you encounter any issues just tag it with the above-mentioned label.

[Bisaloo]

Was the observatory removed in the process? I can't find anything about it on the new UI.

[ivysrono]

https://bugzilla.mozilla.org/show_bug.cgi?id=1345893
So, Please set WebExtension's HTTPSew:
Firefox ESR 52+
or
Firefox Release 54+

[jeremyn]

Thanks @Hainish , I installed https-everywhere-5.2.16.1-eff.xpi and after forcing "Check for updates" twice in about:addons -> "gear", it's updated to the latest https-everywhere-2017.5.19.1-eff.xpi , so that looks fine.

The upgrade from 2017.5.19-eff.xpi to 2017.5.19.1-eff.xpi didn't prompt me to restart my browser for updates to take effect, is that normal?

Also, like @Bisaloo , the Observatory screen is gone, as is the "Preferences" button in about:addons.

[Hainish]

@Bisaloo @jeremyn wrt the observatory, there is no way to do certificate introspection within the WebExtensions API currently. We're working with the Chrome team to make this possible in the future. Right now, this is an intentional omission - hopefully we'll be able to add it back in the future.

@jeremyn this is expected - WebExtensions are restartless - no need any more to restart the browser after addon upgrades!

@ivysrono Interesting, I'll look into this.

[AlexBeLi]

I have some problem in private browsing mode with webext version on latest Nightlly (2017-05-22). Somebody have too?

[jeremyn]

I get the problem @AlexBeLi is getting in private browsing mode on Firefox 53.0.2.

[Hainish]

@Bisaloo, @AlexBeLi please open separate issues for these, and link to this issue. I (or one of the other collaborators) can then add the webextensions-firefox tag to it.

[J0WI]

See also some missing features in the chrome/WebExtension version:

1291

1292

1293

1681

1779

The user rule feature as we have it in Firefox is quite important for testing.

[J0WI]

@Hainish for the observatory you should get in touch with Mozilla https://bugzilla.mozilla.org/show_bug.cgi?id=1215059

[Hainish]

@J0WI we're in touch with Mozilla, but as far as WebExtension API development, all of the momentum moving forward with API specs has been within Chrome. We're working closely with them to make this possible, but it likely won't make MVP by the time we deploy WebExtensions for Firefox.

[Hainish]

I've released a new testing version, 2017.6.1.1337 - get it at https://www.eff.org/files/https-everywhere-test/index.html

From now on the last sequence of the version for test releases will be 1337, for our elite testers :]

[Hainish]

I've released testing version 2017.6.9.1337 - get it at https://www.eff.org/files/https-everywhere-test/index.html. You won't be auto-updated to the latest version, unfortunately.

[pwd-github]

Have Mozilla still not got back to you regarding a secure signing mechanism @Hainish ?

[jeremyn]

@Hainish It's been mentioned a few times in different places but just to reiterate, it's critical for testing that we have a way to put custom ruleset files into a HTTPSEverywhereUserRules/ directory or have equivalent functionality. Is that definitely prioritized for Firefox 57?

[Hainish]

@jeremyn the creation / deletion of custom rulesets is already built into the WebExtensions addon. There is an open ticket for migration of custom rulesets: https://github.com/EFForg/https-everywhere/issues/10007

[Hainish]

@sabret00the no word yet.

[jeremyn]

@Hainish I just installed https://www.eff.org/files/https-everywhere-test/https-everywhere-2017.6.9.1337-eff.xpi into a new Firefox profile but I don't see an HTTPSEverywhereUserRules/ directory under the new profile directory. Has this functionality been built into a future version of the add-on that you haven't released yet?

[Hainish]

@jeremyn I'm not certain whether WebExtensions has the ability to do filename enumeration within a path, so it'll be hard to implement the HTTPSEverywhereUserRules/ path as we've done in XPCOM. You can create & remove custom rules through the UI, and I'm working on a way to migrate the existing custom rules to the internal extension storage before the WebExtensions release is deployed.

If using the external HTTPSEverywhereUserRules/ is part of your maintainer workflow, we may have to find an alternative.

[jeremyn]

@Hainish It is important to have a way to test the behavior of an actual XML ruleset file. If you can sketch out the relevant limitations from WebExtensions, maybe I can suggest alternatives. For example cut-and-pasting-and-editing an XML file in a textarea included in the UI could work.

[Hainish]

@jeremyn you can throw the following into the developer console for the extension:

all_rules.addFromXml((new DOMParser()).parseFromString('<ruleset name="Example.com"><target host="example.com" /><target host="www.example.com" /><rule from="^http:" to="https:" /></ruleset>', "text/xml"));

To help with getting a ruleset into the compact format you need to run that command, I've added a flag to merge-rulesets.py: --source_dir. With this, you can throw rulesets into a custom directory, and it will generate a default.rulesets file there with the compacted rulesets.

[Hainish]

I've released testing version 2017.7.21.1338 - get it at https://www.eff.org/files/https-everywhere-test/index.html. You won't be auto-updated to the latest version, unfortunately.

[jeremyn]

@Hainish I'm sorry I haven't gotten back to you since your comment in https://github.com/EFForg/https-everywhere/issues/9958#issuecomment-315202693. I was able to get a new test rule to work although the process is significantly rougher than before.

I'm noticing problems with rule changes made with the debug console not taking effect after I've visited a site. For example if I visit a site, add a rule, then revisit the site, the new rule doesn't do anything. Similarly if I add a rule, visit the site, modify the rule, then revisit the site, the modification doesn't take effect.

[J0WI]

What is the future of the Fennec (Firefox for Android) version?

[Hainish]

@jeremyn I'm looking into the problem you've reported now.

@J0WI the Fennec version is lagging behind in WebExtensions support. Last I checked, it did not support various UX elements that are built into the current Fennec XPCOM version. It does still perform redirects, though. So for some time the WebExtensions version will suffer reduced user control, until we're able to rebuild the UX.

[Hainish]

@jeremyn this is due to the targets being cached. You can clear this by adding one additional line:

all_rules.addFromXml((new DOMParser()).parseFromString('<ruleset name="Example.com"><target host="example.com" /><target host="www.example.com" /><rule from="^http:" to="https:" /></ruleset>', "text/xml"));
all_rules.ruleCache = new Map();

[jeremyn]

@Hainish I can confirm that the new Map() line lets me change a custom ruleset for a page I've already visited, thanks.

[jeremyn]

I wish we had some way of checking with users generally but speaking just for myself, I almost never interact with the HTTPS Everywhere UI in Firefox for Android. So as long as the core functionality is there, that's fine with me for now.

[jeremyn]

@Hainish Another functionality that the WebExtensions version misses is this: if I paste a URL into the address bar, but don't press enter/visit the URL, the add-on UI will show what ruleset matches that URL and whether that ruleset is enabled. One use case is when I'm reviewing an updated version of an existing ruleset, I might copy the new version of the ruleset into the HTTPSEverywhereUserRules/ directory with a different ruleset name and then paste one of the URLs in both the old and new versions of the ruleset into the address bar. I can then disable the old version of the ruleset without visiting the page or worrying about interactions with the new ruleset.

Is this functionality something we can add in?

[Hainish]

@jeremyn I've never encountered this functionality, and we're unlikely to add it - I haven't seen that anywhere in the code.

[jeremyn]

@Hainish To be clear, it's not a tricky or obscure thing. To reproduce: paste freerangekitten.com into the address bar and then observe that Freerangekitten.com shows up as a rule in the add-on UI.

Maybe this happened "naturally", without explicit coding effort, due to how Firefox processes the matching logic for the legacy add-on.

Anyway it's pretty useful and I'm sorry it won't be carried over.

[J0WI]

btw. Mozilla released Firefox Nightly 57 this week. I can confirm that HTTPS Everywhere stopped working there.

[J0WI]

btw: I played a bit around with GitHubs new project board:
https://github.com/EFForg/https-everywhere/projects/1

[ghost]

@J0WI Can you upload developer tools log to Gist?

[Hainish]

@J0WI this is excellent, I had no idea this existed! This probably isn't strictly a replacement for Trello, if we decide to have that as well, since each "project" would be a single card on the Trello board.

[jeremyn]

@Hainish The "level" of each card (issue vs project) is arbitrary, there's no reason why you can't have multiple GitHub Projects boards at different levels.

It would be nice for us to track all of the user-driven projects out there using GitHub Projects, basically any of the existing issues with checklists such as https://github.com/EFForg/https-everywhere/issues/11203, https://github.com/EFForg/https-everywhere/issues/10882, or https://github.com/EFForg/https-everywhere/issues/10843.

I would prefer GitHub Projects over Trello due to GitHub Projects being significantly more visible to people looking at this GitHub repository.

[Hainish]

@jeremyn I was just pointing out that this is conceptually different from Trello's stories, since they are more high-level and not necessarily tied to any issue. I'm aware that these can be any arbitrary level we want, I'm not prescribing any specific tool for this purpose.

[Hainish]

As of #11800, the Firefox addon is an Embedded WebExtension.

I would like to deploy this early next week (Monday or Tuesday) as version 2017.8.14 or 2017.8.15. This will be the same version on both Firefox and Chrome, and we will only have one signed tag, rather than two.

To add custom rules, you can still add them to your HTTPSEverywhereUserRules path, and subsequently change the extensions.https_everywhere.webextensions-migrated pref to false so that the extension knows to run another migration.

I hope that the next few days will shake out any remaining bugs before deployment. I will not be available for the rest of the week, so next opportunity to address them will be next week.

To allow the maximum amount of time for users to restart their browsers, we will be deploying the fully WebExtensions addon shortly before the cutoff date. This means that the nightly versions and then the dev versions will stop working before we make the switch. This is an unfortunate but necessary trade-off.

[pwd-github]

Could you not have the full WebExtension version available on the developer channel at the point you make the switch to the embedded WE in order ensure that Nightly users are able to retain the benefits of HTTPS Everywhere? That's what a bunch of other developers are doing.

[ghost]

@sabret00the Nightly is not intended for everyday use. It is used by many people though.

[gitarra]

@koops76 Nightly _is_ meant for everyday use.

How do you imagine Firefox QA is done?

As a matter of fact Mozilla is currently aggressively promoting Nightly as a daily driver for power users as they try to gain marketshare back.

[Hainish]

@sabret00the we have had a very unofficial development channel for testers of the WebExtensions version. This isn't a bad idea, and will let our developers at least install a version of HTTPS Everywhere that they can use. I've created an issue for it: https://github.com/EFForg/https-everywhere/issues/11805

[mcomella]

FYI: it was announced that tomorrow's Firefox Nightly will disable legacy extensions by default (assuming no unforeseen problems).

I asked a relevant follow-up question on Mozilla's IRC:

Regarding legacy WebExtensions being disabled by default on Nightly, if an extension is legacy (and thus disabled) but updates to a non-legacy extension, will it be automatically re-enabled?"
i believe so, unless it has been explicitly disabled by the user
what andym says is correct
```

[ghost]

@gitarra Nightly is a testing version.

[gitarra]

@koops76 Thanks for letting me know! I thought I was on the ESR channel. /s

Jokes aside:

If you are a power-user ... you should use Nightly (ideally as your main browser but you can also use it alongside Firefox on the release channel or another browser).

https://wiki.mozilla.org/Nightly

I guess thats official enough?

[ghost]

@Hainish How can I get the version for Firefox that is already completely WebExtensions-based? I would like to test it with Nightly.

[ghost]

@gitarra OK, I was wrong, Nightly used to be a testing version.

[Bisaloo]

@koops76 https://www.eff.org/files/https-everywhere-test/index.html or compile it yourself from the webextensions branch and flip the pref in nightly to enable unsigned addons.

[Hainish]

@Bisaloo this is not an officially supported dev channel, but yes it's possible to install that on >= 57.

[J0WI]

Years ago, we had a dev version (with auto update) for Firefox on EFF.org to beta test some rules.

Edit: Somebody found that dev version: #11819

[stevenwdv]

Would it really not be possible to add an option to use HTTPSEverywhereUserRules for storage? The current UI lacks support for exporting rules and easily importing them as we were able to do with HTTPSEverywhereUserRules. Synchronizing on starting the extension (or after disabling and enabling it via the UI) would be helpful.

[ghost]

@stevenwdv Not possible. Extensions can't access file system anymore.

[J0WI]

Please see #10822 for the custom rules discussion.

[mgol]

@Hainish

To allow the maximum amount of time for users to restart their browsers, we will be deploying the fully WebExtensions addon shortly before the cutoff date. This means that the nightly versions and then the dev versions will stop working before we make the switch. This is an unfortunate but necessary trade-off.

Please release the purely-webextension-based version some time before the Firefox 57 release, though. Mozilla Add-ons has a review process that's not instant. I imagine the number of version bumps in this period may be larger than usual which may enlarge review times. It'd be good to submit it early enough so that it passes the review before the Firefox 57 release.

The version hosted on your own site could be updated later, of course.

[Hainish]

Thanks @mgol, that was our plan as well.

[davidhedlund]

@koops76 Is HTTPS Everywhere a hybrid WebExtension yet?

[ghost]

@davidhedlund I think so.

[gloomy-ghost]

@davidhedlund I see your comment at decentraleyes as well. You can simply check weather there is a hasEmbeddedWebExtension key available in the install.rdf to see if it is a hybrid WE addon

[davidhedlund]

Excellent answer @gloomy-ghost!

[Hainish]

Development version 2017.9.12.1337 has been released: https://www.eff.org/https-everywhere#developers_note

[J0WI]

Next week, Firefox 57 will ship to beta.

[ghost]

@Hainish Are we ready for WebExtensions?

[Hainish]

We've been ready for quite some time. We are allowing the maximum window for people who have upgraded to the Embedded WebExtension from the XPCOM extension to restart their browsers. Full WebExtensions will launch in late October or early November, as I've already explained here. Our timeline has not changed.

[Hainish]

For reference, the Firefox release schedule is available here: https://wiki.mozilla.org/RapidRelease/Calendar.

Firefox Stable will make the transition to 57 on November 14th. Transitioning to full WebExtensions in late October should give us enough padding to fix any issues if they arise as well as allow time for AMO review.

[J0WI]

What about the open ToDos for Webextensions?

[jeremyn]

Seconding @J0WI 's question about the remaining open issues. For reference, they are https://github.com/EFForg/https-everywhere/issues?q=is%3Aopen+is%3Aissue+label%3Awebextensions-firefox and many of them are serious.

[Diapolo]

Firefox Quantum 57.0b3 (64-Bit) is now on my machine, sadly HTTPSEverywhere is not yet ready :-/.

[strigona-worksight]

@Diapolo You can install the developer version which is WebExtension here: https://www.eff.org/https-everywhere

[Diapolo]

@strigona-worksight That is working fine, thank you. 2 more questions, will I now stay on the developer version or will that be upgraded to release version once finished? I had 2 versions listed in parallel, the old non-WebExtension one (which I removed manually) and the new developer version, why?

[pwd-github]

I'm not sure if it's been noted, but if you upload the WebExtension to AMO and set the minimum version to 57, users that don't have 57 won't be upgraded until their browser updates to 57.

[Hainish]

@pwd-github we plan to do this regardless, but doing so would shorten the time that the embedded WebExtension has the opportunity to migrate users settings.

[FranklinYu]

Not sure whether it helps, but AMO provides beta channel.

[mgol]

I'm not sure if it's been noted, but if you upload the WebExtension to AMO and set the minimum version to 57, users that don't have 57 won't be upgraded until their browser updates to 57.

@pwd-github we plan to do this regardless, but doing so would shorten the time that the embedded WebExtension has the opportunity to migrate users settings.

@Hainish How so? Users of Firefox <57 should get an update to the latest supported version, shouldn't they? Only Firefox 57+ users (currently beta & nightly) would get the update to a WebExtension version and those are versions where the embedded WebExtension wouldn't work anyway.

[J0WI]

The first full WebExtensions release is just around the corner: #13420.

[Hainish]

Full WebExtensions release has been made. Thank you for all your hard work in fleshing out the bugs missing features. There's still some work to be done to replace some of the features that we lost when moving from XPCOM. This can be tracked here: https://github.com/EFForg/https-everywhere/projects/1