Https-everywhere: Discussion about out-of-band ruleset updates

[jeremyn]

(@Hainish, @cowlicks:)

There's a code branch https://github.com/EFForg/https-everywhere/tree/sign-rulesets that is intended to allow ruleset updates out-of-band from regular version updates. The idea is that HTTPS Everywhere will request ruleset updates from a central EFF server periodically with the effect that rulesets will update from time to time without the version year.month.day changing.

Privacy Badger does something similar for a whitelist it uses. Recently there were some problems with the updates which lead to a frustrating experience for me, see https://github.com/EFForg/privacybadger/issues/1466. After I understood what was happening, I argued against it, see for example https://github.com/EFForg/privacybadger/issues/1466#issuecomment-315792144 and the related discussion.

I think this is also a bad idea for HTTPS Everywhere and I'd like to understand why this change is being made. I would at least like to avoid the user experience of things cycling between breaking and working for no apparent reason that I had with Privacy Badger.

[ghost]

I am not sure why do you ask to not update rulesets outside of extension update mechanism. The current update mechanism is secure enough to not be the weakest link. The weakest link is the browser extension update mechanism, since Chrome uses SHA-1 and Firefox uses both SHA-1 and MD5 (!!!) for extension update verification, while we use SHA-256. I guess this may be closed. Thanks for letting others hear your opinion though.

[ghost]

uBlock Origin and Privacy Badger both update their data out of the band, I am not sure if anyone sees this as a problem. Maybe the extension "phoning home" may be a concern, but that can be fixed by out of band updates being optional, regular extension updates still including the default ruleset library. This "phoning home" is also not a concern for Tor Browser users since Tor anonymizes the update downloading.

[jeremyn]

@koops76 My concerns aren't about the crypto, it's about the surrounding user and maintainer experience. You can read through https://github.com/EFForg/privacybadger/issues/1466 to get an idea of what I mean.

In any case I would specifically like to hear from the EFF why they are doing this so I understand the motivation.

[Bisaloo]

@jeremyn, I know you are mainly waiting about the EFF answer but if I may chime in:

I understand and I agree with your concerns. On the other hand, separating the rulesets updates from the core updates has several upsides in my opinion, which would still make this change a positive move for our users.

Right now, when a ruleset fix is pushed on master, we have to either:

• rush an update because it concerns a high traffic website (https://github.com/EFForg/https-everywhere/pull/9110)
• tell the users to wait a month for this change to reach them (and we know this is problematic because it's common to have reports saying "X is broken" when a fix has been pushed for weeks but hasn't been released yet)

In short, I think this could be a net gain depending on how frequent the rulesets updates would be. So I guess my follow-up question is: how often do you plan to push the rulesets updates @Hainish?

In any case, this change and this behaviour should be made very clear at different places (changelog, add-on description, FAQ, etc.) to alleviate some of the concerns raised by @jeremyn.

(As a broader point, I think it would be really helpful if you kept the community in the loop when you are making big changes like this one. Not necessarily to argue but at least to inform so we can prepare for what's coming. This was already an issue in https://github.com/EFForg/https-everywhere/pull/10538.)

[cschanaj]

So I guess my follow-up question is: how often do you plan to push the rulesets updates

AFAIK, we make a release every 2 weeks with exceptions when there is blocking PRs. The benefit of out of band updates is not obvious unless we are pushing much more frequent than that.

Unlike uBlock, HTTPSE and PB are privacy focused extensions. We should avoid phoning home while leaving the users unnoticed (as they are privacy/ security conscious). IMHO, out of band ruleset updates should be made on request (require user interactions) or on a opt-in basis.

Besides, we rarely make core code changes. @Hainish How often are we going to make core releases when we have out of band ruleset updates? We should make the (ruleset & core code) release policies clearly documented.

@jeremyn thanks for bringing this issue to the discussion!!

[ghost]

Unlike uBlock

uBlock Origin is a privacy focused extension, since blocking ads also decreases the amount of tracking and there are also lists (some enabled by default) specifically made to block trackers, even if they are not showing any ads.

[ghost]

Besides, we rarely make core code changes.

Exactly the reason why out-of-band ruleset database updates are desired.

[ghost]

IMHO, out of band ruleset updates should be made on request (require user interactions) or on a opt-in basis.

Privacy Badger already pings to EFF for updates and I'm not sure if it can be disabled. I guess this is OK since if the user does not trust EFF they would have not installed the extension. I'm not sure if the option should be opt-out, or opt-in, with the extension asking the user when first installed. Opt-out seems better since the user may close the tab the extension opens on startup without paying any attention to its content, resulting in them not getting the ruleset updates quickly.

[Hainish]

There are a number of reasons for out-of-band ruleset updates.

1. Lax security of AMO updates
   A year ago today, Ryan Duff posted about a vulnerability discovered accidentally in the way Mozilla does key pinning in Firefox. This would allow anyone granted a misissued certificate for addons.mozilla.org to update extensions arbitrarily. Given the broad permissions traditional extensions are granted, this is a serious issue that is mitigated in part with the XPCOM deprecation. But it's a troubling sign that the security of AMO was circumvented by pure happenstance, and does not engender confidence in the Mozilla addon update infrastructure.
2. Waiting for code review
   On the topic of AMO, we have in the past had to wait up to a couple of weeks before a new release is reviewed and published. In communication with Mozilla, we've been able to have them prioritize review, and this time span has shortened. But occasionally we'll observe that a new version will again be taking a long time for review. Of course, this is not a problem for the self-hosted version. But if there is a critical site that is in need of an update, the process of fixing it is made more difficult.
3. Reliance on AMO, Chrome Webstore, Opera Addons
   This is related to but not the same as the above. The release process is simply made more difficult and cumbersome by having to upload to the various browser addon stores every time a release is made. This is also the case for the self-hosted Firefox version, since a signature is required by Mozilla before posting. It makes the release process much more laborious.
4. Keys we control
   Given (1), Tor Browser is readying itself to disable the update channel for extensions.
   In order to ensure ruleset updates are reliably delivered, we have to move to an out-of-band delivery channel. With the sign-rulesets branch merged, updating the rulesets will be possible using keys that we control. This will rely on our own infrastructure and keys that are pinned in the extension itself.
5. Multiple update channels
   Moving in this direction also makes the extension more modular. With update_channels.js, we allow the extension to specify multiple update channels. This will facilitate, for example, an intranet to deliver another rulset bundle to its constituents without publicly divulging secret FQDNs. This will also allow the Tor Browser to deliver its own ruleset bundle, with its own pinned key, via HTTPS Everywhere. This possibility is powerful for the reasons Paul Syversion, Tor creator, describes in this whitepaper.
6. Possibility of granular updates in the future
   Currently, the rulesets comprise the vast majority of the extension download size. Of the 1.7M compressed xpi, a mere 421K is left to everything else - mostly translations. This means that delivery of the rulesets will be roughly equivalent to delivery of the extension itself, currently. But this transition opens up the possibility of delivering granular ruleset updates to extensions, which will vastly lower the total size of downloads in a unit time.

In the future we may also expose the update channel to users, so that they themselves can compile ruleset bundles. But this will require us weighing the costs and benefits of adding an additional maintenance burden.

[jeremyn]

@Hainish Thanks for the detailed response.

I would say that my concerns could be met by adding the following:

• Inform users of the auto-update mechanism and allow them to enable/disable it. The Observatory UI is an example of what this might look like.
• Allow users to reset the rulesets to the "stock" set that came with their current version of the add-on.
• Version the ruleset updates and display this version to the user somewhere.

Optionally:

• Publish information somewhere about the EFF infrastructure responsible for serving updates, to give confidence about its reliability. Maybe a status page/dashboard would be good here.

The motivations for all of these are, I hope, obvious, but I can elaborate if needed. None of these should interfere with the reasons you've given.

Also, to repeat, my experience with similar functionality in PB was that the update target (the whitelist) was repeatedly and quietly wiped. If we move to this update model for HTTPS Everywhere then I consider it a more-than-theoretical possibility that the rulesets could be similarly wiped or broken. The above items give users the ability to lock down or reset HTTPS Everywhere to a known-good version.

[Hainish]

@jeremyn

Also, to repeat, my experience with similar functionality in PB was that the update target (the whitelist) was repeatedly and quietly wiped. If we move to this update model for HTTPS Everywhere then I consider it a more-than-theoretical possibility that the rulesets could be similarly wiped or broken. The above items give users the ability to lock down or reset HTTPS Everywhere to a known-good version.

This bug was due to the fact that the whitelisted resources in Privacy Badger were not being verified before being applied. When the website had a bug, the whitelist would return a blank result. In this unfortunate edge-case, Privacy Badger happily applied this whitelist, thus causing the extension to malfunction.

This will not happen with the ruleset updates, since every ruleset update is cryptographically signed by the extension developers, then verified in each users' extension _before_ being applied.

Given this, in what case would you envision the auto-update mechanism causing a fail case so large that we should expose extra options to the user? If privacy is the issue, we should keep in mind that already the users browser pings the extension store every day to check for new updates. EFF has a more stringent privacy policy than AMO or Google, so that shouldn't be a concern.

[Hainish]

This being said, I do think you're right in this:

Version the ruleset updates and display this version to the user somewhere.

[Hainish]

The ruleset updates will already have a timestamp associated with them, when applied. This is to keep track of whether we need to update the rulesets upon subsequent checks. We can simply use this timestamp, formatted as YYYY-MM-DD, as the version that is displayed to the user.

[jeremyn]

The problem isn't with a specific bug, it's about the process. It's like your criticism of Mozilla in "Lax security of AMO updates": a bug was found, Mozilla fixed it, but now you're uncomfortable with their update process. Same thing here, right? A bug in the EFF update process was found, the EFF fixed it, but now a reasonable person could be uncomfortable with the EFF's update process. This reasonable person should have the ability to disable or revert EFF updates.

For privacy, it's not about the privacy policy. It's about giving users control over outbound network requests and software updates. To quote myself from https://github.com/EFForg/privacybadger/issues/1466#issuecomment-315843564:

...there is an endless list of software creators who want to build in quiet, uncontrollable update processes to their software. They always think they are justified. Users complain because they feel a loss of control over their system and because it causes unexpected breakage. This is no different. The EFF heavily criticized Microsoft for this sort of thing during the Windows 10 rollout.

[HTTPSNowhere]

@Hainish

There's still the issue of the need of a fallback mirror; For example, in China EFF infrastructure is blocked there, so people with HTTPS Everywhere installed there can't benefit from this feature. Is there anyway to add some fallback mirror to for example directly from Github (which is thankfully not blocked there)?

[cschanaj]

@HTTPSNowhere I guess that Github is also blocked in China... see https://en.wikipedia.org/wiki/Censorship_of_GitHub#China

[HTTPSNowhere]

@cschanaj

No, it is no longer blocked.

[ghost]

@HTTPSNowhere Is CloudFront blocked in China?

[HTTPSNowhere]

@koops76

No (I'm not in China but it's known that CloudFront isn't blocked there, and it's the type of domain front that works with Tor there = i.e. meek-amazon in Tor Browser).

Also using CloudFront may seem like a waste when there are Github releases that are free and already offer "unlimited" bandwidth.

[ghost]

@HTTPSNowhere GitHub won't be happy if we would use it as a free CDN.

[HTTPSNowhere]

@koops76

It's only for fallback downloads, which will affect only a small percentage of total users for which EFF infrastructure is blocked. But if they're happy with using CloudFront for all users then that's as great as well.

[Hainish]

@HTTPSNowhere the fallback mirror is a good point. I expect if we simply register a new single-purpose domain to serve the rulesets from, this will be allowed through the firewall, but redundancy is always a good thing. We could just serve this from an Amazon S3 instance which is not blocked in China.

[ghost]

@Hainish We may use S3 directly instead of CloudFront, but that will be much more expensive than using CloudFront over S3. Better use Google Cloud Storage.

The advantage is that if the URL is similar to https://s3.amazonaws.com/EFForg/rulesets.json the only way to block it is to block the entire domain s3.amazonaws.com.

[ghost]

We may also use Google Cloud Storage which has global edge caching and has the same domain for all buckets, making blocking specifically HTTPS Everywhere ruleset updates difficult. It may also be cheaper than S3. Google is blocked in China. GCS isn't currently blocked but it likely will be in the future.

[ghost]

@HTTPSNowhere We will need a CDN anyway. We're talking millions of downloads for each ruleset update.

[HTTPSNowhere]

@Hainish

Thanks for stating that a new domain will be chosen exclusively for this task, I thought initially that it would be a subdomain for eff.org.

[jeremyn]

Assuming my suggestions in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330706469 for enabling/disabling this update functionality are not implemented and users are required to accept these out-of-band updates, what's the intended behavior if the requests for updates always fails? An example might be users behind a firewall that simply drops outbound requests it doesn't recognize.

[jeremyn]

More concerns:

What about users with very limited or expensive bandwidth? There may be users who don't want to use or can't afford even tiny amounts of extra bandwidth. Currently these users can get all updates offline, if someone else downloads the entire extension to a disk and then shares the disk with them.

Regarding my UI suggestions, unfortunately it seems creating a UI for WebExtensions in Firefox for Android is especially challenging (see @Hainish 's comment here: https://github.com/EFForg/https-everywhere/issues/9958#issuecomment-317824367). so maybe it won't be possible to implement my UI suggestions there even if we wanted to. I suppose mobile users are also the most likely to be sensitive to bandwidth availability and price.

Also, what about users in areas who might be endangered by visiting eff.org or a known mirror? They should be able to turn off these requests.

[ghost]

@jeremyn I don't think we should approve third parties to supply updates through physical media. Even if the rulesets are signed the media may contain malware in addition to the rulesets.

[ghost]

@jeremyn Re. the third point: These users should not install HTTPSE if it's too dangerous. We shouldn't have others' safety depend on how well we hide our requests.

[ghost]

Overall, I suggest multiple mirrors for people who don't want their update requests to go directly to EFF.

[jeremyn]

@koops76 You can download HTTPS Everywhere directly from the EFF at https://www.eff.org/files/https-everywhere-latest.xpi with right-click "Save Link As". In fact it might be the same person who downloads the .xpi in one location and then takes the disk with them to another location to install on other systems. In this case the only parties involved are the EFF and the user. HTTPS Everywhere is also distributed indirectly through Tor, Tails, and Debian, all of which can also go on physical media.

For

These users should not install HTTPSE if it's too dangerous.

I don't think most users expect HTTPS Everywhere to make outbound calls so they might not be prepared for it to do this. I was surprised when I found out that PB does this.

HTTPS Everywhere is recommended at https://techsolidarity.org/resources/basic_security.htm, which is aimed at what I think is an important use case: a less-technical activist or journalist, one who possibly has had their computer/phone/internet security configured by a technical person in a one-off setting. Take a look at this Hacker News discussion regarding the Tech Solidarity link I gave, in particular the comment from Thomas Ptacek (tptacek) that reads in part:

These instructions are written for unsophisticated users, particularly journalists and activists, and were written with feedback from those users....

We're simultaneously working with the airport lawyer groups (there's a huge one at ORD). It's been jarring to realize how many compromises are required to make things workable for groups of non-experts to use. Just getting software installed is a major hassle, so anything you install or customize needs to be really worth the effort.

It is not reasonable to assume that these less-technical end users will understand and manage the risk involved with their phone contacting eff.org when they take it with them into for example a war zone.

I can't speak for the EFF and I can't say for certain whether it endorses this class of user for HTTPS Everywhere. I strongly believe that it does, or at least gives the impression that it does (see https://ssd.eff.org/en/module/how-circumvent-online-censorship), and if so, it needs to take this use case into account.

[ghost]

I suggest to obfuscate the requests behind innocuous domains provided by cloud services.

Examples:

• s3.amazonaws.com
• storage.googleusercontent.com
• raw.githubusercontent.com
• cdn.rawgit.com

[ghost]

@jeremyn Solution: do not go into a war zone?

[HTTPSNowhere]

@jeremyn

I think @Hainish took the best decision when he said,

the fallback mirror is a good point. I expect if we simply register a new single-purpose domain to serve the rulesets from, this will be allowed through the firewall, but redundancy is always a good thing. We could just serve this from an Amazon S3 instance which is not blocked in China.

So there's no worry about someone trying to connect to *.eff.org since ruleset updates will be fetched from an entirely different domain, and if they fail they'll be downloaded from a fallback mirror at s3.amazonaws.com.

@koops76

For what it's worth, googleusercontent.com and rawgit.com are blocked in China.

[Bisaloo]

On a related note, will it be possible to route ruleset updates through Tor? I seem to recall that the SSL observatory had this option.

[cschanaj]

Using CDNs as a redundancy could be dangerous if EFF infrastructure was block by local governments. This is because local governments may request CDNs service providers to collect IP addresses (through legal actions) for various "legal reasons", to be allowed to enter the market.

Routing updates through Tor could be a remedy. (Otherwise, the updates should fail AS-IS).

[jeremyn]

@HTTPSNowhere You make a good point that requests to domains like s3.amazonaws.com should be less obviously suspicious than requests to eff.org.

On the other hand, any unusual activity can be used to fingerprint or highlight users. I have no idea what non-suspicious internet traffic looks like in places like rural Afghanistan or South Sudan but I guess there are places where even requests to s3.amazonaws.com can flag users as suspicious. It may also be that mirror A is normal in one place but mirror B isn't, with the reverse being true somewhere else. Maybe all the popular sites in one place use Amazon as a CDN, but all the popular sites in another place use Google instead.

Also I agree with @cschanaj that mirrors are at risk of legal pressure. I mean, rawgit.com was suggested, but apparently rawgit.com is run by just one person. He might be a swell guy but that's obviously a huge single point of failure. That's an extreme example, of course.

I suppose it is possible to set up mirrors in a way that minimizes risk to users, but I think that would require some careful thought and actual research, not just speculation like we're all doing here.

I don't understand the suggestions to use Tor. Are people arguing that we should include a mini-Tor client in HTTPS Everywhere? Or are people talking about just the use case involving the Tor Browser?

[Bisaloo]

I don't understand the suggestions to use Tor. Are people arguing that we should include a mini-Tor client in HTTPS Everywhere? Or are people talking about just the use case involving the Tor Browser?

As far as I remember, there was an option in the SSL observatory to route the traffic through Tor if it was set up on the computer.

I am not saying this would completely solve the issues you are raising but it could help in some cases and I think it would be nice to have as an option.

[Bisaloo]

Other than that. I personally like the out-of-band updates but I fully agree that an offline option is necessary as well for the various reasons brought by @jeremyn.

I am not sure which should be the default though.

[jeremyn]

@Bisaloo I found this ( https://tor.stackexchange.com/a/1879 ) regarding HTTPS Everywhere and Tor from @diracdeltas :

When you enable SSL Observatory, we POST a copy of the certificate chain you saw to an EFF server, along with the time, server domain, and server IP address. We don't keep logs of your IP address, but for extra anonymity, we try to send the POST request through Tor if you have Tor installed. You don't have to be running Tor browser for this to happen.

So you already have to have Tor installed and running, it doesn't entirely come with HTTPS Everywhere.

[ghost]

In WebExtensions, to make a request over a proxy, you would need a native application installed separately on the user's system.

One example of an extension that needs a native application to work is DNSSEC Validator.

[ghost]

@jeremyn There are places where ANY request to foreign websites is suspicious, let alone using Tor, so your argument is not valid. S3 is often used by many websites to host files. Not sure why a request to S3 can be suspicious.

[jeremyn]

@koops76 As you say, "There are places where ANY request to foreign websites is suspicious...". HTTPS Everywhere needs to be very careful before forcing customers to make these requests.

[ghost]

@jeremyn They're screwed either way, to install HTTPSE they would have to access Google, Mozilla or EFF servers.

[jeremyn]

@koops76

they would have to access Google, Mozilla or EFF servers.

The user can download the extension while they're somewhere safe as I described before and then disable all updates while they're somewhere less safe.

I'm not really sure where you're coming from in this discussion. The point I'm making at the moment is something like, "Unexpected and uncontrollable web requests can potentially put some users at risk, so HTTPS Everywhere shouldn't do that." Do you entirely disagree with that? Do you think the UI suggestions I made wouldn't fix it? Do you think the potential risk is so small that it's not worth the added complexity? It's okay if we disagree but I'm not sure what we are disagreeing about, exactly.

[ghost]

It's still possible to detect the redirects HTTPS Everywhere makes.

[ghost]

Need a code example?

[jeremyn]

@koops76 "Need a code example?" No, I can imagine what you mean.

It might help to recognize what I'm talking about as a form of user fingerprinting. Just because it's possible to attack/observe a user in one way doesn't mean HTTPS Everywhere should go out of its way to make it easier to do that in other ways. For example the EFF talks about steps people can take to defend against browser fingerprinting at https://panopticlick.eff.org/about#defend-against even though a complete defense is effectively impossible against a really determined adversary.

[Hainish]

I'll remind participants in this discussion to maintain a respectful tone with regard to other contributors and maintainers. The tone of this thread has veered at times to be less than respectful, and this is a project that necessitates an atmosphere where everyone feels welcome, regardless of difference of opinion or background.

We shouldn't prescribe use cases for HTTPS Everywhere users, but assume that all sorts of populations around the world will be using the extension. This includes populations at risk, and be mindful that these at-risk populations are located in globally disparate locations. We should also assume that there is a point of diminishing returns when considering the different scenarios users may face, and at some point make a decision given our best judgement. This is all we can do without rabbit-holeing into a discussion of calculated risk and which group deserves more attention vis-a-vis another at-risk group. At some point you just need to make the call based on the available information.

On a related note, will it be possible to route ruleset updates through Tor? I seem to recall that the SSL observatory had this option.

In the past, we've detected if Tor was already running on a system before making the option available to enable submission of certificates through Tor. This entailed first attempting a SOCKS request on port 9150, which is the Tor daemon running in Tor Browser. As a fallback, we've tried 9050 to see if a system Tor process was running.

As certificate introspection is not (yet) available within WebExtensions, we don't have this code included in HTTPS Everywhere. It is important to keep in mind that use of Tor is in most cases detectable, unless the user is using a pluggable transport which specifically tries to obfuscate traffic analysis. This itself may put users at risk.

Choosing a single-purpose domain for ruleset distribution, with a fallback, seems the most reasonable path at the moment. The usage of HTTPS Everywhere is already fingerprintable. It has a massive effect on browser traffic patterns which is trivial to detect. So the fact that the domain has a single purpose, to distribute rulesets, places our users in no additional risk. The fallback option has to be dealt with internally, pursuant to our privacy policy, so that any CDN that we use will be able to see the IP addresses of our users. We have to ensure that they aren't irresponsible with this data, which is a matter of policy.

[ghost]

@Hainish I think the primary server should be CDN, if that's blocked, the extension should make a request to a non-CDN server.

[jeremyn]

@Hainish Could you please specifically respond to my suggestions in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330706469 to let users enable/disable/reset out-of-band updates? If that's simply not an option then I'd like to know.

[Hainish]

@jeremyn my comments will have to wait until next week. We don't even have a launch date for this feature so comments here should not be time pertinent.

On September 22, 2017 6:12:34 PM PDT, Jeremy Nation [e-mail] wrote:

@Hainish Could you please specifically respond to my suggestions in
https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330706469
to let users enable/disable/reset out-of-band updates? If that's simply
not an option then I'd like to know.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-331595736

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[jeremyn]

@Hainish Okay, that's fine.

I wish this feature had been discussed with the community before substantial work had been done. It may have set a better tone for this discussion and issue https://github.com/EFForg/https-everywhere/issues/12087 ("Use Ed25519 instead of RSA for ruleset signatures"). A simple issue a few months back like "We're thinking about delivering ruleset updates outside of the normal browser update systems, with a switch over in . Here are some reasons why. We already do something like this in PrivacyBadger. Thoughts?" Then people can have their say with the proper expectations and at least feel heard even if you ignore specific suggestions in the end.

[HTTPSNowhere]

I think there are some misconceptions that should be cleared out: Concerning that the CDN may cooperate with some government to hand out the IPs of those who use HTTPS Everywhere: this doesn't make sense, since as mentioned the use of HTTPS Everywhere is already fingerprintable, and there's nothing spectacular about using it. In fact, this didn't even prevent the Tor Project from bundling a pluggable transport that relies mainly on Amazon AWS (meek-amazon) which is the one that is mainly used in China, despite that the usage of Tor is more dangerous and suspicious.

[jeremyn]

I agree that HTTPS Everywhere can be fingerprintable but I don't concede that because it's fingerprintable by some means, we should give up trying to hide its presence entirely.

I suppose the normal attack involves watching what requests a user makes after a user visits a page that is known to be covered by a complex HTTPS Everywhere ruleset. If the user makes requests that are consistent with HTTPS Everywhere rewrites, then the user has probably installed HTTPS Everywhere, otherwise they probably have not. Please correct me if I misunderstand the argument.

However suppose that the monitor is trying to retroactively detect its presence, and the ISP only has a deduplicated, non-protocol-linked list of domains the user visited. Maybe the ISP has limited resources for storing user traffic, or maybe the ISP is a public VPN service which deliberately keeps limited logs. In this case it may be impossible to detect HTTPS Everywhere's presence, _unless_ one of those domains is something obvious like effcdn.org.

I also disagree that HTTPS Everywhere's presence is always entirely unremarkable. You could track a person's location if you can detect HTTPS Everywhere, they have HTTPS Everywhere installed on a mobile device, and they are going into an area where almost no one else is going to have it.

The key here is really how much informed choice the user has about controlling these updates. If the user is informed about these updates, they can shut them off, and (ideally) they are opt-in, then the EFF can go all out supporting it. They can configure update channels using every major CDN service, configure updates through Tor with a variety of plugins, everything. Then the user can decide what works best for them. On the other hand, if the user is forced to accept these updates with no choice and is not informed about what's going on, the EFF needs to exercise extreme caution and act in a way that is safe for literally every conceivable user. (And I believe it is impossible for the EFF to exercise enough caution in this case.)

[HTTPSNowhere]

@cschanaj

Also it's easier for authorities to monitor DNS requests for the first domain than to block that domain and then ask the fallback mirror's CDN provider to provide logs. Also such thing is basically useless, what do you gain from learning that someone uses HTTPS Everywhere?

[ghost]

@HTTPSNowhere

You could track a person's location if you can detect HTTPS Everywhere, they have HTTPS Everywhere installed on a mobile device, and they are going into an area where almost no one else is going to have it.

Except of this, nothing. Good idea @jeremyn.

[cschanaj]

My argument is that the use of third party CDN needed to be justified.

AFIAK, there is little information an adversary can gain. It is like the situation when websites claiming that they are collecting anonymous data and nothing personal will be collected. Since people are using their service and thus they should trust them, but people still want to block the tracking (especially the third-party ones).

When it comes to privacy, different people might have their own concerns. IMHO, falling back to a third party CDN when EFF domains are not accessible might not be a good idea.

P.S. You can also track the user habit, possible time-zone/ occupations of a https-everywhere user by observing the _regular pings_ to the EFF domains. Here I assumed that some users do quit the browsers when they finish everything and they have quality internet connections, such that disruption to the pings are mostly because of the user habits.

P.S.2 There is limited support of https-everywhere on tablet/ phone, https-everywhere users are almost certainly using Firefox/ Tor despite whatever the user-agent header tell. This will also expose the existence of another extension installed which spoof user-agent. :smile:

[ghost]

@cschanaj EFF servers may not handle such load without CDN. It will take about a few terabytes of data per month.

[ghost]

@cschanaj I use Chrome, not Tor Browser or Firefox, so you're wrong.

[cschanaj]

@koops76 I am not aware that it is possible to install HTTPSE on Chrome on an Android/ iOS devices. My apologies if I am wrong.

[ghost]

@cschanaj It's possible to install it on Firefox for Android (Fennec).

[ghost]

I suppose that HTTPS Everywhere will - if this is implemented - only download diffs and not the entire rulesets from start, is this correct? (If there's diffs then the download size can be very small)

[cschanaj]

AFAIK, defaults.rulesets is not really diff friendly in its raw format.

6.7M default.rulesets-2017.09.12.json
6.8M default.rulesets-2017.10.4.json

running

diff -u default.rulesets-2017.09.12.json default.rulesets-2017.10.4.json > hello.patch

14M hello.patch

Unless

cat default.rulesets-2017.09.12.json | python -m json.tool > pretty.rulesets-2017.09.12.json
cat default.rulesets-2017.10.4.json | python -m json.tool > pretty.rulesets-2017.10.4.json

diff -u pretty.rulesets-2017.09.12.json pretty.rulesets-2017.10.4.json > hello1.patch

174K hello1.patch

using diff has a huge potential but I am not really sure how this is going to be implement in the extension.

P.S. I am not sure whether or not it is possible to implement diff in javascript. So this is just for the reference on the diff obj size as compared to the actual ruleset.

[jeremyn]

One way to prepare a minimal update would be to check the git history for changes to ruleset files.

[Hainish]

@I-have-50-open-PRs-to-HTTPS-EVERYWHERE:

I suppose that HTTPS Everywhere will - if this is implemented - only download diffs and not the entire rulesets from start, is this correct? (If there's diffs then the download size can be very small)

This is the eventual goal, but that's not how it's currently implemented. I agree this would be a big win, being that compared to the entirety of the rulesets, the diff is quite small. If the browser hasn't checked for updates recently (for instance if it hasn't been opened for a while) we could provide a mechanism to make available all previous diffs, performing an incremental migration.

The application of diffs can be difficult, though a cursory search reveals that there are potentially promising libraries for this purpose.

@cschanaj:

P.S. You can also track the user habit, possible time-zone/ occupations of a https-everywhere user by observing the regular pings to the EFF domains. Here I assumed that some users do quit the browsers when they finish everything and they have quality internet connections, such that disruption to the pings are mostly because of the user habits.

This is no different than the daily ping to addons.mozilla.org, though. I don't envision us pinging any more often than daily.

[Hainish]

@jeremyn

I wish this feature had been discussed with the community before substantial work had been done.

If I had, then someone would most likely have wished I had a proof of concept was made available before asking for input. At any given point I'll be working on several features, I don't think it's reasonable to halt development and not work on a feature just because there may be feedback.

[zero77]

@Hainish
Is this currently being implemented or is it still just an idea being debated, which may or may not be added.

[ghost]

@zero77 https://github.com/EFForg/https-everywhere/tree/sign-rulesets

[jeremyn]

@Hainish

At any given point I'll be working on several features, I don't think it's reasonable to halt development and not work on a feature just because there may be feedback.

That depends on how useful you expect community feedback to be, and how much ownership you think the community should have over features.

[Bisaloo]

@Hainish, I've seen assumptions about this in other threads but we haven't heard any official word on this yet.

Do you plan on eventually allowing users to download rulesets from alternative sources or is this just a mechanism to simplify ruleset updates for clients which will remain EFF-only?

Because this could potentially be a good opportunity to revive projects like https://github.com/chris-barry/darkweb-everywhere.

[ghost]

@koops76 @HTTPSNowhere @cschanaj For the CDN issue I think using a p2p distribution approach using IPFS: https://ipfs.io/ may be a great solution to those problems. In addition IPFS can't be censored by blocking DNS https://ipfs.io/blog/24-uncensorable-wikipedia/ There's a JS implementation of IPFS: https://github.com/ipfs/js-ipfs
But I don't think this should be given top priority currently.

[Hainish]

@Bisaloo the plan is to publish the rulesets on a publicly accessible endpoint, with a signature. So yes, anyone will be able to download the rulesets and implement them in their own software. This includes downstream dependencies such as Brave or anyone else.

[Hainish]

And check that they are valid by checking the signature, too.

[Bisaloo]

@Bisaloo the plan is to publish the rulesets on a publicly accessible endpoint, with a signature. So yes, anyone will be able to download the rulesets and implement them in their own software. This includes downstream dependencies such as Brave or anyone else.

Sorry, I wasn't clear. My question is: will users be able to download rulesets from an alternative source?

In other words, is this a solution to https://github.com/EFForg/https-everywhere/issues/4380?

[jeremyn]

To expand on @Bisaloo 's question, this means that:

• HTTPS Everywhere would need to allow for alternate sources, and
• the providers of alternate sources would need to be able to sign their rulesets so HTTPS Everywhere allows them (either using a pre-installed key, or allowing users to add their own key), or
• HTTPS Everywhere would need to allow users to turn off the signature check for some sources

The apt/apt-key architecture used by Debian/Ubuntu is a potential model here.

[Hainish]

@Bisaloo my thought is that at first, we would have the update channels hard-coded into the extension. The way it's currently implemented, update channels is an array, but it only contains one entry: the canonical rulesets. If Tor wanted to add a separate ruleset channel when they include HTTPS Everywhere within Tor Browser, they could do so by modifying update_channels.js at build time.

In time, I think it would be wonderful if we provide a UX so that users are able to add update channels and signing keys for those channels themselves. So ideally, someone could create a "Darkweb Everywhere" or similar ruleset and a signing key, and have users subscribe to it.

[Bisaloo]

Related: https://trac.torproject.org/projects/tor/ticket/2161

[brainwane]

@Hainish said,

At any given point I'll be working on several features, I don't think it's reasonable to halt development and not work on a feature just because there may be feedback.

@jeremyn said,

That depends on how useful you expect community feedback to be, and how much ownership you think the community should have over features.

This is a pretty important sub-conversation. I think there's a possible middle road here involving earlier heads-ups as well as clear signals to everyone involved about what phase a decision's in (e.g., "super open to fundamental change" versus "let's figure out how to implement this well").

Towards that goal, I've posted a tentative architectural/feature roadmap to the HTTPS Everywhere mailing list that I hope project maintainers will comment on; the items marked "Potential" in particular would probably benefit from replies on that list, or in relevant issues in this repo, on whether they'd be useful and implementation concerns/suggestions.

[diracdeltas]

I am the ex-maintainer of HTTPSE and now a maintainer of Brave, which is a downstream user of HTTPS Everywhere rulesets. I just wanted to say that it would be pretty useful for us to have the HTTPS Everywhere rulesets available separate from the extension itself on an endpoint with some level of trustworthiness, ranging from "valid TLS cert on eff.org" to "code signed by offline HTTPS Everywhere signing key". My current update process is: wait for Bill to email out about an extension update, verify that there are no breaking changes to ruleset format, download the XPI, unzip, extract rulesets, re-package for Brave. Compared to other upstream lists like Tracking Protection, HTTPS Everywhere is relatively difficult to auto-update for a downstream project like Brave.

[jeremyn]

It seems to me the main advantages of out-of-band updates are to allow more frequent ruleset updates and to let users choose to consume different ruleset feeds. From @diracdeltas 's comment it sounds like Brave just wants a more convenient way to consume the official rulesets at the same pace as the regular release cycle. We could solve that problem within the regular release process, for example @Hainish could separately package whatever @diracdeltas normally extracts and put a signed version of that up on the EFF website. The more complex update process proposed here isn't needed for that.

[Hainish]

After careful consideration I've decided to incorporate a few sub-features based on contributor feedback:

1. Delivery of rulesets from a non-EFF domain. In situations where EFF is censored, this may prove useful. Of course, the censorship regime could also block the ruleset domain as well. However, even the GFC is rather spurious in what domains it chooses to block, and it is more likely that the specific high-profile domains hosting politicized content will be blocked, rather than single-purpose functional domains. In the future, we can build out the functionality to have multiple update channels, as a back-up if one is blocked.
2. Toggle for OOB ruleset delivery. This should default to _on_, to ensure the maximum number of people are able to receive ruleset updates in a timely manner. However, some users may wish to turn auto-updates off. This toggle can be built into the options ui.
3. Use the RSA-PSS ciphersuite. RSASSA PKCS1 v1.5 is weak, even in the signing context. WebCrypto supports a cipher that we should preference - RSA-PSS. Let's use this, at least until WebCrypto incorporates Ed25519.

With these improvements, a relatively simple update mechanism can be added which addresses all the considerations I've listed above. Over time, we can build out a more comprehensive ux for the update channels, as I noted above.

Newcomers to this issue: you can track the progress of this feature in the sign-rulesets branch.

[Hainish]

Closing this for now, since I've incorporated feedback from this discussion. Happy to have more feedback, though!

[jeremyn]

Some of my points were not addressed, such as those in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330706469 (versioning, reset-to-stock, status dashboard) and my concern in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330997935 about what happens if updates are permanently unavailable for a particular user.

[Hainish]

@jeremyn updates will not cease to be available through extension updates. These will continue to be delivered for the foreseeable future. I suggest opening separate issues for feature specifics.

[jeremyn]

@Hainish I opened this issue to have a discussion here, and the comments I linked are over three months old now. I understand wanting to wrap up an open-ended discussion but I brought those points up very early on. It would be great if you could give at least high level answers to those concerns here. If you implement those features and I have questions/suggestions for the implementation, I can make new issues then.

[Hainish]

@jeremyn I have addressed these issues, repeatedly in comments.

See https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330711860, https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-358424399, as well as having implemented the ruleset toggle you mentioned in https://github.com/EFForg/https-everywhere/tree/sign-rulesets. Again, if you have specific requests on the development of this branch, let's move that to individual issues.

[jeremyn]

@Hainish You did mention your support for versioning in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330711860, thanks for pointing that out. Also I suppose that for the user whose updates are permanently broken, the implied fix is to wait until the next full extension update. You haven't, as far as I know, addressed the reset-to-stock or status dashboard ideas, but those aren't really that important.

You keep mentioning the https://github.com/EFForg/https-everywhere/tree/sign-rulesets branch. Are you expecting contributors to monitor that branch and jump in if they have questions? Is that instead or, or will there also be, an announced alpha/beta release with enough time allowed to get and incorporate feedback?

[Hainish]

@jeremyn For reset to stock, I think that can be built out as part of a more comrpehensive UX that I mention in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-344444263. I envision this as being dashboard-like, if I take your meaning correctly.

Incorporating the sign-rulesets into a beta release seems like a good idea, and relates to the email sent by @brainwane over the list. It does open broader issues of beta channel release overhead, which I'm hesitant to introduce. For now, the sign-rulesets branch should be considered on a track for feature inclusion, and I would encourage anyone interested in this feature to check the branch out and play with it. I've also created a new label, https://github.com/EFForg/https-everywhere/labels/sign-rulesets, and I'm adding it to this issue now.

[jeremyn]

@Hainish So, my main concern with the functionality of this feature is to avoid the negative experience I had with the similar feature in Privacy Badger, which I discussed in my earliest comments in this discussion. I didn't know the feature existed in Privacy Badger, was surprised to find out it was contacting the EFF for updates, things cycled through breaking and working, and I couldn't stop, reset, or even know when these updates were happening. I'm not really concerned about the inclusion of any one specific subfeature, but rather that the unified whole is robust to failure, even in hostile networks, and that problems can be avoided, or fixed automatically or fixed by very low technical users. There's no mock/whiteboard/design document for what you are implementing so it's difficult to discuss it at that level, so instead we've been talking about individual subfeatures, which misses the point.

(I'm not saying you need to produce a design document or anything, I'm just trying to explain one reason why this conversation has been perhaps a little awkward.)

I probably won't track an active development branch in order to give comments. Maybe other contributors will do that. I think you would find it irritating for people to do that, most developers don't want to get bug reports on stuff they are actively coding. I understand what you mean about the burden of making a beta release, particularly in this case where you have no guarantee that anyone will review it even if the beta release is packaged beautifully. I'm not sure what to suggest about that. Maybe you could build a week or two gap in the middle of the development cycle for sign-rulesets, where you'll have something stable-ish for people to review if they want to, but also plan to work on something else during that gap so the time isn't wasted if no one responds.

[Hainish]

@Hainish So, my main concern with the functionality of this feature is to avoid the negative experience I had with the similar feature in Privacy Badger, which I discussed in my earliest comments in this discussion. I didn't know the feature existed in Privacy Badger, was surprised to find out it was contacting the EFF for updates, things cycled through breaking and working, and I couldn't stop, reset, or even know when these updates were happening. I'm not really concerned about the inclusion of any one specific subfeature, but rather that the unified whole is robust to failure, even in hostile networks, and that problems can be avoided, or fixed automatically or fixed by very low technical users.

The reason for that failure, as well as why it doesn't affect this branch, was explained over four months ago: https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-330711605. sign-rulesets was never affected, since we have had a robust process for ruleset verification since day one. In Privacy Badger, no verification was present at all, afaik. I did not work on that feature, and the underlying cause of that bug is absent here. So a bug in another project I do not work on seems hardly relevant here. It is an EFF project, but there are a different set of people working on the security-focused HTTPS Everywhere and the privacy-focused Privacy Badger.

After implementing the sub-features in https://github.com/EFForg/https-everywhere/issues/12606#issuecomment-358424399, I think the most sensible thing to do is to solicit reviews, with instructions on how to build and test from sign-rulesests. If no one responds, that will be that - all I can do is extend the invitation. I wouldn't be blocking my time on this feature - in any case, I'd be working on other things.

[jeremyn]

@Hainish Where Privacy Badger failed, for me, wasn't in whether it verified downloads, but in that it didn't tell the user what was happening and gave them no tools to fix problems. If you want to talk about how you're avoiding the Privacy Badger problem, the discussion should be about UX/UI improvements, not about cryptographic signing, which is good but not really the issue.

If you had a phone, and one day you discovered the phone has been bricked, and you learn it's because some update failed, would you be upset at the vendor because of the update bug, or would you be upset because there was a way for your phone to be bricked without warning and with no way to recover it? Probably the second, and you wouldn't really care about the update bug at that point.

That sounds reasonable about announcing a review. If no one responds, there's not much you can do. Is there, or will there be, a ruleset endpoint available for testing?

[wonderchook]

@jeremyn from a user perspective you are suggesting it be clear how to turn the feature on and off, plus having some sort of indication of an error if things fail?

[jeremyn]

@wonderchook Yes, pretty much. Turn on or off, show the current version, report errors in a friendly way, and allow users to roll back updates.

[Hainish]

@jeremyn I'll consider the best way to balance user controls and clear design for this PR. Notably, I've mentioned that certain UX controls will have to be added later, as we build out this feature. Others (such as display of the ruleset version and source) may make sense for initial rollout.

[Hainish]

https://github.com/EFForg/https-everywhere/tree/sign-rulesets now includes versioning information for rulesets in the popup.