Https-everywhere: Option to turn mixedcontent rulesets on

[Bisaloo]

How can we turn all mixedcontent rulesets on at once with the WebExtensions HTTPS-Everywhere?

The following command doesn't seem to work anymore:

extensions.https_everywhere.enable_mixed_rulesets;true

Related: #9958

[jeremyn]

@Bisaloo WebExtensions add-ons shouldn't count on having access to about:config, see https://wiki.mozilla.org/WebExtensions/FAQ#Will_I_have_access_to_about:config_or_the_preferences.3F . So it makes sense that HTTPS Everywhere isn't doing anything with that setting.

[ivysrono]

8506 enable mixedcontent on all platform by default.

[Bisaloo]

@ivysrono, that's not really the purpose of this issue. It is a great find and helps a lot with mixedcontent issue but it doesn't turn on HTTPS-Everywhere platform="mixedcontent" rulesets.

[jeremyn]

A more general version of this issue is a way to turn on any combination of platforms, not just mixedcontent, using a checklist.

[Hainish]

A lot of the rules that have been marked as platform="mixedcontent" were broken when the browsers first introduced MCB. I suspect many sites have fixed their mixed content problems in response to MCB being the default behavior now, but these rulesets should be manually checked rather than blanket-enabled, or else we risk blocking access to site resources. Currently there are ~1325 such rulesets:

user@https-everywhere ~/workspace/https-everywhere/rules (webextensions) $ git grep "mixedcontent" | wc -l
1325

I suspect that our tagging of these rules as mixedcontent is old enough that these rules should probably be reviewed anyway. In https://github.com/EFForg/https-everywhere/blob/master/CONTRIBUTING.md#general-info, I've made a note for those interested in contributing to prioritize the mixedcontent rules.

[jeremyn]

@Hainish I think this issue is to add the option to enable all the mixedcontent rules, not that we should enable them all by default.

[Hainish]

We could add an option to localStorage, as I note in this comment: https://trac.torproject.org/projects/tor/ticket/21323#comment:15

The platform="mixedcontent" attribute was always intended as a short-term solution, see https://bugzilla.mozilla.org/show_bug.cgi?id=878890#c20. I'm inclined to deprecate turning on mixed-content issues by default, since all browsers that we support are blocking mixed content or in the process of turning MCB on. Is there a practical benefit to enabling them all by default rather than ensuring we remove the mixedcontent flag from rulesets that still have it, if they aren't breaking?

[Bisaloo]

Is there a practical benefit to enabling them all by default rather than ensuring we remove the mixedcontent flag from rulesets that still have it, if they aren't breaking?

IMO, yes. I prefer enabling mixedcontent ruleset even if they trigger MCB.

I used to get over the broken parts here and there due to MCB and now, I don't even notice it because I am using the upgrade-insecure-content CSP.

I totally agree we should audit platform="mixedcontent" rulesets and see if we can turn them on by default for all users but I like having the option to turn them on even the ones that actually cause MCB.

[jeremyn]

This "implement a settings page" (in WebExtensions) mini-tutorial looks highly relevant. It uses the storage API, which it says is like localStorage but has advantages such as persisting even if the user clears their data. We should also make sure this setting persists in private browsing mode.

I don't personally enable mixedcontent rules by platform but we should see this feature as allowing users to enable any platform, not just mixedcontent, like I said in a previous comment. The compelling use case is to allow users in China to enable regional/China platform rulesets all at once.

[Hainish]

@jeremyn we're using the storage API for all persistence within WebExtensions currently. I was using localStorage as a synonym, which isn't strictly correct. The specific storage engine is either sync or local, depending on what is available.

For mixedcontent, my suspicion that problematic sites have fixed their issues when they saw their sites were broken, once the browsers started implementing MCB. Putting this to the test on a random sampling:

user@https-everywhere ~/workspace/https-everywhere/rules (webextensions) $ git grep -i mixedcontent | shuf | head -n 10 | cut -d":" -f1
Oversee.xml
AEUP.eu.xml
Docelu.pl-problematic.xml
DC_Power.eu.xml
MIMS.xml
DjurRattsAlliansen.se.xml
Roccat.org.xml
zongheng.com.xml
hitfile.net.xml
Poppy-Sports.xml

I get the following results:

Oversee.xml - automatically forwards to HTTPS
AEUP.eu.xml - no content shown on HTTP, invalid certificate on HTTPS (using old cipher)
Docelu.pl-problematic.xml - HTTPS endpoint does not exist
DC_Power.eu.xml - automatically forwards to HTTPS
MIMS.xml - no difference between HTTP and HTTPS (no mixed content)
DjurRattsAlliansen.se.xml - HTTPS endpoint does not exist
Roccat.org.xml - downgrades you to HTTP
zongheng.com.xml - HTTPS endpoint does not exist
hitfile.net.xml - no difference between HTTP and HTTPS (no mixed content)
Poppy-Sports.xml - invalid certificate on HTTPS (common name mismatch)

There are a lot of issues in this random sampling, but notably none of them are due to mixed content. Note that the rules that had certificate errors or no HTTPS are currently broken in the Tor Browser, for instance. To fix this and get rid of the mixedcontent flag altogether, we should

1. Gather a list of all the mixedcontent rules
2. Remove the mixedcontent flag from them, and
3. Disable all of those rules that break, using our fetch tester.

[jeremyn]

@Hainish Another angle is: here is a list of recent commits with platform="mixedcontent". The most recent one was 4 days ago. Active MCB still breaks sites in the wild so we can't assume that a mixedcontent warning is outdated. I would probably rather delete any given mixedcontent ruleset than enable it, if I had to choose.

[Bisaloo]

For mixedcontent, my suspicion that problematic sites have fixed their issues when they saw their sites were broken, once the browsers started implementing MCB.

I disagree here. I think websites that had a previously fully working https configuration and that saw their website break when MCB was first introduced indeed fix these issues.

But many websites, probably because they are in a slow process of implementing https (but as we saw in the case of Stackexchange, this can take years), still have mixed content issues. I encounter them in almost a daily basis.

[Hainish]

@Bisaloo I suspect many, if not the plurality, of sites that had problems with mixed content simply reconfigured their servers to remove HTTPS entirely. That is at least true for my random sample. Note that entirely half of the platform="mixedcontent" rulesets above were not accessible over HTTPS at all.

I'm also okay with, as @jeremyn suggested, removing these rulesets rather than doing the extensive testing to determine just how broken they are.

[Bisaloo]

• https://www.lemonde.fr
• https://blog.slate.fr
• https://www.20minutes.fr
• https://blog.linuxmint.com/

and so many more. I don't think a sample of 10 sites among 1256 can give us a solid idea of what is going on.

I also don't understand. You seem to be reluctant to keep mixedcontent rulesets because of the amount of work required to create and maintain those rulesets.

But I (and I expect many others) really want those rules, even if they cause MCB, and I will work on it. So the way I see it, the choice is quite simple:

• Keep mixedcontent rulesets, even those that cause MCB (but as I said earlier, I am all for an audit so that some of these rules can be turned on for everyone, without platform="mixedcontent") and please some users who prefer the extra security over usability.
  I also would like to add that this is the current situation with the legacy XPCOM addon. I don't see this issue as a new feature request, but simply a request to keep the new WebExtensions HTTPS-Everywhere on par with its predecessor.
• Remove mixedcontent rulesets. In this case, I will keep working on them on my end because I need them. But that's a bummer because no one else will be able to benefit from my hours of work to create such rulesets.

[jeremyn]

I agree with @Bisaloo 's latest comment in that there is no special reason to do anything with these mixedcontent rulesets right now. @Bisaloo wants an option to turn them on as a group in the WebExtensions add-on, and it's fine to say no to that request right now, but that doesn't mean that now we have to either purge or enable all these rulesets. Let's just leave them alone.

Also, mixedcontent rules are enabled by default for Tor users, so removing the mixedcontent rules will strictly reduce the security of Tor users.

[Hainish]

My intention is to simplify the codebase, given that all supported browsers do MCB, or are about to. But if a mixedcontent rule is problematic, obviously it should be removed (as I said in my previous comment.)

10 is a small sample size given the ~1325 mixedcontent rulesets, but that was just an initial pass which required a lot of manual work. 50% of these rulesets were not just problematic, but site-breaking. While it's possible that it's a less endemic problem within the entire set of mixedcontent rulesets, it's also possible that it's a greater problem. This is definitely a problematic situation which requires some kind of action, and I'm willing to concede that removing mixedcontent rules may not be the solution. I can add an option to enable mixedcontent rules in the addon for power-users, this is no problem - it will require going in to the debugging console instead of simply flipping a pref, though.

A better solution for problematic rulesets is probably similar to what we did two years ago - a massive fetch-test sweep of all rulesets, and disabling those that cause problems. This is likely to produce false-positives (even if we do multiple sweeps across different global network perspectives, and only disable those that fail all of them) due to edge-cases. In fact, this is what https://github.com/efforg/https-everywhere-full-fetch-test is meant to facilitate, but I never got around last year to running it. But this is a separate issue.

[ivysrono]

8755

[jeremyn]

I feel like I'm misunderstanding something. If a ruleset is marked mixedcontent then it is, by definition, problematic: it is too broken by MCB to enable by default. These rulesets also can have other problems besides mixed content, but this is true for any ruleset. This doesn't mean urgent action is needed, at least not urgent within the extremely limited resources available to the project.

[Hainish]

The attribute platform="mixedcontent" means that for browsers that do block mixed content, insecure content will be blocked on the HTTPS endpoint and result in disruption of the user experience in some substantive way. Conversely and necessarily, for browsers not blocking mixed content, an HTTPS site will be loading resources insecurely.

It should not cause problems in the way I stated above - namely, it should not fail a fetch test. This is what I mean by problematic, in context.

[Hainish]

This issue has been fixed, you can enable mixedcontent rulesets in the debugging console:

1. Navigate to about:debugging
2. Ensure "Enable add-on debugging" is enabled by clicking the check box to "on"
3. Click the "debug" button next to HTTPS Everywhere
4. In the console window that appears, type

let store = require('./store');
store.set({enableMixedRulesets: true});

You can then check with subsequent runs that mixedcontent rulesets are enabled by typing in the same console window:

store.get({enableMixedRulesets: false}, item => console.log(item));

[Bisaloo]

Thank you @Hainish for the addition of the feature!

I agree it would useful to run https-everywhere-checker at some point but I think it would be good if we sort out all the previous default_off='failed ruleset test' first.

[Bisaloo]

@Hainish, I can't get it to work and I am not sure why.

I followed your instructions and I checked that I indeed get

Object { enableMixedRulesets: true }

when I run

storage.get({enableMixedRulesets: false}, item => console.log(item));

even after restarting Firefox.

But mixedcontent rulesets are not enabled.

Here is a test url.

[Bisaloo]

I still can't get it to work with Version: 2017.7.21.1338.

Can anyone try to reproduce or reopen this issue if needed?

[J0WI]

@Hainish the workaround described in https://github.com/EFForg/https-everywhere/issues/9971#issuecomment-306637654 does not work anymore and it's not practical.
About 5% of our rules are currently disabled because of mixed content and another unknown number because of incomplete certificate chains (#6966). An option to enable additional rules could enhance coverage/security a lot for advanced users (or Tor browser?). Please consider to reopen this issue.

[jeremyn]

I'm not opposed to this option, but a potential problem is that users will enable mixedcontent rulesets, and then later see a bunch of broken pages, and not make the connection that enabling the option is what is breaking the pages. These users will then disable HTTPS Everywhere entirely, generate support requests, leave bad reviews etc.

We might take a different approach and add a sort of slider option for security, like Torbutton, with "Safest" the same as "Block all unencrypted requests", "Safer" as regular operation but with mixedcontent rulesets enabled, and "Standard" as what it normally does now.

[Hainish]

@J0WI I've updated https://github.com/EFForg/https-everywhere/issues/9971#issuecomment-306637654 to reflect how to enable mixed content rulesets with the new modular system.

@jeremyn I see how what you describe could be a problem, but I don't see how a security slider would be a solution. I don't think it's necessarily intuitive that raising the security level on a slider will break pages, and people may forget that they changed their security settings. In that case, the negative results you mention will still happen.

[Hainish]

Relatedly, in Firefox 60, they're introducing a pref to upgrade mixed content on secure pages, similar to how Upgrade Insecure Requests works. I've requested that this pref be exposed to WebExtensions so that we can enable it.

[jeremyn]

It is a continuum going from using standard rulesets, to using standard plus mixedcontent rulesets, to blocking everything, so the option is more naturally described with a slider instead of, for example, multiple checkboxes.

The difference between the Torbutton slider UI, and an equivalent simple radio button, is that the slider UI gives space to describe both the pros and (importantly) cons of each setting.

[J0WI]

Relatedly, in Firefox 60, they're introducing a pref to upgrade mixed content on secure pages, similar to how Upgrade Insecure Requests works. I've requested that this pref be exposed to WebExtensions so that we can enable it.

That's only for mixed passive content and it's already enabled by default, isn't it?

[Hainish]

That's only for mixed passive content and it's already enabled by default, isn't it?

Ah, looks like you're right. Thanks @J0WI.

[Hainish]

@J0WI I think the pref is only defaulted to true for nigthtly: https://bugzilla.mozilla.org/show_bug.cgi?id=1435733#c7

[J0WI]

@J0WI I've updated #9971 (comment) to reflect how to enable mixed content rulesets with the new modular system.

@Hainish I was not able to enable them. store.get({enableMixedRulesets: false}, item => console.log(item)); returns Object { enableMixedRulesets: true }.

[Hainish]

@J0WI that result indicates that it is in fact enabled.

[Hainish]

@J0WI ah, you're saying that they aren't enabled despite this result?

[J0WI]

@J0WI ah, you're saying that they aren't enabled despite this result?

Exactly.

[cschanaj]

See #15070