Https-everywhere: Full Fetch Test: Pending for March 2020

[zoracon]

Announcing to our ruleset maintainers that I plan to run this soon. So far, I have amended old config files and currently updating check_rules.py from the suggestions given from the last fetch test.

The tasks listed from the last time that are complete:

• [x] Instead of disabling the entire ruleset, remove individual hosts (#1998)
• [x] The report that is generated when a ruleset is disabled (beside the Disabled by https-everywhere-checker because text) should include a timestamp

Before Each Run

• [x] Update the cipher list in https://github.com/EFForg/https-everywhere/blob/c7da7b7eb704a697cc3d44e406c1912e45b954a9/test/rules/src/https_everywhere_checker
  /http_client.py#L48-L49 with ciphers from an updated version of Firefox ESR (https://github.com/EFForg/https-everywhere/pull/18964)
• [x] Create an issue (or send an email) announcing that there will be a forthcoming run of the full fetch test
• [x] Update certificate bundle https://github.com/EFForg/https-everywhere/blob/master/utils/create-platform-certs/README.md
• [x] Update User agent in http.checker.config #19004

Also, I wanted to be able to run this test and have it be reviewable for the ruleset maintainers when it is pushed. So I open to what can make PR's more reviewable in the future.

I am also going to track this in a project in the repo so that way progress can be tracked with fetch tests in the future and hopefully run this quarterly and not every 2-3 years anymore.

[J0WI]

We should review some of the failed top-* rulesets before submitting a new ruleset update. Probalby also for .onion rules, because the results might not be very reliable there.

[zoracon]

We should review some of the failed top-* rulesets before submitting a new ruleset update. Probalby also for .onion rules, because the results might not be very reliable there.

@J0WI Should I hold off on the ruleset update for this week? Or are you referencing the Fetch Test update?

[J0WI]

Should I hold off on the ruleset update for this week? Or are you referencing the Fetch Test update?

The scheduled release is not affected. The fetch test will cause a bunch of disabled rules and we probably do not want to ship a ruleset update to the API right afterwards that disables important rules for all users.

[J0WI]

The whitelist needs to be updated after each run. See also #18967.

[pipboy96]

@zoracon We also need to update the certificate bundle.

[zoracon]

@J0WI I am doing my best to figure out a reviewable fetch test result this time around. I don't expect the results the fetch test to be all patched at once, but I will likely spend time over the next month (or two) implementing the results with the rest of the maintainers here.

[cschanaj]

We might also want to update the user-agent of the https-everywhere-checker to reflect the current browser requirement before the next full fetch test. This prevent absurd fetch errors when some websites perform user-agent sniffing and return different content.

https://github.com/EFForg/https-everywhere/blob/bc3c6a94b98da1282acc2222562788e137347c52/test/rules/http.checker.config#L19

[pipboy96]

@cschanaj I think we should use current Tor Browser's user agent.

[cschanaj]

@cschanaj I think we should use current Tor Browser's user agent.

I agree it is a good idea to follow the Tor Browser configuration, a relatively up-to-date Tor Blog entry can be our reference: https://blog.torproject.org/browser-fingerprinting-introduction-and-challenges-ahead

[zoracon]

As an update, it seems like all steps are covered with updates and adjustments. Now I am running a soft test to adjust the new workflow for reviewable patches.

For context, the current git flow for the fetch tests runs git diff > /patch/diff.patch. I am adjusting the portion to utilize git format-patch.

Right now I am testing a utility that breaks up the patch by file.

That way I can write a script that runs git apply --check <patchfile> and then give over reviewable patches over to the maintainers and myself for subsequent review.

[zoracon]

Soft run complete, took about a full day, thinking about posting up the fully disabled rulesets in a PR so the actual fetch test rounds can just focus on the ones with targets disabled.

[pipboy96]

@zoracon That's a good idea! Looking forward to your PR. Also, can we fully delete rulesets if all targets return NXDOMAIN?

[J0WI]

18964 has introduced a regression in ruleset testing: #19034

[cschanaj]

blocked by #19330

[zoracon]

Closing this out since this process has been run, reviewed, and merged.
Postmortem here: https://github.com/EFForg/https-everywhere/wiki/Postmortem:-Fetch-Test-April-2020