Https-everywhere: Cultivate a checklist to prepare for next full-fetch-test run

[Hainish]

Type: other

There was a fair amount of discussion and work to be done after the last run of the full fetch test, which disables rulesets if we observe problems on any of their hosts - see https://github.com/EFForg/https-everywhere/pull/10538.

A list of tasks were formulated to perform before the next run of the test. These take the form of both one-time tasks that will generically help our workflow, as well as tasks that should be performed before each run. I'm sure I'm missing something, please help me cultivate this list.

One-Time Tasks

• [x] Instead of disabling the entire ruleset, remove individual hosts (#1998)
• [x] The report that is generated when a ruleset is disabled (beside the Disabled by https-everywhere-checker because text) should include a timestamp
• [ ] Run comment-removal script (#11963)

Before Each Run

• [ ] Update the cipher list in https://github.com/EFForg/https-everywhere/blob/c7da7b7eb704a697cc3d44e406c1912e45b954a9/test/rules/src/https_everywhere_checker/http_client.py#L48-L49 with ciphers from an updated version of Firefox ESR
• [ ] Create an issue (or send an email) announcing that there will be a forthcoming run of the full fetch test
• [ ] Before merging, from HEAD on master do a git diff --name-only <base commit> (see https://github.com/EFForg/https-everywhere/pull/10538#issuecomment-310465613)

[cschanaj]

We should clean up the obsolete comments (if any) after each run. See also #11963, which has been pending review for some time. Currently, round 200+ rulesets are having obsolete comments.

P.S. If we are going to have a new checker which remove individual hosts instead of disabling the entire ruleset, #11963 need not to be merged. However, please run the script in #11963 before the next fetch-test. thanks!

[cschanaj]

In https://github.com/EFForg/https-everywhere/pull/10516#issuecomment-313884825, it is suggested that

One thing we may want to do is after every run of the fetch tester, create a list of the rulesets that would be deleted due to consecutive disables with the same reason, and then manually review if they actually should be deleted.

Any chance that an official such list will be available after the next full-fetch-test?

[Bisaloo]

Didn't you also update the ruleset-whitelist.csv format to be able to use it in full-fetch-test?

https://github.com/EFForg/https-everywhere/issues/10227

[Hainish]

@cschanaj currently there is no automated mechanism for deleting rulesets that have been disabled for the same reason after subsequent runs. The deletion tool has to be created before we submit the results of those deletions for review.

[Hainish]

@Bisaloo with the creation of ruleset-whitelist.csv, we are now whitelisting rulesets from the fetch test the that are expected to fail the fetch test, and whitelisting rulesets from the coverage test that are expected to fail the coverage test.

This should help future runs of the full-fetch-test tool, since we will no longer be whitelisting rulesets from fetches that are only expected to fail the coverage test.

Either way, the full-fetch-test tool should be able to use the new whitelist format OOTB, since it is not touching that whitelist directly and is relying on internal tools included in this repo to do that work.

Does this answer your question?

[Bisaloo]

Yes, I'm sorry, I should have read the code first. I didn't realise that full-fetch-test was using regular tests from this repo.

[pipboy96]

@zoracon @andresbase @Hainish What's the current status of this?

[zoracon]

@pipboy96 I'm planning on running a fetch test this fall.