Fenix: Pageload experiment: -O2, -Oz build flags

[liuche]

Why/User Benefit/User Problem

Switching to the -O2 flag has pageload perf improvements, but increases apk size by ~6MB.

Need this experiment to measure tradeoffs.

What/Requirements

• [x] Stability experimentation on 3.2 1/22-1/27 w/ GV 74 Nightly
• [ ] A/B test on Firefox Preview Release (pending Phase 1): 1/27-2/14 w/ GV 73 Beta
• [ ] Decide which version of GV to use for upcoming releases
• [ ] Update GV version in AC right before 3.3 goes out

This experiment would be run in staged rollout through Google Play, because Mako doesn't have this capability. Johan is assisting with this.
Corey from the Data Science team is involved in analyzing the results for retention.

Acceptance Criteria (how do I know when I’m done?)

Data Science has results that allow us to make a perf decision.

Meeting notes

[yoasif]

Don't want to spam here, but I think this is a no-brainer - if it actually results in pageload performance, a larger apk size pays for itself in terms of marketshare and people willing to try the browser because of the page load performance.

[liuche]

https://crash-stats.mozilla.org/report/index/c0845347-0e97-4402-ae61-6cb020200125

[liuche]

New crashes on 73b: https://crash-stats.mozilla.org/topcrashers/?product=Fenix&version=73.0b0

[ecsmyth]

Updated pre-experimental impact analysis. Initial phase 2 experiment results expected o/a Feb 18

[ecsmyth]

Initial analysis of phase 2 data found no statistically significant difference in crash rates and a statistically significant improvement in page load performance.

[ecsmyth]

Plan for phase 3

[liuche]

Update here: We'll be rolling out 4.2 with GV Beta 75, which does not have the experiment enabled.