Kuma: T - Create a performance test harness to assess mobile performance.

[tobinmori]

Acceptance Criteria

• [x] the test harness should allow testing of Kuma optimization variants (e.g. with or w/o bcd, css, js) and output a performance benefit grade (this grade can later be used to weigh cost/benefit of implementation)

[peterbe]

Ideas for variants:

• inline the critical CSS and lazy load the rest of the <link rel=stylesheet>s
• remove the BCD signalling (CSS + JS)
• replace SVGs in HTML with <img src="png">
• replace SVGs in CSS with PNGs
• remove auth modal JS + CSS code
• inline CSS inside iframe without lazy loading the rest

Crazy ideas, because why not:

• Remove serialized JSON state inside DOM and replace it with an external fetched .json file
• Ditch the webfonts (including the preloading!)
• No React JS bundle

[schalkneethling]

replace SVGs in HTML with

While working on some tangentially related work I have identified a number of these instances where we should move these to be background images added via CSS.

The main reason we want an SVG embedded in the HTML vs. as an img source or via CSS is so that we have all of the styling power SVG and CSS provides. One of the other reasons was to avoid some HTTP requests.

The latter is not that much of an issue for us so, the latter is the main reason.

For a lot of our SVG iconography, we really do not need much more than positioning and setting dimensions. As a result, these can all be moved to CSS.

replace SVGs in CSS with PNGs

For most(all?) of these cases the bits difference between the PNG and SVG will probably be minimal. The big benefit to having these be an SVG is that it is scalable and as such, SVGs display so much better on high-dpi monitors.

You can get the same with PNG but, you have to manage two versions of the asset and then load one for "normal" displays and the high-dpi version for high-dpi displays. As such, I would be careful with this one 😄

inline CSS inside iframe without lazy loading the rest

I really like this idea with the interactive example iframes. I reckon this is something we should just do 👍

Ditch the webfonts (including the preloading!)

Nice idea for an experiment. I also do feel that we can get down to only one custom webfont and that is ZillaSlab Bold which is then only used for headings level 1-5.

[atopal]

Some other ideas:

• remove unnecessary CSS
• remove interactive examples

[peterbe]

Progress

Here's a mid-sprint update...

I have all the tooling collected here: https://github.com/peterbe/mdn-harness
It's pretty nice because nothing is manual on specific basis. Suppose the the Array.forEach pages in prod, you just need to run yarn run download ..., yarn run generate-variants ..., s3cmd ....

CloudFront is a bit special though. I manually created each and every CloudFront distribution under my personal AWS account by painstakingly making sure they all get the same settings.
When I was done I jotted down all the URLs here: https://github.com/peterbe/mdn-harness#current-variants
The names are a bit cryptic and kinda requires you to open up the scripts: https://github.com/peterbe/mdn-harness/tree/master/src/variants and map it yourself.

At the time of writing here's the list:

• baseline
• inlinecss
• nobcdsignalling
• ssrprism
• inlinecssextreme
• lazyloadiframe
• noreact
• inlinecssiframe
• nodatauris

@atopal Once we're a bit more confident about the variants we want to play, we can start putting these into SpeedCurve for synthetic testing. If you think it helps, feel free to put them in now already.

At the time of writing I'm waiting for a massive "Visual Comparison" report in WebPageTest but it's not finished yet. MIght take a while. In case it helps, here's the test URL.

Results

The README in the repo has notes about all the commands but the tl;dr; is that I ran:

lighthouse-batch -f cloudfront-sites.txt --html --params "--only-categories=performance --preset perf --throttling.cpuSlowdownMultiplier=6"

which produces the following summary:

[
  {
    "url": "https://d2fm08353pu8pw.cloudfront.net/index.html",
    "name": "d2fm08353pu8pw_cloudfront_net_index_html",
    "file": "d2fm08353pu8pw_cloudfront_net_index_html.report.json",
    "html": "d2fm08353pu8pw_cloudfront_net_index_html.report.html",
    "score": "0.85",
    "detail": {
      "performance": 0.85
    }
  },
  {
    "url": "https://d2um11o9s382bw.cloudfront.net/index.html",
    "name": "d2um11o9s382bw_cloudfront_net_index_html",
    "file": "d2um11o9s382bw_cloudfront_net_index_html.report.json",
    "html": "d2um11o9s382bw_cloudfront_net_index_html.report.html",
    "score": "0.95",
    "detail": {
      "performance": 0.95
    }
  },
  {
    "url": "https://d3ohduisxjll24.cloudfront.net/index.html",
    "name": "d3ohduisxjll24_cloudfront_net_index_html",
    "file": "d3ohduisxjll24_cloudfront_net_index_html.report.json",
    "html": "d3ohduisxjll24_cloudfront_net_index_html.report.html",
    "score": "0.88",
    "detail": {
      "performance": 0.88
    }
  },
  {
    "url": "https://d1ca36dt8e8ima.cloudfront.net/index.html",
    "name": "d1ca36dt8e8ima_cloudfront_net_index_html",
    "file": "d1ca36dt8e8ima_cloudfront_net_index_html.report.json",
    "html": "d1ca36dt8e8ima_cloudfront_net_index_html.report.html",
    "score": "0.88",
    "detail": {
      "performance": 0.88
    }
  },
  {
    "url": "https://d1jqaxyu5i06vi.cloudfront.net/index.html",
    "name": "d1jqaxyu5i06vi_cloudfront_net_index_html",
    "file": "d1jqaxyu5i06vi_cloudfront_net_index_html.report.json",
    "html": "d1jqaxyu5i06vi_cloudfront_net_index_html.report.html",
    "score": "0.95",
    "detail": {
      "performance": 0.95
    }
  },
  {
    "url": "https://dzyoyiq5bbseg.cloudfront.net/index.html",
    "name": "dzyoyiq5bbseg_cloudfront_net_index_html",
    "file": "dzyoyiq5bbseg_cloudfront_net_index_html.report.json",
    "html": "dzyoyiq5bbseg_cloudfront_net_index_html.report.html",
    "score": "0.73",
    "detail": {
      "performance": 0.73
    }
  }
]

If I now reverse that back to the variants based on performance it's:

• baseline - 85
• inlinecss - 95
• nobcdsignalling - 88
• ssrprism - 88
• inlinecssextreme - 95
• lazyloadiframe - 73

Let the disclaimer and excuses being...

• In reality we run the Interactive Example iframe on a different domain. Can't do that here. But it's fair for every variant in a sense. Having to SSL negotiate and set up a second HTTPS thing for the iframe definitely costs but can't see that here.
• You can't completely delete all things related to BCD Signalling because its CSS is included in the main CSS bundle. So this variant just deletes the /static/build/js/react-bcd-signal.{hash}.js JS file.
• inlinecssextreme is dangerous and should be avoided. If there's CSS needed that depends on late actions by the user you might not have the CSS for it.
• In ssprism it just removes the work of Prism inside the massive react-main.{hash}.js bundle. But what this variant means is that there's nothing to do for the client-side Prism because all code snippets have already been converted.

[peterbe]

Here's the WebpageTest report
It's wildly unexpected and unpredictable. I don't trust it to make a comparison. I'd MUCH rather use SpeedCurve and do a bunch of samplings and see how they compare when you contrast their medians.

[peterbe]

Update It seems setting cache-control isn't working :(
I need to figure out why even though I set the Customize on each CloudFront's Object Caching and it doesn't appear to work.

[atopal]

Thanks for the update Peter. Do you have access to Speedcurve to set up what you need? Let me know if I can help with anything.

[peterbe]

Update I wrote a script called run_lighthouse-batch.sh which runs the lighthouse-batch command repeatedly and between each turn, it aggregates the numbers inside each report/lighthouse/summary.json. The numbers are surprisingly stable:

[
{
"url": "https://d2fm08353pu8pw.cloudfront.net/index.html",
"name": "d2fm08353pu8pw_cloudfront_net_index_html",
"file": "d2fm08353pu8pw_cloudfront_net_index_html.report.json",
"html": "d2fm08353pu8pw_cloudfront_net_index_html.report.html",
"score": [
"0.87",
"0.87",
"0.87",
"0.88",
"0.87"
],
"detail": {
"performance": [
0.87,
0.87,
0.87,
0.87,
0.88
],
"performance__count": 5,
"performance__average": 0.8719999999999999,
"performance__median": 0.87
},
"score__count": 5,
"score__average": 0.8719999999999999,
"score__median": 0.87
},
{
"url": "https://d2um11o9s382bw.cloudfront.net/index.html",
"name": "d2um11o9s382bw_cloudfront_net_index_html",
"file": "d2um11o9s382bw_cloudfront_net_index_html.report.json",
"html": "d2um11o9s382bw_cloudfront_net_index_html.report.html",
"score": [
"0.94",
"0.93",
"0.94",
"0.95",
"0.94"
],
"detail": {
"performance": [
0.93,
0.94,
0.94,
0.94,
0.95
],
"performance__count": 5,
"performance__average": 0.9399999999999998,
"performance__median": 0.94
},
"score__count": 5,
"score__average": 0.9399999999999998,
"score__median": 0.94
},
{
"url": "https://d3ohduisxjll24.cloudfront.net/index.html",
"name": "d3ohduisxjll24_cloudfront_net_index_html",
"file": "d3ohduisxjll24_cloudfront_net_index_html.report.json",
"html": "d3ohduisxjll24_cloudfront_net_index_html.report.html",
"score": [
"0.87",
"0.87",
"0.89",
"0.88",
"0.88"
],
"detail": {
"performance": [
0.87,
0.87,
0.88,
0.88,
0.89
],
"performance__count": 5,
"performance__average": 0.8780000000000001,
"performance__median": 0.88
},
"score__count": 5,
"score__average": 0.8779999999999999,
"score__median": 0.88
},
{
"url": "https://d1ca36dt8e8ima.cloudfront.net/index.html",
"name": "d1ca36dt8e8ima_cloudfront_net_index_html",
"file": "d1ca36dt8e8ima_cloudfront_net_index_html.report.json",
"html": "d1ca36dt8e8ima_cloudfront_net_index_html.report.html",
"score": [
"0.87",
"0.88",
"0.88",
"0.88",
"0.88"
],
"detail": {
"performance": [
0.87,
0.88,
0.88,
0.88,
0.88
],
"performance__count": 5,
"performance__average": 0.8779999999999999,
"performance__median": 0.88
},
"score__count": 5,
"score__average": 0.8779999999999999,
"score__median": 0.88
},
{
"url": "https://d1jqaxyu5i06vi.cloudfront.net/index.html",
"name": "d1jqaxyu5i06vi_cloudfront_net_index_html",
"file": "d1jqaxyu5i06vi_cloudfront_net_index_html.report.json",
"html": "d1jqaxyu5i06vi_cloudfront_net_index_html.report.html",
"score": [
"0.95",
"0.95",
"0.96",
"0.95",
"0.95"
],
"detail": {
"performance": [
0.95,
0.95,
0.95,
0.95,
0.96
],
"performance__count": 5,
"performance__average": 0.952,
"performance__median": 0.95
},
"score__count": 5,
"score__average": 0.952,
"score__median": 0.95
},
{
"url": "https://dzyoyiq5bbseg.cloudfront.net/index.html",
"name": "dzyoyiq5bbseg_cloudfront_net_index_html",
"file": "dzyoyiq5bbseg_cloudfront_net_index_html.report.json",
"html": "dzyoyiq5bbseg_cloudfront_net_index_html.report.html",
"score": [
"0.80",
"0.80",
"0.81",
"0.81",
"0.80"
],
"detail": {
"performance": [
0.8,
0.8,
0.8,
0.81,
0.81
],
"performance__count": 5,
"performance__average": 0.804,
"performance__median": 0.8
},
"score__count": 5,
"score__average": 0.804,
"score__median": 0.8
},
{
"url": "https://d1cap84gly6pqw.cloudfront.net/index.html",
"name": "d1cap84gly6pqw_cloudfront_net_index_html",
"file": "d1cap84gly6pqw_cloudfront_net_index_html.report.json",
"html": "d1cap84gly6pqw_cloudfront_net_index_html.report.html",
"score": [
"0.90",
"0.89",
"0.90",
"0.90",
"0.90"
],
"detail": {
"performance": [
0.89,
0.9,
0.9,
0.9,
0.9
],
"performance__count": 5,
"performance__average": 0.898,
"performance__median": 0.9
},
"score__count": 5,
"score__average": 0.898,
"score__median": 0.9
},
{
"url": "https://d29rjqiwb8121o.cloudfront.net/index.html",
"name": "d29rjqiwb8121o_cloudfront_net_index_html",
"file": "d29rjqiwb8121o_cloudfront_net_index_html.report.json",
"html": "d29rjqiwb8121o_cloudfront_net_index_html.report.html",
"score": [
"0.88",
"0.88",
"0.88",
"0.88",
"0.88"
],
"detail": {
"performance": [
0.88,
0.88,
0.88,
0.88,
0.88
],
"performance__count": 5,
"performance__average": 0.8800000000000001,
"performance__median": 0.88
},
"score__count": 5,
"score__average": 0.8800000000000001,
"score__median": 0.88
}
]

[peterbe]

This is the SpeedCurve "Site": Mobile Performance Test Harness

Here's the Favorite report

It's been set up to run a synthetic test once every 24h. So it'll take some time for the numbers to stabilize. (Note-to-self; How do I trigger a build of them all manually??)

[peterbe]

And the winner is...

Best Lighthouse median, after 5 runs:

1. 
   
   • 94 - inlinecss
   
   
2. 
   
   • 94 - inlinecssextreme
   
   
3. 
   
   • 89 - noreact
   
   
4. 
   
   • 89 - noiframe
   
   
5. 
   
   • 88 - nobcdsignalling
   
   
6. 
   
   • 87 - baseline
   
   
7. 
   
   • 87 - ssrprism
   
   
8. 
   
   • 87 - inlinecssiframe
   
   
9. 
   
   • 87 - nodatauris
   
   
10. 
    
    • 70 - lazyloadiframe
    
    

How

For each URL I ran lighthouse with these options: --only-categories=performance --preset perf --throttling.cpuSlowdownMultiplier=6 (mobile is the default in Lighthouse) from my laptop (macBook Pro)
Each run gives you a summary (example) which I then stored accumulatively and eventually took the numbers and ran mean and median on them.

Notes on variants

• The noiframe means no Interactive Examples which is a hugely popular feature.

• The inlinecssextreme is "extreme" because once it has extracted the critical (above and below the fold) CSS, it discards all external CSS links. So things that only visible/enabled after user action might not have CSS. With some imagination you can still make this work by doing some whitelisting. For example, the CSS used by the modal only matters if you clicked on "Sign in" or if the whoami deems you should display the mugshot.

• The noreact is also extreme because without React we don't get the header JS functionality, the Subscription stuff, or the BCD signalling functionality. Also, the BCD signalling JS code isn't actually React. It's vanilla JS but built with Webpack.

• The lazyloadiframe is terrible actually. It takes a screenshot of what the Interactive Examples iframe looks like and puts that into the HTML plus some JS code that, on document load, swaps the image for the iframe. This is causing a rather ugly flickering and the image probably doesn't match the device. Plus, the image might fool the user to think it's working/done but it's "fake".

• Just because you remove the /static/build/js/react-bcd-signal.{hash}.js file, there's still a bunch of CSS related to the BCD signalling that you have to pay for in the main CSS bundle. I couldn't easily refactor that out. At least the dino-happy.svg and dino-confused.svg don't get loaded. To iterate, in all other variants you have to download 3 extra .svgs (sum total ~5KB) even if you never click the blue "What are we missing?" thing.

• In ssrprism the <pre> blocks with code are turned in the HTML, in Node, and put into the index.html rather than letting some Prism client-side JS code do that conversion in the browser on load. However, the main JS bundle still has all the Prism code which I couldn't remove. It's still there making the JS heavy but it's never doing anything.

• I never made a variant that ditches all Web fonts. It's too extreme and disruptive. Also a degree of running out of time. There might be an opportunity to only load one of the web fonts. @schalkneethling might have more details.

Conclusion

The spread is almost disappointingly small. The differences diminish when you use a higher CPU device.

One thing I noticed throughout the testing was that the heat of the CDN makes a big difference. I noticed it when I ran the first load where almost all HTML and static assets are cold, the numbers are generally lower. MDN does a good job with its static assets but a terrible job with the HTML and there the latency to the CDN might be even more important because the browser can't start download stuff until the HTML has come in.

We can still append more variant experiments.

But for now, this is it. To discuss the effort it would take to actually implement these is in https://github.com/mdn/kuma/issues/6895

[peterbe]

Another interesting note is that if you take the baseline variant its total weight on disk is 1.2MB. But if you gzip each and every single file it comes down to 504.0 KiB.
Had you used zopfli instead of gzip it comes down to 492.0 KiB (2.4% smaller).
Had you used brotli instead of gzip it comes down to 448.0 KiB (11% smaller).

To do brotli on all HTML and static assets requires a more advanced and complex CloudFront integration. I would like to note that this is highly feasible to do in Yari 2.

Now, if you do that same comparison just for the index.html file:

• baseline - 150,585 bytes
• gzip - 34,645 bytes
• zopfli - 32,901 bytes (5% smaller than gzip)
• brotli - 21,181 bytes (39% smaller than gzip!!)

[atopal]

Hey @peterbe can you please add the "no interactive examples" option? I'd love to see results for that.

Done. --peterbe

[peterbe]

Note-to-self; It would be nice to try out to put the critical CSS into a <link rel=stylesheet href=_critical.css> just see how it behaves.

Here it is: https://dsxk913f42qzd.cloudfront.net/index.html

[peterbe]

Here's a WebPageTest visual comparison of baseline vs inlinecss vs linkcriticalcss