Sage: Benchmarking performance

[retlehs]

would like to see some stats on:

• asset build times
• WP load times

between sage 9 and 10 (maybe even 8, too). previously when we tried to add laravel mix to sage 9 (on webpack 3, not 4) the build times were worse.

let's get some stats together so that we can use them during our communication to the public about sage 10

• [x] I've read the guidelines for Contributing to Roots Projects

[kellymears]

I refactored a Sage 9 theme into Sage 10 and really only did what was necessary to get it working -- no auxiliary refactoring took place, which was hard for me. ⛳️

The starting point was the current mainline 10.0.0-dev branch. Controllers are now composers, and I built a service provider in order to provide one of the composers with a directly ported navwalker. Everything else is the same. It also feels exactly the same. But I benched it -- three rounds for each.

Sage 9

Bench 1

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: | 20.846 seconds |
|Complete requests: | 1000 |
|Failed requests: | 0 |
|Keep-Alive requests: | 0 |
|Total transferred: | 9198002 bytes |
|HTML transferred: | 8671000 bytes |
|Requests per second: | 47.97 [#/sec] (mean) |
|Time per request: | 208.461 [ms] (mean) |
|Time per request: | 20.846 [ms] (mean, across all concurrent requests) |
|Transfer rate: | 430.89 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
|Connect: | 108 | 164 |152.9 | 134 | 1389 |
|Processing: | 31 | 42 |20.6 |38 | 211 |
|Waiting: | 30 | 41 |20.6 |37 | 201 |
|Total: | 142 | 206| 154.5 | 174 | 1440 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 174 |
| 66% | 180 |
| 75% | 185 |
| 80% | 188 |
| 90% | 209 |
| 95% | 337 |
| 98% | 585 |
| 99% | 1306 |
| 100% | 1440 (longest request) |

Bench 2

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: | 19.798 seconds |
|Complete requests: | 1000 |
|Failed requests: | 0 |
|Keep-Alive requests: | 0 |
|Total transferred: | 9198002 bytes |
|HTML transferred: | 8671000 bytes |
|Requests per second: | 50.51 [#/sec] (mean) |
|Time per request: | 197.983 [ms] (mean) |
|Time per request: | 19.798 [ms] (mean, across all concurrent requests) |
|Transfer rate: | 453.70 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
| Connect | 106 |153 124.3 | 133 | 1417 |
| Processing | 30 | 42 19.4 | 38 | 231 |
|Waiting | 30 | 41 19.4 | 37 | 226 |
|Total | 140 | 195 |125.9 | 173 | 1457 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 173 |
| 66% | 179 |
| 75% | 182 |
| 80% | 185 |
| 90% | 197 |
| 95% | 312 |
| 98% | 447 |
| 99% | 1288 |
| 100% | 1457 (longest request) |

Bench 3

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: 20.133 seconds |
| Complete requests: | 1000 |
| Failed requests: | 0 |
| Keep-Alive requests: | 0 |
| Total transferred: | 9198002 bytes |
| HTML transferred: | 8671000 bytes |
| Requests per second: | 49.67 [#/sec] (mean) |
| Time per request: | 201.335 [ms] (mean) |
| Time per request: | 20.133 [ms] (mean, across all concurrent requests) |
| Transfer rate: | 446.14 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
| Connect: | 107 | 156 | 142.2 | 132 | 2409 |
| Processing: | 31 | 41 | 18.5 | 38 | 211 |
| Waiting: | 30 | 40 | 18.2 | 37 | 185 |
| Total: | 147 | 197 | 143.4 | 170 | 2466 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 170 |
| 66% | 176 |
| 75% | 179 |
| 80% | 182 |
| 90% | 198 |
| 95% | 324 |
| 98% | 370 |
| 99% | 1306 |
| 100% | 2466 (longest request) |

Sage 10

Bench 1

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: | 20.878 seconds |
|Complete requests: | 1000 |
|Failed requests: | 0 |
|Keep-Alive requests: | 0 |
|Total transferred: | 10559047 bytes |
|HTML transferred: | 10016000 bytes |
|Requests per second: | 47.90 [#/sec] (mean) |
|Time per request: | 208.785 [ms] (mean) |
|Time per request: | 20.878 [ms] (mean, across all concurrent requests) |
|Transfer rate: | 493.88 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
|Connect: | 112 | 165 | 165.1 | 133 | 1410 |
|Processing: | 30 | 42 | 21.8 | 38 | 422 |
|Waiting: | 29 | 41 18.2 | 37 | 191 |
|Total: | 146 | 207 166.4 | 173 | 1459 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 173 |
| 66% | 178 |
| 75% | 183 |
| 80% | 186 |
| 90% | 206 |
| 95% | 330 |
| 98% | 1267 |
| 99% | 1298 |
| 100% | 1459 (longest request) |

Bench 2

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: | 20.172 seconds |
|Complete requests: | 1000 |
|Failed requests: | 0 |
|Keep-Alive requests: | 0 |
|Total transferred: | 10559052 bytes |
|HTML transferred: | 10016000 bytes |
|Requests per second: | 49.57 [#/sec] (mean) |
|Time per request: | 201.724 [ms] (mean) |
|Time per request: | 20.172 [ms] (mean, across all concurrent requests) |
|Transfer rate: | 511.17 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
|Connect: | 107 | 157 | 148.3 | 130 | 1406 |
|Processing: | 31 | 42 | 27.7 | 38 | 580 |
|Waiting: | 30 | 41 | 24.8 | 37 | 580 |
|Total: | 141 | 199 | 150.7 | 170 | 1445 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 170 |
| 66% | 175 |
| 75% | 179 |
| 80% | 181 |
| 90% | 197 |
| 95% | 312 |
| 98% | 553 |
| 99% | 1302 |
| 100% | 1445 (longest request) |

Bench 3

Overview

| measurement | value |
|:-----|:----|
|Time taken for tests: | 19.676 seconds |
|Complete requests: | 1000 |
|Failed requests: | 0 |
|Keep-Alive requests: | 0 |
|Total transferred: | 10559037 bytes |
|HTML transferred: | 10016000 bytes |
|Requests per second: | 50.82 [#/sec] (mean) |
|Time per request: | 196.763 [ms] (mean) |
|Time per request: | 19.676 [ms] (mean, across all concurrent requests) |
|Transfer rate: | 524.06 [Kbytes/sec] received |

Connection Times (ms)

min | mean[+/-sd] | median | max |
| :----|:------ | :------------ |:-------|
|Connect: | 110 153 121.1 | 134 | 1410 |
|Processing: | 31 | 41 20.7 | 38 | 410|
Waiting: | 30 | 40 17.2 | 37 | 201 |
Total: | 144 | 194 122.8 | 172 | 1456 |

Percentage of the requests served within a certain time (ms)

| Requests served | ms |
|:--- |:--- |
| 50% | 172 |
| 66% | 177 |
| 75% | 180 |
| 80% | 184 |
| 90% | 191 |
| 95% | 312 |
| 98% | 431 |
| 99% | 1284 |
| 100% | 1456 (longest request) |

[kellymears]

For build times Sage 10 definitely took the cake -- but the difference was far less pronounced when not building for a production env.

Sage 9 appears to have generated smaller files and that holds up to scrutiny in the browser console too -- about 20kb more for Sage 10, averaged over a couple pages. Practically, this boils down to about a 3-5% difference in time to paint (in my case).

Sage 9

 DONE  Compiled successfully in 7650ms                                                                                                                                                                                       7:54:41 AM

                                  Asset       Size  Chunks             Chunk Names
               styles/main_2f56a313.css    40.3 kB       0  [emitted]  main
               scripts/main_2f56a313.js     177 kB       0  [emitted]  main
         scripts/customizer_2f56a313.js  752 bytes       1  [emitted]  customizer
                            assets.json    1.08 kB          [emitted]

Sage 10

 DONE  Compiled successfully in 1248ms                                                                                                                                                                                       7:50:55 AM

                 Asset      Size               Chunks             Chunk Names
       /scripts/app.js  18.5 KiB         /scripts/app  [emitted]  /scripts/app
/scripts/customizer.js  1.41 KiB  /scripts/customizer  [emitted]  /scripts/customizer
  /scripts/manifest.js  6.08 KiB    /scripts/manifest  [emitted]  /scripts/manifest
    /scripts/vendor.js   936 KiB      /scripts/vendor  [emitted]  /scripts/vendor
        styles/app.css  48.9 KiB         /scripts/app  [emitted]  /scripts/app

Both sites earned a 100 in lighthouse. I love Sage 10 quite a lot and look forward to using it until Auttomatic complete their mission of total self destruction.

[swalkinshaw]

Do we know what's increasing the asset sizes?