Lighthouse: Spinning my own PageSpeed Insights?

[bjesus]

Hi there! I'm seeing major differences in scoring between my Lighthouse installation and Google PageSpeed Insights. I assume there are many reasons for that - probably a different machine, a different Lighthouse version, location etc. What can I do to make my results closer to the official GPSI setup? Does anyone know or did someone experiment on what's the most similar AWS/GCP machine to run LH on, using which setup, where in the world, and what version?

The reason I can't use GPSI is because we're sometimes doing some internal DNS lookups, but when the scores are so different between our LH and GPSI, people just don't trust them.

Thank you!

[patrickhulce]

cc @exterkamp @jazyan

You might find this article on some PSI differences useful.

The configs for Lighthouse used in PSI are in this repo.

https://github.com/GoogleChrome/lighthouse/blob/master/lighthouse-core/config/lr-mobile-config.js
https://github.com/GoogleChrome/lighthouse/blob/master/lighthouse-core/config/lr-desktop-config.js

[bjesus]

Thanks, that is useful. Is there any information regarding what type of machine would be the closest to PSI? I suppose it matters if I'm running this on a 16gb/8cores machine vs a 1gb/1core machine, doesn't it?

[patrickhulce]

Is there any information regarding what type of machine would be the closest to PSI?

I'm not sure what is closest to PSI (maybe @exterkamp knows if he is allowed to share?) but the cheapest GCP instance you'd want to run Lighthouse on is n2-standard-2. Our recommendations for hardware are in the Run on Adequate Hardware docs.

[bjesus]

Thanks you! Isn't it though that this document is more like "minimum requirements" needed to run Lighthouse, and if I'll get anything stronger than these specs, my score will also go higher? Or does Lighthouse generally reports the same score even if the machine is blazing fast?

Would still be curious to know what's the closest to PSI, if it's possible to share.

[patrickhulce]

"minimum requirements" needed to run Lighthouse, and if I'll get anything stronger than these specs, my score will also go higher?

Kind of. Since most of the economical cloud infra out there is billed based on cores and memory and not the strength of each core, we wouldn't expect scores to meaningfully improve by increasing either of those beyond our recommendations. We also recommend scaling horizontally with more instances rather than vertically with stronger machines, so our "minimum requirements" are essentially our recommendations as well.

[dvelasquez]

I have been doing some calibration on the metrics using lighthouse for a while.
Each lighthouse run gives you a benchmarkIndex and if you are getting scores under 500, you will encounter very erratic results, check this comment from @patrickhulce explaining this: https://github.com/GoogleChrome/lighthouse/issues/9085#issuecomment-535178985
Long story short, keep an eye in the benchmarkIndex of each run and if it goes under 500, do something to increase the cpu power.
After we started scoring ~500 the metrics stabilised and we started getting results very close to PSI (~3 difference).

[patrickhulce]

I think we've covered your questions here @bjesus but if you have more issues feel free to reopen.