Lighthouse: Lighthouse CLI on Circle CI scores higher than expected

[ataylorme]

Using the Chome extension, in incognito mode with other extensions disabled, I am getting scores in the mid-80's.

I added Lighthouse to my continuous integration process tests running on the live website and a new environment created for pull requests. The idea being if the overall performance score is more than 5 points worse on the pull request the Circle CI build will fail, disallowing a merge on GitHub.

To test this is intentionally created a pull request that should receive a lower score by adding a large image to the page and disable Varnish full page cache so the response is served directly from WordPress.

As expected, the Lighthouse report run from my local machine using the Chrome extension returns a performance score of 86 with a first paint of3,120ms.

Unexpectedly, the Lighthouse report run on Circle CI using the CLI version returns a performance score of 100 with a first paint of770ms.

The Circle CI log for the job running the Lighthouse test can be found here and the main repository code is open source and available here.

I expected something running on a CI server to be slightly faster but the large discrepancy (2,350ms) in first paint time and the resulting perfect performance score, for a URL that has known issues, is causing my desired use case of checking the score as part of a CI process to be ineffective.

Any suggestions are most welcome.

[patrickhulce]

Thanks for reporting @ataylorme great to see you're adding Lighthouse to your CI! 🎉 👏

It looks like your code is running Lighthouse on headless where throttling is not available. For this reason, we actually recommend running regular headful Chrome with XVFB in server environments (see the docs).

Let us know if you run into any issues getting XVFB running or you find something in those docs to be unclear!

related #3997 #2008 #2675

[ataylorme]

Thanks, @patrickhulce for the quick reply! The scores make more sense with no throttling in headless mode. Is there a maintained Docker image with Chrome, Lighthouse and XVFB I can use?

[ebidel]

You can use https://github.com/ebidel/lighthouse-ci. Working on bringing this work in officially to this repo: https://github.com/GoogleChrome/lighthouse/issues/3715

[ataylorme]

Great, thanks @ebidel. I've subscribed to that thread. I was able to use https://github.com/ebidel/lighthouse-ci in the meantime. Thank you for the work there! Really comprehensive.

Once a Docker image comes out I'll switch over to running Lighthouse locally in Circle CI.