Lighthouse: Does it work with headless Chrome?

[mohsen1]

Should I stop using Xvfb in our CI?

[ebidel]

Some related material:

• https://github.com/GoogleChrome/lighthouse/blob/master/docs/headless-chrome.md
• https://github.com/GoogleChrome/lighthouse/issues/1645
• https://github.com/GoogleChrome/lighthouse/issues/1970
• https://github.com/GoogleChrome/lighthouse/issues/1932

When I tested some time ago, there were bugs with headless that skewed the lighthouse perf numbers. It's possible those have been worked out since. See my comment in https://github.com/GoogleChrome/lighthouse/issues/1970#issuecomment-292579943 for more info.

[wardpeet]

@ebidel can we close this? As your IO talks cover lots of this

[paulirish]

Lighthouse works with --headless but we need to do some testing to verify any perf skew is acceptable.

[alekseykulikov]

@paulirish @ebidel I did some testing for --headless flag, and perf metrics are quite different.
It seems, in headless mode there're no device emulation or at least network throttling.
General pattern: perf score is higher, run time is faster and it's not possible to get 100 on pwa score, because of load-fast-enough-for-pwa audit.

---

I wrote a simple script:

const lighthouse = require('lighthouse')
const { launch } = require('lighthouse/chrome-launcher/chrome-launcher')

function launchChromeAndRunLighthouse (url, launchFlags = {}) {
  return launch(launchFlags).then(chrome => {
    const flags = { output: 'json', port: chrome.port }
    return lighthouse(url, flags).then(r => {
      return new Promise((resolve) => setTimeout(resolve, 500)).then(() => { // wait a little
        return chrome.kill().then(() => {
          console.log('url: %s, launch flags: %j', url, launchFlags)
          console.log('fast pwa: %s, %s', r.audits['load-fast-enough-for-pwa'].score, r.audits['load-fast-enough-for-pwa'].debugString)
          console.log('fmp: %s', r.audits['first-meaningful-paint'].displayValue)
          console.log('pwa score: %s, perf score: %s', getScore(r, 'Progressive Web App'), getScore(r, 'Performance'))
          console.log('run time: %sms', r.timing.total)
        })
      })
    })
  })
}

function getScore (result, scoreName) {
  const score = result.aggregations.find(({ name }) => name === scoreName)
  return score ? (100 * score.total).toFixed(2) : null
}

const url = 'https://developers.google.com/'
launchChromeAndRunLighthouse(url)
// launchChromeAndRunLighthouse(url, { chromeFlags: ['--headless', '--remote-debugging-port=9222'], port: 9222 })

regular run of https://developers.google.com/:

url: https://developers.google.com/, launch flags: {}
fast pwa: true, undefined
fmp: 3527.8ms
pwa score: 54.55, perf score: 61.82
run time: 23548ms

run with --headless flag:

url: https://developers.google.com/, launch flags: {"chromeFlags":["--headless","--remote-debugging-port=9222"],"port":9222}
fast pwa: false, First Interactive was found at 2,673.186, however, the network request latencies were not sufficiently realistic, so the performance measurements cannot be trusted.
fmp: 2384.6ms
pwa score: 45.45, perf score: 91.59
run time: 17579ms

the similar pattern appears with other websites, for example https://www.nytimes.com

url: https://www.nytimes.com, launch flags: {}
fast pwa: false, Under 3G conditions, First Interactive was at 20,274.633. More details in the "Performance" section.
fmp: 2584.3ms
pwa score: 45.45, perf score: 29.59
run time: 49481ms

url: https://www.nytimes.com, launch flags: {"chromeFlags":["--headless","--remote-debugging-port=9222"],"port":9222}
fast pwa: false, First Interactive was found at 11,977.294, however, the network request latencies were not sufficiently realistic, so the performance measurements cannot be trusted.
fmp: 967.2ms
pwa score: 45.45, perf score: 55.47
run time: 37904ms

[ebidel]

@alekseykulikov awesome analysis. I've also observed network throttling issues under headless chrome. Basically, no built-in latency to the requests...which is why the "load-fast-enough-for-pwa" audit fails and the perf numbers are off.

I'm going to pass this off to the Headless Chrome team to see what they can do :)

[ebidel]

This is a known issue with headless chrome atm: https://bugs.chromium.org/p/chromium/issues/detail?id=728451 (no support for network throttling)

[paulirish]

Mostly resolved by #2675 but needs docs, basically.