Lighthouse: Add RUM data

[ebidel]

One thing I've been thinking a lot about is how to incorporate RUM data into LH reporting.

LH suggests many ideal targets for developers to hit, in particular perf numbers. Incorporating real-world data from sources like httparchive and caniuse could be a helpful barometer for devs to see where they're landing.

Where RUM data would be useful:

• see how your app stacks up against other top sites. In theory, the top sites are well built and are showcases of the web platform.
• help developers prioritize fixing perf issues.
• are the ideal numbers LH suggests reasonable?
• Analytics: what are the most important things I should address for _my_ users? e.g. have users input an Google Analytics ID and use the Google Analytics Reporting API to compare a run with the app's actual user base
• caniuse: "LH told me to use this new API, but it has 50% browser support". See https://github.com/GoogleChrome/lighthouse/issues/704

For httparchive data, I threw together a webservice on App Engine that always pulls from the latest snapshot of httparchive's bigquery dumps. Turns out they didn't have a JSON API.

endpoint: https://lighthouse-viewer.appspot.com/data
demo: https://lighthouse-viewer.appspot.com/

I also have a node script that we could run during report generation. The benefit of the webservice is that anyone can use it and it's highly cached.

POC UI:

There's a lot we could do in this space. I'm interested in hearing everyone's thoughts. Need some UI explorations. We could annotate certain audits (like the screenshot) or have an entire new section for RUM stuff.

@paulirish @igrigorik

[igrigorik]

I like where you're heading. Providing a baseline to help developers understand how their pages stack up is a great way to help flag outliers in particular... _"oh hey, my page seems to be consuming way more image bytes, maybe I should..."_

FWIW, imgix page weight tool does a pretty good job of this (e.g. sample report). In particular, I really like that they show the percentile stats ("The site as measured is in the 6th percentile for all mobile sites") and the possible win ("Optimizing images could move this site to the 5th percentile of mobile sites") if you apply their recommendations... Granted, determining the latter is not possible in many cases, but even the current percentile on its own is already helpful in providing a lot of context.

---

As a separate topic, one thing to be careful with is number+stats fatigue. We don't want to overwhelm the user with all the comparisons and percentiles. To address that, we could selectively highlight particular metrics (e.g. page weight), and/or only show the percentiles when certain thresholds are hit -- e.g. we've noticed that you're page is in 85th+ percentile for JS bytes, perhaps you should... and vice versa (as a footnote, perhaps): congrats, your page is in top 10% for JS bytes..

[paulirish]

On the roadmap for q2 2018.

[mgifford]

Ok, I have to admit I didn't know this one - Real User Monitoring = RUM.

My mind went elsewhere.

[connorjclark]

On the roadmap for q2 2018.

lol. so, where is it now :)

[wardpeet]

Is this still something we want to put in lighthouse locally? This feature currently lives in PSI with the Crux implementation. Making audits/result be aware of analytics will probably be a big assignment.

[alekseykulikov]

@wardpeet the field-performance plugin does precisely this. It adds CrUX data using PSI API to Lighthouse report https://github.com/treosh/lighthouse-plugin-field-performance

[paulirish]

@alekseykulikov btw as of a month ago, we have a crux-only mode for the api, though you can currently only get origin-level data.

https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=origin%3Ahttps%3A%2F%2Fdevelopers.google.com

(note the origin: in the URL)

We're talking about supporting url: in there. too.

cc @exterkamp

[alekseykulikov]

Thank you for adding this @paulirish! Indeed, the url: mode can be handy, and field-performance plugin will use this new API.

[exterkamp]

hot take: url: is GA :smile:

url vs. origin