Lighthouse: RFC: Add support for Performance Budgets

[addyosmani]

Goal: Let's make performance budgets front-and-center in your developer workflow

Performance budgets enable shared enthusiasm for keeping a site’s user experience within the constraints needed to keep it fast. They usher a culture of accountability that enable stakeholders to weigh the impact to user-centric metrics of each change to a site.

If you're using Lighthouse locally (via DevTools) or in CI, it should be clear when you're stepping outside your team's agreed on perf budgets.

UX mocks we're evaluating

See the complete set of UX mocks

Setting budgets

A budget can be set for one or more metrics and will tell the team that the metric can’t go above or below a certain amount.

Referencing work by Tim Kadlec, metrics for perf budgets can include:

• Milestone timings:  timings based on the user-experience loading a page (e.g Time-to-Interactive). You’ll often want to pair several milestone timings to accurately represent the complete story during page load.

• Quantity based metrics : based on raw values (e.g. weight of JavaScript (KB/MB)). These are focused on the browser experience.

• Rule-based metrics:  scores generated by tools such as Lighthouse or WebPageTest. Often, a single number or series to grade your site.

Teams who incorporate budgets into their workflow will often have CI warn or error a build if a PR regresses performance.

Budgets could vary by whether you're in production or dev, target device class (desktop/mobile/tablet) or network conditions. We should evaluate to what extent developers will need/want to differenciate between these considerations when setting a budget.

Some examples of budgets

• Our home page must load and get interactive in < 5s on slow 3G/Moto G4 (lab)
• Our product page must ship less than 150KB of JavaScript on mobile
• Our search page must include less than 2MB of images on desktop

Budgets can be more or less specific and the thresholds applied can vary based on target networks and devices.

While Lighthouse already provides some default thresholds for metrics like Time-to-Interactive or large JavaScript bundles, a budgeting feature enables teams to adapt budgets to fit their own needs.

Where should budgets be incorporated? (strawman)

Please note this is just a rough sketch for what an integration could look like

Potential touch points for rolling out a performance budgeting solution:

• Chrome DevTools: Support for reading a budget.js/json file for a project. Workspaces could detect the presence of such a file and ask you if you want it applied for the project. Highlight where useful that budgets have been crossed (e.g. Lighthouse, Network panel)
• Lighthouse: We have many options here, some of which are mocked out lower down. One is setting a budget via a config file (per above). Your Lighthouse report highlights any time metrics/resource sizes cross that budget.
• Lighthouse CI: Reads a provided budget.json/budgets.js which will fail the build when supplied budgets are crossed. (e.g TTI < 5s, total JS < 170KB). Work towards this being easy to adopt for use with Travis/GitHub projects.
• Third-party tooling?: Encourage adoption of budget.js/json in popular tools (webpack, SpeedCurve, Calibre, framework tooling, CMS)
• Potentially in RUM dashboards, although we might want to consider what this means.. e.g. TTI vs FID budgets

Defining a performance budget

Ultimately, we should leave this up to teams but give them some strong defaults.

Walking back from Alex Russell's "Can You Afford It?: Real-world performance budgets", this may be:

• Time-To-Interactive < 5s on Slow 3G on a Moto G4 (~4-5x CPU throttling)
• JavaScript budget of < 170KB if targeting mobile.
• Budgets for other resources can be drawn from a total page weight target. If a page cannot be larger than 200KB, your budget for images, JS, CSS, etc will need to fit in.

Budget targets and the resource constraints that drop out of them will heavily depend on who your end users are. If you're attempting to be interactive pretty quickly on a low-mid end device, you can't be shipping 5MB of JavaScript.

budgets.js/budgets.json

If we were to opt for a budgets.js/json style configuration file, this could look something like the following (strawman):

module.exports = {
    {
        "preset": "mobile-slow-3g",
        "metrics": {
            "time-to-interactive": {
                "warn": ">5000",
                "error": ">6000"
            },
            "first-contentful-paint": {
                "warn": ">2000",
                "error": ">3000"
            }
        },
        "sizes": {
            "javascript": {
                "warn": ">170",
                "error": ">300"
            },
            "images": {
                "warn": ">500",
                "error": ">600"
            }
        }
    },
    {
        "preset": "desktop-wifi",
        "metrics": {
            "time-to-interactive": {
                "warn": ">3000",
                "error": ">4000"
            },
            "first-contentful-paint": {
                "warn": ">1000",
                "error": ">2000"
            }
        },
        "sizes": {
            "javascript": {
                "warn": ">700",
                "error": ">800"
            },
            "images": {
                "warn": ">1200",
                "error": ">1800"
            }
        }
    }
}

However, we should be careful not to tie ourselves too tightly to this as variance/thresholds may change how we think about configuration. Paul and Patrick have done some great research about presets and we could lean more into that as needed.

Metrics threshold considerations

Lighthouse runs may vary, especially across different machines.

As we explore performance budgeting, we should consider what impact this may have on how users set the thresholds they use for budgeting. Some options as we've talked to projects like lighthouse-thresholds:

Option 1: Multiple LH runs (e.g. "runs": 3 in config)

Pros

• Easy to implement.
• Easy to understand what's occurring for consumers.
• Could calculate the Median Absolute Distribution, accounting for outliers.

Cons

• Very small (non-significant) sample size.
• More likely to be viewed as failing on network issues rather than site-specific issues. (ie. the I'll-retry-this-build-until-it-works solution)
• Multiple runs on the same machine will still suffer from the same network problems.
• CPU and time in CI environments (fought so hard for low build times, shame to let that increase significantly)

Option 2: Define thresholds in ranges

e.g. "first-contentful-paint": { "threshold": 1000, "deviation": "15%" }

Pros

• Results could be compared with user-defined deviation allowances.
• Results could fail if a results has deviated from the budget more than x%.

Cons

• Consumers are unlikely to want to set both an upper & lower bound (eg. why would it matter if TTI was really low?).
• A range definition could lead to a perception of unreliability.

Option 3: Generate measurement data and commit that to code

e.g. A user runs lighthouse-thresholds measure which triggers 10 or more LH runs, performed locally and results/data saved to a file. That file could then be committed and used to compare runs in CI/PRs against. If budgets are updated, so is the generated file.

Pros

• Would take more time to implement.
• Allows for a large sample size of measurements.
• More accurate calculation of Mean Absolute Deviation
• Opportunity to calculate and use standard deviations.
• Site is measured and budgeted against real data.
• Allows for more complicated statistical analysis like a Mean-Variance Analysis
• This file would be a lot like test snapshot files, so devs are already used to seeing things like this.
• In the absence of this file it could just revert to doing a regular threshold check.

Cons

• More complicated to communicate.
• Requires an accurate production run on a local machine (or elsewhere) - though could just be run against actual production locally.
• Another file to maintain and keep in source code along with the budget file.

Option 4: Distributed runs . e.g. LH is run on multiple machines multiple times

Pros

• Much more accurate performance representation.
• Network issues are eliminated (or at least reduced).
• Much more likely to be seen as reliable

Cons

• Difficult to implement
• Lots of http requests for a distributed solution = slow build?
• Resource intensive, possibly expensive
• CI machines have gigabit networks, performance is likely a lot better than real-world - does it matter?

Options 5: be smarter about how the metrics are computed e.g. Run LH a small number of times but be smarter about how that data is measured against thresholds.

Pros

• Can more accurately filter outliers.
• Can more accurately create a measure of Statistical Dispersion

Cons

• More difficult to communicate to consumers.
• Still such a small sample size.
• Statistics aren't magic.

How is this beneficial to Ligthhouse?

This would allow Lighthouse to reduce the friction for developer adoption of performance budgets, helping more sites hit a decent Lighthouse performance score.

Are you willing to work on this yourself?

Yep. I'm happy to own overall technical direction for budgeting and have asked @khempenius (implementation), @developit (framework integrations) and @housseindjirdeh (advisory) to work on this too.

Existing solutions supporting performance budgeting

• SpeedCurve's perf budgeting and alerts
• Calibre supports performance budgeting
• Performance budget builder by Brad Frost
• Lighthouse CI with target scores
• Lighthouse Thresholds - (pass or fail based on thresholds for audit categories)
• pwmetrics by Paul Irish
• Webpack performance budgets feature
• bundlesize for tracking JavaScript bundle sizes
• PerformanceBudget.io

[housseindjirdeh]

This is exciting! 🎉🎉🎉

Some initial thoughts (feel free to ignore):

• +1 to providing strong defaults and for possibly exploring budgets.js/json adoption with other tools.
• Do we need to separate for DEV/ PROD? I would assume developers would only really care about how well their site runs in production. 🤔
• Definitely see the benefit of differentiating for device/network and hope we can allow for this but still make the configuration process as easy as possible.
• It looks like this is leaning towards an opt-in approach. Have we considered _always_ showing default budgets to LH DevTools/CI where they can only be over-written by adding a config file? Not saying this approach is necessarily better, but it's a thought.
• If we go with opt-in, have we thought about providing some indicator to the UI? We would want to let as many developers know that this is something they can (and should) use. For example, a message/toast/etc that says something like "Provide a budgets.json file to set performance budgets! Click here to see more information."
• Would this work with the Chrome Extension?

[tkadlec]

FINALLY had a look at this—super excited about it!

Echoing @housseindjirdeh, I love the idea of budgets.js/json becoming something that could be used across other tools.

Have we considered always showing default budgets to LH DevTools/CI where they can only be over-written by adding a config file? Not saying this approach is necessarily better, but it's a thought.

I like this too. Will likely lead to more people using performance budgets (and budgets.json). It will also help provide some general awareness and a starting point for folks.

I may be overthinking it, but couldn't threshold accomplish the same thing as warn/error? I'm picturing folks setting a budget and then LH warns if it's between the budget and the threshold and errors if it exceeds the threshold.

[addyosmani]

Thanks for the feedback, Tim and Houssein!

+1 to providing strong defaults and for possibly exploring budgets.js/json adoption with other tools.
Echoing @housseindjirdeh, I love the idea of budgets.js/json becoming something that could be used across other tools.

This validation is quite helpful. I'm bullish on trying to see if we can adopt a source-of-truth for budget thresholds across tools :)

It looks like this is leaning towards an opt-in approach. Have we considered always showing default budgets to LH DevTools/CI where they can only be over-written by adding a config file? Not saying this approach is necessarily better, but it's a thought.

The way I was viewing budgets was that by default Lighthouse already flags when performance is suffering on user-happiness metrics using a curve for TTI, FCP etc. We don't yet do this as much for static resource sizes (e.g. JS size is too large). We could find a way for the medians and point-of-diminishing returns to be our "defaults" for budgets but this probably requires discussion with the broader team... :)

https://github.com/GoogleChrome/lighthouse/blob/7e969b20e568108949f19dab53309ae0d9faa912/lighthouse-core/audits/metrics/interactive.js#L44

I may be overthinking it, but couldn't threshold accomplish the same thing as warn/error? I'm picturing folks setting a budget and then LH warns if it's between the budget and the threshold and errors if it exceeds the threshold.

Good suggestion. That may indeed work as a simpler alternative to setting thresholds for warn/error. We can play around with the configuration of thresholds (for budgets.json) as @khempenius starts to explore the implementation :)

If we go with opt-in, have we thought about providing some indicator to the UI? We would want to let as many developers know that this is something they can (and should) use. For example, a message/toast/etc that says something like "Provide a budgets.json file to set performance budgets! Click here to see more information."

Like minds. I was thinking we could do something similar to the in-DevTools infobars about adding folders to Workspaces to encourage usage. This would involve discussions options with the DevTools team but worth us thinking about when and where we might show these messages.

Do we need to separate for DEV/ PROD? I would assume developers would only really care about how well their site runs in production.

One pro for DEV budgets is that they may be easier to hit if CI/production aren't the first time you experience warnings about the perf impact of your changes.

That said, non-PROD budgets are inherently complicated to accurately predict. Running an entire build (with minification, tree-shaking etc) can be costly while you're still working on your app. DEV might also include artifacts like debugging helpers (see React) which are stripped out for prod, but useful during iteration. @developit is thinking about DEV-time budgets and how feasible it would be to include them.

Would this work with the Chrome Extension?

As we wouldn't have access to Workspaces from the extension, we might be able to get it working with a slightly different UX (e.g. you manually select budgets.json and it is applied to the report). This will require some further conversation with the Lighthouse team to nail down how feasible it would be.

[khempenius]

Here's how I envision this might work (broken down into "v1" and "v2" releases). Thoughts on this?

V1

V1 would introduce basic functionality:

• User can specify budgets for one or more profiles (e.g. "desktop-wifi", "mobile-3g", etc.)
• Budgets can include any combination of the following metrics types: weights, requests, and timings.
• Lighthouse would automatically include a default budgets.json file for any projects without one. These defaults could be overwritten by providing a budgets.json file.

API (budgets.json)

// Parameters optional unless otherwise noted

var config = {

    "budgets": [{ //required

        "profile": "desktop-3g", //required

        "kilobytes": {

            "total": 1000,

            "css": 100,

            // ...

            // All options: total, css, fonts, html, images, javascript, video, other

        },

        "requests": {

            "total": 100,

            "thirdParty": 60,

            // ...

            // All options: total, thirdParty, css, fonts, html, images, javascript, video, other

        },

        "timings": {

            // All units are in milliseconds

            // Can be specified with or without tolerance

            "firstMeaningfulPaint": 500,

            "timeToFirstInteractive": {

                target: 625,

                tolerance: 100

            },

            "timeToConsistentlyInteractive": 900,

            "custom": {

                "whatever-you-named-that-performance-mark": 850

            }

        }

    }]

}

Comments:

The proposed supported timings are deliberately limited. I felt like a large selection of supported metrics could be a double-edged sword: good because they're thorough or bad because you end up with decision paralysis trying to choose between the 57 million ways you could be measuring rendering.

Usage with different development environments

Use one budgets.json file for multiple development environments.

// Declare some configs

// ...

var configs = {

    "development": config,

    "staging": anotherConfig,

    "production": yetAnotherConfig

}

module.exports = configs[process.env.NODE_ENV]

Default budgets.json

Multiple TBD profiles, all weight metrics only, i.e:

var config = {

    "budgets": [{

        "profile": "TBD",

        "kilobytes": {...}

    }]

}

Comments:

I have mixed feelings about a default budgets.json that only uses weight metrics, but I am leaning towards it because: a) the out-of-the-box configuration for timings would probably closely duplicate Lighthouse's existing metrics and would therefore be redundant, b) if you're not using this a CI-like environment (and even if you are) the timing metrics are probably going to fluctuate a lot, which may make for a frustrating introduction to perf budgets.

Possible downside to this: Users not realizing that these can be a part of performance budgets and/or are supported.

V2

V2 would be primarily focused on features for CI-environments:

• Persistence of results
• Graphs of metrics over time (i.e. builds)
• Support for multiple runs (timing metrics would be calculated using medians).
• Monitoring of deterministic metrics (i.e. weights & requests) for changes or regressions from build to build. (For example: "Total kilobytes should not increase by more than 3%" or "Images should not increase by more than 5 kilobytes").

API (budgets.json)

// Parameters optional unless otherwise noted

var config = {

    "budgets": [{ /* see v1 API */ }],

    "ci": {

        "logResults": true, // default value is false

        "multipleRuns": 3,

        "monitorChanges": {

            "kilobytes": {

                "total": "3%",

                "images": "5kb"

                // ...

                // All options: total, css, fonts, html, images, javascript, video, other

                // Can be specified in absolute or relative (percentage) terms

            },

            "requests": {

                "javascript": 0

                // ...

                // All options: total, thirdParty, css, fonts, html, images, javascript, video, other

            }

        }

    }

}

[kaycebasques]

You all probably had a good reason for this but it seems funky to me that resourceSizes and resourceCounts are separated.

Current:

[
    {
        "path": "/",
        "resourceSizes": [
            {
                "resourceType": "script",
                "budget": 125
            },
            {
                "resourceType": "image",
                "budget": 300
            },
            {
                "resourceType": "total",
                "budget": 500
            },
            {
                "resourceType": "third-party",
                "budget": 200
            }
        ],
        "resourceCounts": [
            {
                "resourceType": "total",
                "budget": 100
            },
            {
                "resourceType": "third-party",
                "budget": 0
            }
        ]
    }
]

Expected:

[
    {
        "path": "/",
        "budgets": [
            {
                "type": "script",
                "size": 125
            },
            {
                "type": "image",
                "size": 300
            },
            {
                "type": "total",
                "size": 500,
                "count": 100
            },
            {
                "type": "third-party",
                "size": 200,
                "count": 0
            }
        ]
    }
]

[paulirish]

hmmmmmm... yeah I do really like your proposal, kayce. @khempenius wydt???

aint no time like the present for a breaking change :)

[brendankenny]

we can land the current audit PRs and then change (if we want to), FWIW

[khempenius]

I think my reasoning for going with that API was the idea that a budget,
rather than a resource type, was the correct "base unit"/abstraction to
build all of this around. By not lumping things together, this would make
it easier to support other features later on:

A couple features that I see people using frequently, that I can see LW
supporting eventually:

• Setting budgets for absolute (KB) or relative (KB %) change between builds
• Differentiating between pass/warn/fail
• Alerting
• Setting budgets for a particular file(s) (a la bundlesize)

I think as a user, I'd expect to be able to do all these things on the
granularity of a single budget. Obviously this could be supported
regardless of API, but my concern was that while the {resourceType, count, size} format would be more convenient early-on, it wouldn't age as well.

Two other random comments that may be relevant:

• Setting budgets for resource counts is also something that I think 90-95%
  of people won't use.
• Hopefully this will support timing budgets at some point. At first
  glance, I'm not wild about mixing budget types in one array (i.e. if we
  were to drop the resourceSize, resourceCount, etc. properties and flatten
  this) - but I'm not firmly attached to that opinion.

On Tue, Apr 30, 2019 at 4:44 PM Brendan Kenny [e-mail]
wrote:

we can land the current audit PRs and then change (if we want to), FWIW

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleChrome/lighthouse/issues/6053#issuecomment-488108546,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOLOGAUEGVDR3GQQARBM33PTCVUFANCNFSM4FV2SEMA
.

[paulirish]

@khempenius got it. planning for budget-specific options, etc.

seems cool, that makes sense. 👍

[kaycebasques]

Future flexibility could be handled by making size and count objects instead. Having to duplicate the path is the thing that would be error prone for me. E.g. I update the path on line 20 but forget to update the one on line 25. Organizing around path makes more sense to me.

P.S. sorry to be "that guy" that discusses API stuff close to crunch time. Just noting my thoughts and I know that if I don't mention now I'll forget to later. I'll work with @khempenius's design and maybe this will just be something to revisit (or laugh at) later

[
    {
        "path": "/",
        "budgets": [
            {
                "type": "script",
                "size": {
                   "kilobytes": 125
            },
            {
                "type": "image",
                "size": {
                  "change": 5
                }
            },
            {
                "type": "total",
                "size": {
                  "kilobytes": 500
                },
                "count": {
                  "total": 100
                }
            },
            {
                "type": "third-party",
                "count": {
                  "change": 2
                }
            }
        ]
    }
]

[khempenius]

Re: Path
The budgets are currently based around paths, so if I’m interpreting what
you’re saying correctly, that shouldn’t currently be an issue.

In order words, each path should be distinct (note: that’s not enforced,
but in practice it should be if). And then any applicable budgets (counts,
sizes) are set as properties on the same object.

On Tue, Apr 30, 2019 at 3:45 PM Kayce Basques [e-mail]
wrote:

Future flexibility could be handled by making size and count objects
instead. Having to duplicate the path is the thing that would be error
prone for me. E.g. I update the path on line 20 but forget to update the
one on line 25. Organizing around path makes more sense to me.

P.S. sorry to be "that guy" that discusses API stuff close to crunch time.
Just noting my thoughts and I know that if I don't mention now I'll forget
to later. I'll work with @khempenius https://github.com/khempenius's
design and maybe this will just be something to revisit (or laugh at) later

[
{
"path": "/",
"budgets": [
{
"type": "script",
"size": {
"kilobytes": 125
},
{
"type": "image",
"size": {
"change": 5
}
},
{
"type": "total",
"size": {
"kilobytes": 500
},
"count": {
"total": 100
}
},
{
"type": "third-party",
"count": {
"change": 2
}
}
]
}
]

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleChrome/lighthouse/issues/6053#issuecomment-488142305,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOLOGCV3RMQTCW7I4ZSTYTPTDDZVANCNFSM4FV2SEMA
.

[kaycebasques]

Whoops I've got poo brain. On second look I see that are organized around paths

https://youtu.be/aZdtZIuVzmE

[kaycebasques]

The countOverBudget property that's emitted in the JSON output should be an integer, though, no?

[timker]

Can you consider adding budgets to PageSpeedInsights.

Most of the integrations that have been suggested are "on prem/local", so the networking costs are going to be better than the real world.

[connorjclark]

Can you consider adding budgets to PageSpeedInsights.

It's on our roadmap.

[brendankenny]

@addyosmani @khempenius is it ok to close this now?

I'd like to be able to get a sense of progress, so closing this and (eventually) opening a new issue/issues for v2, CI support, etc would be one way of doing that. "RFC: Add support for Performance Budgets" is obviously not very applicable for those goals, anyway :)

[matthojo]

On an related side note to this issue. I'm currently (re)building PerfLint and plan on adding support for Lighthouse as the primary results source (via Docker). It would be great to discuss adopting support for budgets.js/json in PerfLint too, working towards having the goal of a 'standardised' performance testing configuration that could be shared across tools.
I have briefly spoken to @housseindjirdeh about this at JSConf, so thought I'd extend the conversation here.

[khempenius]

@brendankenny Yep, that sounds good.

@matthojo Wonderful! I just sent you a message. We're in the awkward interim period between closing this thread and opening a new one soon to discuss LightWallet v2, so that's probably easiest in the interim.

[brendankenny]

SG!