Lighthouse: Question: How to test page behind authentication?

[fdn]

I found that authentication is cleared when requesting a page. It makes sense however when testing an app behind authentication it doesn't seem like there is a way to specify an authorization header or cookie to be used.

Reference issue: https://github.com/GoogleChrome/lighthouse/issues/592

Any tips on how I can test a page that is behind authentication?

UPDATE (LATEST METHOD): The best way to currently do this is to load your authenticated page in DevTools, uncheck the "Clear storage" checkbox, and run Lighthouse.

[brendankenny]

Is this in the extension? Cookies should not be cleared, so you should stay logged in.

If this is the CLI, you can run npm run chrome to launch a Chrome with the right flags set, log in to the site, then run lighthouse as normal. It will use the Chrome that was already launched.

We do clear a lot of other things in storage, though, so more exotic forms of authentication may not work.

[fdn]

Sorry, I should have specified that this is from the CLI. I'm trying to automate the process as much as possible so manual authentication will be a problem. Assuming nothing works out of the box, I'll try hacking at the gather source files.

[brendankenny]

It shouldn't require hacking the gather files. How were you intending to authenticate in this workflow?

If you were just going to manually authenticate in Chrome once and then reuse that browser repeatedly, that should still work. Lighthouse will talk to any Chrome launched with --remote-debugging-port=9222 (and that port number can be changed by passing in --port=XXXX to Lighthouse), so you can launch like that and keep that workflow.

You'll probably want to launch Chrome with the other flags here as well to reduce noise due to extensions, prevent first run screen, etc.

[fdn]

I don't intend on manually authenticating Chrome before the run. I'm leveraging Lighthouse for the performance monitoring so I want to do this at scale.

[paulirish]

I can see room for a "setCookie" command being added to the driver, probably very similar to the url blocking PR

[brendankenny]

I don't intend on manually authenticating Chrome before the run. I'm leveraging Lighthouse for the performance monitoring so I want to do this at scale.

Can you explain how your workflow (current or intended) for authenticating in a CI or performance monitoring environment works? Happy to make this easier for folks doing this.

[paulirish]

I'll take a stab at that. For example: I want to perf test the loading of drive.google.com with a very active account, and watch this for regressions. Would need to set cookies on the run configuration because the browser is otherwise a clean slate.

---

To me it seems to be similar to the scripting capabilities of WebPageTest, where things like custom cookies can be set: https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting

[brendankenny]

Would need to set cookies on the run configuration because the browser is otherwise a clean slate

Right, sorry, I understood that part :) my question was about how authentication like this is normally handled (or intended to be handled) in this kind of monitoring situation. setCookie seems like a good solution in this case.

[fdn]

@paulirish SetCookie would be nice. Thinking about it a bit more, SetHeaders might be more generic providing a mechanism for basic auth as well.

[fdn]

I poked around a bit. It looks like there isn't a mechanism for setting cookies in the Chromium API. But there is a mechanism for setting arbitrary headers.

driver.js

  setExtraHTTPHeaders(jsonHeaders) {
    let headers;
    try {
      headers = JSON.parse(jsonHeaders);
    } catch(e) {
      log.warn('Driver', 'Invalid extraHeaders JSON');
      headers = {};
    }
    return this.sendCommand('Network.setExtraHTTPHeaders', {
      headers
    });
  }

gather-runner.js

      .then(_ => driver.clearDataForOrigin(options.url))
      .then(_ => driver.setExtraHTTPHeaders(options.flags.extraHeaders || '{}'))
      .then(_ => driver.blockUrlPatterns(options.flags.blockedUrlPatterns || []));

Invoking

node node_modules/lighthouse/lighthouse-cli http://localhost:8080 --disable-device-emulation --disable-network-throttling --perf --extra-headers="{\"Authorization\":\"Basic YWRtaW46YWRtaW4=\"}"

I was able to send a basic authorization header which allowed me authenticated access. The JSON configuration via the CLI is gross but it works.

[wardpeet]

@fdn looks like devtools has https://chromedevtools.github.io/debugger-protocol-viewer/tot/Network/#method-setCookie
we can add both though

[fdn]

@wardpeet That's much more convenient than constructing the cookie header manually. 👍

Is there anything I can do to help this functionality get implemented?

[paulirish]

@fdn a PR would be awesome. Some mix of #1195 and your comment above makes sense to me. Up to you if you would prefer to do it at the HTTP header level or right to setCookie.

[canamo06]

Any update on this issue? is there a way to do it in the 2.0.0 version? I need to test an authenticated page from CLI.

[fdn]

I never submitted a pull request since the API is quite gross. You might be able to cherry pick my commit into your local... or use the commit from above 2f54c0e

https://github.com/fdn/lighthouse/commits/feature/new-flags

[gilberto-torrezan]

Just a comment, if you're using basic authentication, you can run lighthouse this way:

lighthouse https://user:[email protected]

[patrickhulce]

2599 uses a combination of localStorage and cookies for auth too

Rather than trying to craft specific logic in Lighthouse it might make more sense to make the artifactless custom gatherer story a little easier and allow users to put their arbitrary setup code in the beforePass there.

[wardpeet]

@patrickhulce sounds like a good idea, perhaps we should add an example in how it would work and just refer to that example, people will have to create a custom config but might be good enough. Most people can have the --cookie option which most people use for auth anyway

[unindented]

Any updates on this? Or do you know of an external tool that'll help us do this? We really want to set up lighthouse reporting as part of our CI pipeline, but all our important pages are behind cookie authentication.

[benschwarz]

@unindented, Hey! I'd seen this thread and didn't want to jump in promote my product, but seeing as you asked reasonably directly, it seems more ok to do so.

Calibre runs lighthouse and allows you to create 'test profiles'. A test profile can: Alter the bandwidth, emulate a mobile device and set cookies for authentication. There's also an option to use form-based authentication too, if that's easier.

[runlevelsix]

Is another possible path to achieving testability of pages/sites/apps that require login to allow passing the name of a Chrome profile to Lighthouse or the Chrome Launcher?

I have a test profile that is intended to maintain cookies, local storage, cache, etc. and has no extensions installed. If Lighthouse could be launched with a flag like --use-profile='TEST', that could solve this issue.

This doesn't appear to be the same as #2291, which seems to create the same empty profile in a user-writable location so it can be later deleted. Of course, my idea would also require not deleting the profile afterward (which seems to be the default for Chrome Launcher).

[patrickhulce]

@runlevelsix thanks for the suggestion! That method is already mostly supported by virtue of the fact that you can launch chrome with the profile you need before running Lighthouse and then passing the port that your instance is running on to Lighthouse.

[brendankenny]

I think it's just a question of how many chrome-launcher flags we want to also expose on lighthouse.

If using lighthouse as a module you can just pass in userDataDir to chrome-launcher, but I don't believe there's a CLI-based method of passing in a profile directory (manual-chrome-launcher (usable by chrome-debug if lighthouse is installed globally or yarn chrome in a git checkout) would need to add support)

[wardpeet]

@patrickhulce @brendankenny @runlevelsix I have a branch ready to allow cookies though just haven't found the time to get it PR ready (tests, smoketests). Which allows to set basic cookies through cli. And more advanced config through config.

[sandstrom]

@wardpeet Sounds good! Would it handle other headers too?

We don't use cookies for our single-page app (we use another header to pass the token).

[wardpeet]

No mine just adds cookie support but we might put an example online on how to do it with puppeteer. I'll have more info during the week

[vinamratasingal-zz]

Some more thoughts here: https://github.com/GoogleChrome/lighthouse/pull/3857

Would be good to understand the scope of the problem here (i.e. where do users most want this feature- CLI, DevTools, or Extension), and then create a product plan from there. There is some cool stuff we can do with Puppeteer itself and when we think about integrating the two products together, I'd love to flesh out the product story a bit more.

[rupesh1]

@sandstrom would this work for your use case? https://github.com/GoogleChrome/lighthouse/pull/3732

[sandstrom]

@rupesh1 Yes, it would! 👍

[johntron]

The work @rupesh1 did getting @patrickhulce's changes into a PR is great. Is there a plan for doing this sort of pre-run configuration? I can see others chiming in requesting similar features for local-/sessionStorage, Service Workers, Web Workers, etc.

Would be good to understand the scope of the problem here

• Testing as a specific user using standard authentication schemes (form w/session cookie, basic auth, etc.)
• Testing "private" deployments (e.g. checking a fully-integrated staging environment) using more obscure authentication methods (e.g. custom user agents, special cookies, localStorage values, etc.)

I think if Lighthouse wants to be run in CI (I just integrated with my client's Jenkins pipeline :D), it'll need to expose a variety of methods for this sort of thing. Ignoring authentication for a second, there are other use cases where this sort of pre-run configuration could be valuable (e.g. setting country/language for localization testing or setting customer segmentation cookies for A/B testing). The core developers need to decide if this is the type of usage they want to support with the project.

If the team does want to support more complex use cases, should we consider exposing the low-level Chrome remote interface? Obviously this is a slippery slope, because then other things might be built on top of that – think Selenium – and Lighthouse may be burdened by supporting requests for CRI support from third-parties.

On that note, maybe the answer is to create some abstraction layer with API's for the most common use cases (initially) and exposing this via hooks like _beforeAll(browser)_, _beforePass(pass, browser)_, _afterPass(pass, browser)_, _afterAll(browser)_ where _browser_ has methods like _setCookies(cookies)_, _setHeaders(headers)_, _setLocalStorage(storage)_, etc.

[unindented]

@johntron If they start exposing Chrome remote interfaces, they'll overlap with Puppeteer.

What I'm currently doing is starting up Puppeteer, running whatever pre-run scripts I need (setting cookies, localStorage, etc.), and then telling Lighthouse to connect to the instance of Chromium that Puppeteer already started. That way I get all the power of Puppeteer without the need for Lighthouse to expose more low-level stuff.

[anishkny]

@unindented that sounds like a great approach - do you mind sharing how exactly you are doing this? I can start Puppeteer and get it into a logged in state fine, how do I then get Lighthouse to exercise the Puppeteer controlled Chrome/Chromium?

Thanks!

[unindented]

@anishkny I created a few helper functions:

export async function startBrowser(options: Options): Promise<Browser> {
  return puppeteer.launch({
    headless: !process.env.DEBUG,
    slowMo: 50,
    args: [`--remote-debugging-port=${options.debugPort}`],
  });
}

export async function startPage(browser: Browser, options: Options): Promise<Page> {
  const page = await browser.newPage();

  await page.goto(options.url, { waitUntil: 'networkidle0' });

  await page.waitForSelector('input[type="email"]', { visible: true });
  const emailInput = (await page.$('input[type="email"]')) as ElementHandle;
  await emailInput.type('[email protected]');
  await emailInput.press('Enter');

  return page;
}

export function runLighthouse(options: Options): Promise<any> {
  return lighthouse(options.url, { port: options.debugPort, disableStorageReset: true }, null);
}

And then in my test I do:

  beforeAll(async () => {
    browser = await startBrowser(options);

    const page = await startPage(browser, options);
    await page.close();

    results = await runLighthouse(options);
    // now you can do whatever you want with the info from lighthouse
  });

[johntron]

FYI - #3732 added support for custom HTTP headers to the JS API and the CLI.

[YoungElPaso]

@unindented yeah, I basically did the same thing when I was using Backstop. I feel like that's decent approach since it will x-module / testing framework.

[prescottprue]

Similar approach to @unindented, but we were already using Nightmarejs, so I used that instead of puppeteer:

import Nightmare from 'nightmare';
import lighthouse from 'lighthouse';
import chai from 'chai';
import { get } from 'lodash';

const performanceOnlyConfig = require('lighthouse/lighthouse-core/config/perf.json');

const lhConfig = { port: 5858, output: 'json' };
const nightmareConfig = {
  switches: {
    '--remote-debugging-port': 5858 // to match debut port passed to lighthouse
  },
}

const nightmare = Nightmare(nightmareConfig);

let results 

describe('Performance', () => {
  before(async function () {
    this.retries(3);
    const pathToTest = 'some/path'
    // Navigate to path using nightmare
    await nightmare.goto(pathToTest);
    // Call lighthouse performance audit tool
    results = await lighthouse(pathToTest, lhConfig, performanceOnlyConfig);
  });

  after(async () => {
    await nightmare.end();
  });

  it('First Meaningful Paint less than 10 seconds', () => {
    chai.expect(get(results, 'audits.first-meaningful-paint.rawValue'))
      .to.be.below(10000);
  });
});

[wardpeet]

Here is a way on how to use puppeteer, which you could use to login first.

https://github.com/GoogleChrome/lighthouse/blob/master/docs/puppeteer.md

[gdavalos]

Is there a way to use the lighthouse extension to test a PWA which stores the auth credentials in localstorage? Or do I have to change it so that the credentials are stored in the cookies for it to work?

[patrickhulce]

@gdavalos unfortunately not able to use the extension if kept in localStorage, but you can use DevTools with clear storage off.

[gdavalos]

@patrickhulce you mean the nodejs lib? do you have an example of that?

[patrickhulce]

Oh I was talking about the DevTools panel in Chrome actually, but the CLI works too.

CLI

/path/to/chrome --remote-debugging-port=9222 # login to your app
lighthouse --port=9222 --disable-storage-reset https://url-to-your-app.com

Node
```js
lighthouse('https://my-url.com', {port: 9222, disableStorageReset: true})

[paulirish]

At the very least we need documentation. DevTools users can use "preserve storage" and CLI users can workaround this awkwardly.
We're still investigating how users can script this or use puppeteer.

[stuartsan]

In case it's useful for documenting a workaround for CLI users -- I have a use case where there are a couple auth-token kinda things in localStorage and there isn't a straightforward route to fetching the values needed and then constructing a header to pass in with --extra-headers to make authenticated requests. And I want to stick with the CLI lighthouse invocation. So I'm doing a custom gatherer with a beforePass where puppeteer does the setup, logging me in and getting those values into localStorage:

const Gatherer = require('lighthouse').Gatherer;
const puppeteer = require('puppeteer');

class Authenticate extends Gatherer {
  async beforePass(options) {

    const ws = await options.driver.wsEndpoint();

    const browser = await puppeteer.connect({
      browserWSEndpoint: ws,
     });

    const page = await browser.newPage();
    await page.goto(process.env.TEST_URL);

    await page.click('input[name=username]');
    await page.keyboard.type(process.env.USER);

    await page.click('input[name=password]');
    await page.keyboard.type(process.env.PASSWORD);

    // sign in button
    await page.click('span[class^=Section__sectionFooterPrimaryContent] button');

    // this means the login succeeded
    await page.waitForSelector('.dashboard');

    browser.disconnect();
    return {};
  }
}

module.exports = Authenticate;

A couple things I came across that might be worth noting for this kind of setup:

• It's also necessary to create a custom audit that depends on this gatherer (if no audit depends on it via requiredArtifacts, it'll be skipped), then use a custom config file to extend the default set of audits to include that audit and pass the config file in as --config-path
• I assumed I would want to pass in the --disable-storage-reset flag here to ensure the stuff set in localStorage isn't blown away, but it works ok without it, the values are preserved (it looks like the storage reset happens before the gatherer's beforePass, so that makes sense). And using --disable-storage-reset actually gave me inflated performance scores (100 with it vs 88 without). My guess is that the inflated scores are because I visited authenticated page $FOO with puppeteer first, everything was cached, and then LH got to $FOO and was like "sweet! 100, perfect". But not sure. I don't understand how _without_ --disable-storage-reset I'm getting a more accurate perf score and localStorage is also preserved. UPDATE I think I get it, seems to be that the absence of --disable-storage-reset marks it as a perf run, which clears the browser caches but not other storage, which is what I want.

[connorjclark]

Thanks for sharing @stuartsan! That is an interesting use of a gatherer. It begs for us to implement a "before" hook.

So I understand better, could you describe the limitations you have that prevent a solution like the one described here? It seems that if writing a custom audit works for you, then writing a custom launcher for Lighthouse instead could also work for you.

[stuartsan]

@Hoten, sure thing! I think I definitely _could_ write a custom launcher instead, but the context around why I preferred this setup in my project:

• I'm running lighthouse via CLI in the lighthousebot backend container, inside a CI environment. In the container LH is installed globally.
• Already passing in a custom config file via --config-path (for a different custom audit)
• Testing one page that doesn't need authentication, regular way of launching via CLI works fine
• Testing another page that needs authentication. With the custom gatherer route, I can preserve the lighthouse CLI invocation (just pass in a different config file) so it's the same across anonymous/authenticated pages
• (The exact thing I'm doing is here)

So doing a custom launcher instead wouldn't be a big deal, it's just more convenient and feels a little more standardized, or something, to me, to be able to stick with calling lighthouse and passing in a custom config file across the two cases, rather than reworking the launch and adding a minor layer of indirection (calling ./authenticateAndLaunchLighthouse.js or whatever)

Also, if my observations around --disable-storage-reset are accurate, I think I'd need to pass that option in the custom launcher, to ensure that localStorage is preserved:

lighthouse(options.url, { port: options.debugPort, disableStorageReset: true }, null);

But then it seems like I'd be back to the wrongly inflated perf score, because it wasn't considered a "perf run" and my authenticated page (which is the same url that shows the login form when not logged in) was cached. (If that's right, maybe this is avoidable through clearing the cache before puppeteer hands the page off, but IDK).

Overall I think it's more convenient to be able to preserve the standard way of launching via CLI and also be able to hook in and do some general setup, but I get that it might not be LH's responsibility to support that use case :)

A more general "before" hook would be super sweet IMO.

[SharkBaby]

Similar approach to @unindented, but we were already using Nightmarejs, so I used that instead of puppeteer:

import Nightmare from 'nightmare';
import lighthouse from 'lighthouse';
import chai from 'chai';
import { get } from 'lodash';

const performanceOnlyConfig = require('lighthouse/lighthouse-core/config/perf.json');

const lhConfig = { port: 5858, output: 'json' };
const nightmareConfig = {
  switches: {
    '--remote-debugging-port': 5858 // to match debut port passed to lighthouse
  },
}

const nightmare = Nightmare(nightmareConfig);

let results 

describe('Performance', () => {
  before(async function () {
    this.retries(3);
    const pathToTest = 'some/path'
    // Navigate to path using nightmare
    await nightmare.goto(pathToTest);
    // Call lighthouse performance audit tool
    results = await lighthouse(pathToTest, lhConfig, performanceOnlyConfig);
  });

  after(async () => {
    await nightmare.end();
  });

  it('First Meaningful Paint less than 10 seconds', () => {
    chai.expect(get(results, 'audits.first-meaningful-paint.rawValue'))
      .to.be.below(10000);
  });
});

@prescottprue I am trying to use lighthouse in nightmare , I got below error, Do you have any solution for this issue? need your help .thanks so much.

[paulirish]

We now have great documentation and a working example on how to handle these situations:

https://github.com/GoogleChrome/lighthouse/blob/master/docs/authenticated-pages.md
https://github.com/GoogleChrome/lighthouse/blob/master/docs/recipes/auth/README.md

fixed by #9628