Lighthouse: Chrome flag --user-agent not being respected

[rohit-nair]

Bug

Lighthouse CLI doesn't make requests with the useragent specified using the configuration options, ex --chrome-flags="--user-agent=lighthousebot" as seen from outbound network traffic.

Steps to reproduce

1. Run your network sniffer/inspector tool.
2. Run the following command lighthouse --output json --chrome-flags="--user-agent=lighthousebot" https://pwa.rocks/ > run1.json
3. Inspect the GET request for https://pwa.rocks/ in the tool launched in step 1.

Current behavior?

Useragent value is Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/66.0.3359.30 Mobile Safari/537.36

Additionally, the JSON report shows the useragent as lighthousebot

Expected behavior?

Useragent request header should have the value lighthousebot.

Environment Information

• Affected Channels: CLI
• Lighthouse version: 2.9.4
• Node.js version: 3.10.10
• Operating System: Win10, Win Server 2016, macOS High Sierra Version 10.13.4
• Chrome version: Version 66.0.3359.181 (Official Build) (64-bit)

[patrickhulce]

Thanks for filing the report @rohit-nair!

You'll need to disable device emulation if you want to use a custom user agent. Part of device emulation is overriding the user agent with that of a mobile device. lighthouse --disable-device-emulation --chrome-flags="--user-agent=lighthousebot" works.

[rohit-nair]

Thanks @patrickhulce. Can you point me to a resource that would help me setup device emulation along with a custom, whitelisted (in bot manager) user agent?

[patrickhulce]

Keep in mind once you start changing user agent it can become clear to the site that you're not mobile anymore ;) things may not work as intended.

If it's just screen size you're after, you can get away with just another chrome flag --window-size=412,732.

For the rest of emulation (primarily Device Pixel Ratio), you'll need to setup some custom logic to change the target when it loads. You might be interested in https://github.com/GoogleChrome/lighthouse/blob/master/docs/puppeteer.md#inject-jscss-before-the-page-loads, at that point you can setup your custom device emulation/user-agent with puppeteer.

[brendankenny]

@patrickhulce @paulirish the UA string in the LHR is retrieved from the browser, which means the chrome flag here is still having an effect. Does this mean navigator.userAgent in the page can be different than what the browser is sending with requests due to emulation? We should probably fix that fixme and move to Browser.getVersion sooner rather than later, then :)

[patrickhulce]

Does this mean navigator.userAgent in the page can be different than what the browser is sending with requests due to emulation?

This is always true, we fetch from the browser before we enable emulation so we don't get back the same emulated UA for everything :)

[brendankenny]

I thought we fixed that...

On Mon, Jun 4, 2018, 12:44 Patrick Hulce [e-mail] wrote:

Does this mean navigator.userAgent in the page can be different than what
the browser is sending with requests due to emulation?

This is always true, we fetch from the browser before we enable emulation
so we don't get back the same emulated UA for everything :)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/GoogleChrome/lighthouse/issues/5406#issuecomment-394421764,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATV2zpJejW74B3BsiBZ30DX0nAaDqisks5t5WPzgaJpZM4UXSGb
.

[brendankenny]

Oh, jk, it was the opposite. We actually wanted that behavior :)

Relevant emulation info would be in the runtime settings section.

On Mon, Jun 4, 2018, 12:55 Brendan Kenny [e-mail] wrote:

I thought we fixed that...

On Mon, Jun 4, 2018, 12:44 Patrick Hulce [e-mail] wrote:

Does this mean navigator.userAgent in the page can be different than what
the browser is sending with requests due to emulation?

This is always true, we fetch from the browser before we enable emulation
so we don't get back the same emulated UA for everything :)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
<
https://github.com/GoogleChrome/lighthouse/issues/5406#issuecomment-394421764
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AATV2zpJejW74B3BsiBZ30DX0nAaDqisks5t5WPzgaJpZM4UXSGb

.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/GoogleChrome/lighthouse/issues/5406#issuecomment-394425053,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATV20BpsQw3M_kkG1LC_sPYU8zejcaTks5t5WaQgaJpZM4UXSGb
.

[rohit-nair]

This may be what @brendankenny asked above, or may be I'm reading it wrong. Shouldn't the userAgent value in JSON report reflect the User-Agent that was actually used while making the request? Perhaps the userAgent value reside within the network-requests results along with statusCode etc.

[patrickhulce]

@rohit-nair this was the old behavior, but it's not that useful in most cases because we override the user agent for emulation. We were collecting the information of the device so that we could document the device/chrome version/OS/etc in the report, perhaps the name is a bit misleading though as you point out :)

You are also correct if you want to see what was requested you can look at the network request headers.

[rohit-nair]

Thinking of updating the ReadMe where it mentions --chrome-flags with the caveat regarding --user-agent and --disable-device-emulation. Thoughts?

[patrickhulce]

Hm, I'm not sure just that caveat has a place at the moment, but if you want to add a "configuring chrome" doc or something that has some FAQs about common flags like --headless, this pitfall, our default flags, the full list of available flags, etc, I think that'd be really great!

[wildlyinaccurate]

Resurrecting this old thread because I think there are perfectly valid reasons to want device emulation (throttling, viewport) with a custom user-agent (getting around bot detection with a whitelisted UA modifier).

Keep in mind once you start changing user agent it can become clear to the site that you're not mobile anymore ;) things may not work as intended.

This seems to be the primary reason for LH trumping the UA on emulated tests. Would it be fair to say that passing in Chrome flags is an indication that the user knows what they're doing? Webpage issues from a custom UA feel like an "expected" (or at least manageable) outcome. Chrome flags being ignored feels unexpected, and as far as I can tell doesn't have any manageable workaround.

[patrickhulce]

Webpage issues from a custom UA feel like an "expected" (or at least manageable) outcome

I would've agreed with you once upon a time but I've seen enough brittle UA regex issues now where the server flat out rejects connections that I'm no longer confident in what to expect from UA issues 😆 I agree if you know what you're doing, go for it, but a super visible CLI flag (even if you have to lookup chrome flags) feels too low of a bar for that.

as far as I can tell doesn't have any manageable workaround.

I think disableDeviceEmulation + custom emulation through puppeteer is still the workaround that we feel best balances concerns here.

You either use one of the two premade device profiles or you take control over the environment yourself and do anything you'd like.

The below feels manageable enough to get an exact replica of emulation going.

  browser.on('targetchanged', async target => {
    const page = await target.page();
    if (!page) return;
    await page.setViewport({
      width: 360, 
      height: 640, 
      deviceScaleFactor: 2.625, 
      isMobile: true, 
      hasTouch: true
    });
  });

Concerns

I'm not super comfortable saying in the report that a page load was using the stock emulated device profile when it wasn't.

When would we decide to not override the UA? The CLI is the only case where we know what flags Chrome was launched with, would the rest of Lighthouse integrations continue to ignore the established UA even in patterns that we are actually encouraging like puppeteer? Feels weird to special case a pattern of usage we would want to actively discourage while not handling the case we encourage, so would we need to develop guessing logic to see what Lighthouse thought was an intentional UA change and what wasn't? How does that affect Lighthouse when run on real devices already but emulation is requested?

Sidestepping all of these questions by saying either Lighthouse owned the emulation profile or it didn't again feels like an appropriate balance to me.