Lighthouse: Lighthouse doesn't send Accept request header in start_url audit

[optimalisatie]

Hi!

We are using a service worker that serves HTML pages from cache when the request contains the header Accept: text/html.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept

We discovered that Lighthouse does not send this header in it's audit of the start_url causing the test to result in a false negative.

Failures: Service worker does not successfully serve the manifest's start_url.

[kdzwinel]

Lighthouse is doing a simple fetch for that resource:
https://github.com/GoogleChrome/lighthouse/blob/550b0c40ecbc65f1b132298e3d1a1e1744191378/lighthouse-core/gather/gatherers/start-url.js#L20-L26

By default fetch sends Accept: */* (spec) while for navigation requests it sends Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. In case of start_url audit we want to behave as a navigation request, so this is indeed a LH bug.

As far as I understand, there is no way to change the destination of the fetch request directly, so we should just fake it by sending appropriate headers.

[patrickhulce]

Thanks for filing! This is indeed another wrinkle of us using fetch instead of a browser navigation.

Related #2688

[kdzwinel]

Maybe that's a crazy idea, but how about injecting an iframe with onload/onerror callback? That would give us a real navigation request and simple yes/no answer to a "is start_url cached?" question (I assume we don't really care for the exact status code).

[patrickhulce]

Oh good idea @kdzwinel! Might face other challenges with iframe restriction but might work too

[kdzwinel]

CSP can block an iframe, but TBH the fetch call that we currently have can also be blocked by CSP (connect-src 'none').

As far as I can see we have these options:

1. keep fetch, but add headers
   
   
   
   • 👍 simple
   
   
   • 👎 not a navigation request, can be blocked by CSP
   
   
2. switch to iframe
   
   
   
   • 👍 simple, proper navigation request
   
   
   • 👎 can be blocked by CSP, we are injecting something to the page
   
   
3. CacheStorage.requestCachedResponse
   
   
   
   • 👍 simple, navigation request issue doesn't apply, can't be blocked by CSP
   
   
   • 👎 will produce false negatives if SW is doing something smarter than just caching whole page
   
   
4. navigate main frame to start_url (or open a new tab? offscreen window?)
   
   
   
   • 👍 simple, proper navigation request, can't be blocked by CSP
   
   
   • 👎 audit takes longer, ?? (I'm probably missing something here)
   
   

We can also do a combination of first three, but it will increase complexity and still won't cover 100% of the cases. IMHO last one is probably the cleanest one, but there is probably some good reason (that I don't see right now) why it haven't been chosen in the first place.

[wardpeet]

@kdzwinel I believe step 4 isn't as easy as it is because it will need a pass on it's own to make sure that hte page gets loaded. Doing it in the same pass with other audits will skew the results and can also result in a different final url.

[patrickhulce]

Yeah 4 is the obvious best, but we haven't done it mostly because it's slow to add another pass. #4310 should help with the slowness, so after it lands we might be good to just switch to that 👍

[exterkamp]

We do #️⃣ 4 now. #5159.