Lighthouse: Cannot collect data from PGSI API

Created on 15 Jan 2020 · 15Comments · Source: GoogleChrome/lighthouse

Hi,

We are observing the following errors in the API calls (and the PSI site as well occasionally) every 8 hours since the 4th of December on these specific URLs:

We also have encountered errors since the 10th of August on this URL:

https://www.cultura.com/ https://developers.google.com/speed/pagespeed/insights/?hl=fr&url=https%3A%2F%2Fwww.cultura.com%2F

Most of the time, we are getting a 403 error from Lighthouse: ERRORED_DOCUMENT_REQUEST.

This is the methodology we use to pull the data:

We are using the following API key: AIzaSyBzWhzUwp4PRaC9oT91v4-7D-Vfiv8faIk
The API is queried every 8 hours on 3 pages of 31 sites for mobile and desktop
Each query is spaced in time with a delay of 100ms

To sum up: every 8 hours, 186 requests spaced by 100ms are being sent using the above key.

Any help is appreciated, thank you
Best regards

P1.5 PSLR bug

Source

fxbtr

Most helpful comment

It was removed 👍

exterkamp on 9 Jun 2020

👍2

All 15 comments

Thanks chief! Appreciate you filing this bug. :clap:

This is a known issue, most well described in #2784. So, we'll automatically close this as a duplicate.

_However_, if you believe your bug is different than the cases described there, please comment here with "necessarily-wide-alpaca" and I'll reopen this bug. :robot: Beep beep boop.

devtools-bot on 15 Jan 2020

necessarily-wide-alpaca

fxbtr on 15 Jan 2020

ERRORED_DOCUMENT_REQUEST means that the server replied to our requests with an error status code, the most common codes are described by #2784 and this appears to be a duplicate.

Given that it more commonly occurs with your high volume API request keys and the x-datadome header present in your responses, it looks like server is rejecting PSI requests as bot and/or DDOS protection.

patrickhulce on 15 Jan 2020

ERRORED_DOCUMENT_REQUEST means that the server replied to our requests with an error status code, the most common codes are described by #2784 and this appears to be a duplicate.

Given that it more commonly occurs with your high volume API request keys and the x-datadome header present in your responses, it looks like server is rejecting PSI requests as bot and/or DDOS protection.

Thanks for your reply.

We found out that we are getting a 429 TooManyRequests status code, however we don't appear to be exceeding our quota:

It is my understanding that we are getting a 429 status code but the actual reason we cannot retrieve any data is the reason you provided. Am I correct?

Beyond this point, how can we resolve the issue with the server flagging us as bot and/or DDOS protection?

Best regards

fxbtr on 17 Jan 2020

how can we resolve the issue with the server flagging us as bot and/or DDOS protection?

When you receive ERRORED_DOCUMENT_REQUEST, PSI is not flagging you as bot or DDOS protection you are flagging PSI as a bot and erroring the document. This seems to be exactly what your datadome service is designed to do. Work with them to allow Chrome-Lighthouse requests through their wall.

patrickhulce on 17 Jan 2020

Work with them to allow Chrome-Lighthouse requests through their wall.

Thanks again we'll look into it.

Could the ERRORED_DOCUMENT_REQUEST errors that occurred earlier last month be related to the lastest issue that I mention in my previous post?

To sum up

we are getting ERRORED_DOCUMENT_REQUEST occasionally through https://developers.google.com/speed/pagespeed/insights/
we are getting 429 Too Many Requests systematically since last week via our microservice calling the PSI API with the API key given earlier (and also a regenerated one), despite not reaching our quota

fxbtr on 17 Jan 2020

cc @exterkamp for these 429 questions

patrickhulce on 17 Jan 2020

Hi,

Do you have any insights you could share with us @exterkamp?

Thanks,
Best

fxbtr on 20 Jan 2020

Hi, yup, so the default 100 second QPS is 400 per project, and 60 per user. So, depending on how you are making these requests, 186 with a 100ms spacing (effectively 1000req/100s) is too much. I would suggest spreading out the requests over a longer period to avoid this.

That screenshot is interesting, I'm curious if it still appears that way. We had a recent change to our quota management system so that should possibly be tracking more accurately and have the correct default quota.

exterkamp on 23 Jan 2020

Hi,
thanks I dramatically increased the time between requests, from 100ms to 5 sec. I will let you know if we have better results this way.

As for the screenshot, this is how it looks like right now,

Here is an additional one, as you can see I have been trying a different key recently

fxbtr on 23 Jan 2020

Hi,
I am afraid it didn't work. Let me know if the process is valid: I waited for the daily quota reset at 00:00 PT and then spaced requests to reach 3req/s at a maximum. I ended up with SC 429 and 500.

Is there a way for you to check the actual quota for our key, and how we are using it?
The regenerated key is: AIzaSyBml9ALcG8fu_HRH2vdGwAp3EhC7cg-StU

Thanks

fxbtr on 24 Jan 2020

I can look your quota up by project-id, not by api key. If you want you can email me you GCP project-id to [email protected].

At 3 requests per second that is going to be ~300 requests per 100 seconds, which is greater than the per user quota, so it depends on how you are querying this data. If it is all coming from one machine ("user") then this would be expected because it is >60 req/s per user. If it is distributed over your whole project then that is less expected :confused:

exterkamp on 24 Jan 2020

mail sent

fxbtr on 27 Jan 2020

Resolution here is that per-user quota will be removed in PSI

patrickhulce on 25 Feb 2020

👍1

It was removed 👍

exterkamp on 9 Jun 2020

👍2

Was this page helpful?

0 / 5 - 0 ratings