At Heroku, we've been seeing the following frequently in our error logs:
acme: Error 429 - - <html> <head><title>429 Too Many Requests</title></head> <body bgcolor="white"> <center><h1>429 Too Many Requests</h1></center> <hr><center>nginx</center> </body> </html>
My understanding after speaking with LE is that this is the Overall Rate limit of 20 reqs/second.
We had hoped that ee0018c855fa43a43f29280a5be2e4b94f9ac991 and 45beff7ed309e190408d7ac1c89a7b7751596a73 would address this issue, but there still seems to be a problem.
CC @jsha
A quick look at the logs shows that you do seem to successfully stay under 20 reqs/second in general. How many times per day do you get this error? Can you share the IP addresses from which you get it?
@jsha thanks for looking into this, we are seeing this 2-3 times a day in production.
Unfortunately, that is the only information we get back in the error so I won't be able to provide the IP addresses.
If it's only 2-3 times a day, I wouldn't worry about it too much. It could be slight differences between xenolf/lego's internal rate limiting implementation and how we count rate limits on the server side. If you see an increase, please let us know!
We're now seeing it 7-9 times a day the past two days.
@jsha I can easily trigger it on staging with a fixed IP, could it be useful for you ?
Yeah, that would! Especially if you can provide a little Go script that triggers it.
Anything we can do to help with this... still seeing a fair amount of 429 errors and would love to help clear these up.
We're still working on diagnosing this, thanks for the update that it's still causing you issues!
I believe we've fixed this server-side: https://community.letsencrypt.org/t/html-style-rate-limit-responses/34146.
Proposing to close this. What do you all think?
I think we can. If someone encounters this problem again please open a new issue.