Boulder: "Too many certificates already issued for domain" during issuance, after no new issuance and many renewals this week

[dusek]

Hello,

we think we are running into some rate limiting bug in the Let's Encrypt server.

We have around 28 certificates, out of which some (a lot) contain the domain "brailcom.org". These certificates were issued a few months ago (around June 2016), and only around 1-2 (if any) new certificates containing the domain name prefix "brailcom.org" were issued since then.

We renew all these certificates every two days. This works, as given the rate limits, this goes under the "Renewal Exemption" column (each cert is renewed approx. 3.5 times per week, below the allowed limit of 5 times a week).

However when creating a new certificate today (i.e. not renewing) containing the domain prefix "brailcom.org", with domain names previously unused with Let's Encrypt, we, to our surprise, ran into a rate limit:

$ certbot-auto certonly --domains nepodpis.cz,www.nepodpis.cz,nepodpis.brailcom.cz,nepodpis.brailcom.org
Saving debug log to /path/to/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for nepodpis.cz
http-01 challenge for www.nepodpis.cz
http-01 challenge for nepodpis.brailcom.cz
http-01 challenge for nepodpis.brailcom.org
Using the webroot path /path/to/webroot for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /path/to/keys/1722_key-certbot.pem
Creating CSR: /path/to/csr/1722_csr-certbot.pem
An unexpected error occurred:
There were too many requests of a given type :: Error creating new cert :: Too many certificates already issued for: brailcom.org
Please see the logfiles in /path/to/letsencrypt_log_dir for more details.

I would understand that if renewals are exempt from the "Certificates per domain" limit, I could well be renewing e.g. 1000 certs, each with a single subdomain of a certain registered domain, every 2 days, and still be able to issue 20 new certificates per week containing a new subdomain of the said registered domain. I.e. that using the "Renewal exemption" to its full should not conflict with being able to issue new certificates, that the two things should be orthogonal.

I have an output of the above command with -vvvvvvv verbosity flag saved in a file. If it would help, I could send it over GPG (with unaltered paths) - in such case please provide the appropriate email address and the GPG key, or some other secure means (but maybe the information in this issue is already sufficient enough as it is and the verbose output might not be necessary).

Happens with certbot 0.9.1. Not sure if it happened with 0.8.1, as we did not issue many (or none) new domains before upgrading from it to 0.9.1.

Perhaps somewhat related: #1925

[jsha]

There's a bit of a quirk in how the exemption is implemented. If you've already issued 20 new certificates during a week, you can still renew. However, if you perform the renewals first, that counts towards your 20. I realize this is a bit awkward, but we're unlikely to change it. My recommendation would to switch to a renewal frequency that more closely matches the certificate expiration. For instance, our current certificate expiration is 90 days, and we recommend attempting renewal for each certificate when there are 30 days left on the certificate. That should reduce issues you see with new issuance. Maybe you could share more details about why you use such a high renewal frequency?

[dusek]

Thanks. The high frequency was mostly due to us wanting to "stress test" the renewals so that we would learn about any problems quickly and not 60 days after deploying Let's Encrypt (which was for the first time, at least at such scale). However that already went well, and I simply did not get to easing up the renewal frequency (with e.g. 30 or 60 days renewal frequency, I wanted to think about which (less important) certificates should be renewed first as "testing" ones, and which more important certificates e.g. a few days later, and simply did not get to deliberate about that; now I realize that is not necessary at all, as if a renewal would fail, certbot would try again the very next day).

So to solve this, I think I will have to finally find some time to change the renewal rate to something normal.

However if I think about how it works: if someone had many certificates for a certain registered domain, then they would have to run into a situation where a certain 7-day window would not be good for issuing new certs: they could use the possibility of the renewal exemption to do all renewals e.g. a single day every 60 days (so as not to spread out the renewals all over time and therefore to disable issuing new certs altogether), which would leave most of the 60 days (53 days to be exact) OK for creating new certs, however the 7 days after the "mega renewal day" would always be those where one would not be able to issue new certs (e.g. imagine one has to quickly setup a secure website and time is of essence - there would not be an option to wait 7 days). So I just note it here that easing up the renewal frequency, unless I am missing something, might not be a universal enough solution for someone with many more certificates per registered domain, so one would still have to care about how many certificates per registered domain they have (except for their initial issuance which can be done at rate 20 certs per week), how they are grouped, etc., i.e. something I hoped the Renewal Exemption would make a thing of the past.

[dusek]

@jsha Also I now realized this one more scenario than the one mentioned in my response above:

One uses the "mega renewal day" model mentioned above to renew all certs in a single day every 60 days, so as to leave a 53 day window per 60 days usable for renewals. But when a need comes for a issuing a new cert e.g. in the middle of the 53 day window, one would need to then arrange for this new cert renewal to "align" with (happen on) the "mega renewal day", so as to keep the 53 day window for issuing new certs functional for anyone else. If one did not do that, the cert would renew every 60 days from the day it was initially issued, which was not on the "mega renewal day".

Maybe I am overthinking all of this and in practice this scenario will not happen (there will not be that many certificates per registered domain), but we are not a big organization so I cannot speak from practice here.

[jsha]

You do have a good point, that it's tricky to align all your renewals on the same day so as to take better advantage of rate limiting. We should think about ways for clients to make this easier, and possibly document best practices. Thanks a lot for your feedback on this, it's helpful!

[nneul]

Unfortunate problem is that even with the suggested 60 days - you're still talking about the service basically becoming unusable for new certs after you reach around 160 in total since you'll be renewing 20 a week on average at that point. The 'mega renew day' would work, but that assumes a central point of control.

[doits]

Just hit this too. Maybe that renewals still "eat up" your weekly certificate limit should be documented explicitly on https://letsencrypt.org/docs/rate-limits/, because imo it is counter-intuitive and one would not expect that by reading that page.

[nneul]

Pretty sure I submitted a merge request for that page's content with suggested text for that, but never saw any action/response.

[nneul]

Might have only been a support/feedback request - can't find the reference now.

[MikeLund]

If you perform the renewals first, that counts towards your 20. I realize this is a bit awkward, but we're unlikely to change it.

Is there a reason for this, or just not a high technical priority?

I've been using LE on 85 different subdomains since last year, and this situation bites me hard. My renewals happened automatically during this week, and now when I need to create a new certificate for a new subdomain, I'm blocked out. Obviously we're an edge-case, but this policy confuses me. Would be great if renewals didn't eat from creation ratelimiting...

[cpu]

@MikeLund You'll be interested in seeing this API announcement I posted yesterday.

The tl;dr is that the constraint has been removed from the code and renewals will act the way you're hoping once the configuration flag is activated in the production environment. That might be as early as today but I would monitor the API announcement I linked for the definitive source.

Hope this helps,

[nneul]

Yay!