Aws-cdk: Docs should warn about ACM yearly certificate limit

[fogfish]

I do have fully automated pipelines that provision app stack including all required resources. The stack fails to deploy after a few destroy/deploy iterations:

Error: you have reached your limit of 20 certificates in the last year.

The failure is caused by

import * as acm from '@aws-cdk/aws-certificatemanager'

const cert = new acm.DnsValidatedCertificate(this, 'Cert', { domainName, hostedZone })

The error appears despite a fact that I don't have any certificates on my account.
It seems that AWS CDK consumes Certificate Request limit which is not decreased when stack is released. You can only resolve this by requesting increase of the limit via support centre. However, the hard quota limit is 1000 per account.

This implies that automation of certificates provision is not really an option for teams who does few deployments per days.

It would be extremely helpful to

• mention in the documentation of the package about limits
• advices how to overcome such limitations
• best practice of automated certificate provision

Environment

• CLI Version : 1.21.1
• Framework Version:
• OS : MacOS
• Language : TypeScript

---

This is :bug: Bug Report

[rix0rrr]

You are right, it would be helpful if the docs warned you about this potential failure condition.

A PR to that effect would be welcome.

[fogfish]

What is the best practice about cert automations in your opinion?

[metametadata]

I'm working on developing an automated CI/CD pipeline and while iterating often destroy/deploy the stacks with ApplicationLoadBalancedEc2Service. I'm now stuck with the error:

AWS::CloudFormation::CustomResource | Service/Certificate/CertificateRequestorResource/Default (ServiceCertificateCertificateRequestorResource0FC297E9) Failed to create resource. Error: you have reached your limit of 20 certificates in the last year.

Besides asking Amazon for the limit bump what else can I do? So far I can see these solutions:

• Somehow explicitly provide the pre-created wildcard certificate for all my services. And never delete this cert.
• If possible, ask CDK/CloudFormation to skip destruction of certs on stack destruction.

[fogfish]

The max hard limit is 1000 requests per year. It might not be enough at all to power CI/CD. I am struggle from same. I would strongly recommend first option, just provision wildcard cert and use it across stacks.

[metametadata]

Are you sure there's a hard limit of 1000? Looking at Service Quotas in console the default quota seems to be 2000 (not 20) and it's marked as "adjustable":

The similar observation concerning 20 vs. 2000 was made here: https://github.com/aws-quickstart/quickstart-redhat-openshift/issues/142#issuecomment-453643989.

[cdCarlos]

Same issue here. I do not have any certificate in my console and still getting the error. I'm using AWS CDK to manage my infrastructure. No way to continue.

[metametadata]

Just in case, for now I managed to resolve my issue by requesting a quota bump for ACM certificates created in last 365 days.

[BenoitDel]

same issue Here.

[AlexandreDeRiemaecker]

I have the same issue as described by @fogfish .
Using DnsValidatedCertificate and happily destroying and rebuilding environments in our DEV account - which is new and hasn't been used for anything else yet.

So yeah, after 20 times of deleting and creating new certificates, I bumped into the same error. My service quota is set to the default value - 2000.

The issue on AWS's side here - either an undocumented, invisible limit or the service-quota isn't respected I filed a support request. Unfortunately, the DEV account is on basic support plan. it might never be addressed.

I'm not keen on reproducing this on a PROD account with a premium support plan just for the sake of filing in a request with some support SLA.

It's sad because the POC for our CI/CD strategy with CDK worked well so far - we effortlessly spin up a new dedicated environment when PRs are opened and destroy them again when PR is merged.

[fogfish]

The warning to the developers is one issue but the actual recommendation is another aspect.

I've pivoted a few ideas about the solution and made a conclusion to use cross-stack references.
You can find more details with example in the post. I hope it helps you:
https://i.am.fog.fish/2020/03/18/how-to-fix-error-about-limits-of-tls-certificates-that-caused-by-aws-cdk.html

[AlexandreDeRiemaecker]

Thanks, nice article @fogfish .

In the meantime, AWS support fixed my issue by setting the quota to 2000 (which actually should be the default). The assumption that the default quota of 2000 is not used was true.

Under the lines, I got the following response: "[...] However please be noted that new AWS accounts may start with a quota lower than the default value[...]".

I gave the feedback that I'm not necessarily against newer account having further limitations, but then it should be visibile in the service quota console, at least. Also, it would be even better if the restriction is lifted when the account joins an older AWS organization (it's been a decade!).

Thanks for the nice article @fogfish . We'll probably use a wildcard certificate once we come near the limit of 2000. Because our AWS accounts are per team and environment, I'd say that staying under 2000 cert requests per year is doable. We build up a temporary environment when PRs are opened, and destroy it when it's merged. I guess It would take a lot to reach 2000 PRs in a year.

[kenkit]

Possible way to circumvent this is switching regions.
But remember you can run out of regions.

[BenoitDel]

Possible way to circumvent this is switching regions.
But remember you can run out of regions.

And you can't switch region for cloudfront certificate since the region certificate must be US-east region in this case.

[AlexandreDeRiemaecker]

@kenkit We raised the quota, so no need for workarounds. However, AWS currently shows 2000 as default quota, but that isn't really the case because there seem to be a hidden default quota of 20 on new accounts. For the CDK, this doesn't matter - AWS just has to be more transparent around the default quotas.

On a separate note for people using DnsValidatedCertificate, you should know that this news seems to indicate there is now an official solution in CloudFormation for Certificate Validation since 16/07/2020. However, I didn't get to test it yet.

[gregorypierce]

I just hit this one myself. The limit really is 20 certificates unless you get your quota raised with AWS. Gotta do better if we're going to CI/CD entire stacks.

[kenkit]

I just received 1k requests after requesting limit increase, all you have to do is request limit increase.

[barticus]

Yeah we had this issue earlier in the year. Not CDK related specifically but a bit crap that deleted certificates from failed CF attempts counts towards your default quota of 20. Also not sure if I got a dodgy support person but it took several days as they "had to consult with the ACM team to make sure their systems were protected" before increasing the limit to 40. Sounds very strange as I didnt even have 20 certificates as they'd been deleted.

[rayrapetyan]

It's Nov 2020, and this ridiculous limit is still there... What's that - an attempt to force one to move to an advanced support plan? It will never work, instead one will start looking for an alternate cloud provider!

[hugomallet]

VERY ANNOYING :rage:

[fogfish]

yeah, I am following my pattern of "creating certificate" once for application and re-use it across multiple deployments.

[hugomallet]

Hello,

Finally i solved this issue thanks to the aws developer support.
I was creating two certificates request in my stack. One in the region i use (us-west-2) and one for cloudfront ... in us-east-1 as required by cloudfront ...

So if you have this error, check your all your certificates resources in your stack and your quota in us-east-1 !

That would be very helpful if cdk mention the region in quota errors.
Also logging the id (not the logical id) of the resource would help to find the failing resource.
I create a feature request for this.

[yakim76]

Hi, @hugomallet. Looks like it is a region-specific error. In my case, even an attempt to request a certificate from AWS Web Console leads to the same error, and if chose any other region it works... For my accout, it is reproducible in eu-west-1 region.

[hugomallet]

@yakim76 What "ACM certificates created in last 365 days" quota do you see in the Service Quotas console in eu-west-1 ? 2000 or 20 ?

[mattvb91]

So what is the solution if you already have an existing certificate and want to reuse it without hitting this limit? We are doing this in CI and are hitting this error. Simply increasing it is not a viable option as we will hit it again later

[AlexandreDeRiemaecker]

@mattvb91 The default quota is in fact 2000 per year - which is vastly more than 20! However, for some unknown reasons, AWS is actually limiting to 20 (but the quota is displayed as 2000!) unless you ask them nicely. And you can probably ask for a lot more if you're a customer that can be trusted.

It took 2 business day to get my quota increase request handled, and it was in one DEV account where we didn't have a support plan (we only do that for production).

The limit of 2000 should be fine using separated accounts (not every team in one account) in your organization so you won't hit the limit that fast, and separated pre-production and production accounts also helps to not affect production if you ever actually hit the limit in your development environment (in which case you can probably increase the quota again). If you don't have that, well, you will probably have to give up on the idea of recreating certificates every time.

[mattvb91]

@AlexandreDeRiemaecker maybe i am using the wrong approach. We are provisioning *.mydomain.com wildcard certificates for customers to launch various endpoints so its out of our hands how many get created.

Now this wouldn't be an issue if the requests got reused for the same cert but it seems to count up all the time?

        // TLS certificate
        const certificate = new acm.DnsValidatedCertificate(this, 'SiteCertificate', {
            domainName: `*.${domain}`,
            hostedZone: zone,
            region: 'us-east-1', // Cloudfront only checks this region for certificates.
        });

Is there any way I can do new acm.FromExistingCert( "*.${domain}") to get around this?

[AlexandreDeRiemaecker]

@mattvb91 Actually, I don't really know for sure since I can afford to recreate every time.

I can see two potential ways:

1. Try to use Certificate.fromCertificateArn. Both Certificate and DnsValidatedCertificate implement ICertificate, which is probably all you need as reference for other constructs.
2. If that fails, I would probably create the certificate in a separate stack (that doesn't get updated often), pass over "certificate" as an argument and check if CDK actually manages to define this as a parameter in the generated CF templates (not so certain here if it works, I don't think I've ever tried to pass a construct based on a custom resource to another stack).

[mattvb91]

@AlexandreDeRiemaecker thank you Certificate.fromCertificateArn is indeed what I was looking for. Need to dig the documentation more. Thanks for the help, appreciated

[yakim76]

@yakim76 What "ACM certificates created in last 365 days" quota do you see in the Service Quotas console in eu-west-1 ? 2000 or 20 ?

2000 and actually there are no certificates in this region account at all. In my case, I've just move all infrs to the different region because it does not matter at this moment.

[wlarch]

I encountered the same issue while testing ACM certificates issuing for the same domain.
My service default quota value is identified as 2000. I requested an increase and I am awaiting a reply.