Terraform-provider-aws: aws_acm_certificate "completes" too soon with incorrect state

[BookOfGreg]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.12.6

Affected Resource(s)

• aws_acm_certificate

Terraform Configuration Files

resource "aws_acm_certificate" "new_cert" {
  domain_name = "my.domain.com"
  validation_method = "DNS"
}

resource aws_route53_record new_cert_validation {
  name = aws_acm_certificate.new_cert.domain_validation_options.0.resource_record_name
  type = aws_acm_certificate.new_cert.domain_validation_options.0.resource_record_type
  zone_id = aws_route53_zone.my_zone.id
  records = [aws_acm_certificate.new_cert.domain_validation_options.0.resource_record_value]
  ttl = 60
}

resource aws_acm_certificate_validation new_cert {
  certificate_arn = aws_acm_certificate.new_cert.arn
  validation_record_fqdns = [aws_route53_record.new_cert_validation.fqdn]
}

Expected Behavior

Apply successfully

Actual Behavior

Result of apply

aws_acm_certificate.new_cert: Creating...
aws_acm_certificate.new_cert: Creation complete after 3s [id=arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid]

Error: Invalid index

  on cert.tf line 7, in resource "aws_route53_record" "new_cert_validation":
   7:   name = "${aws_acm_certificate.new_cert.domain_validation_options.0.resource_record_name}"
    |----------------
    | aws_acm_certificate.new_cert.domain_validation_options is empty list of object

The given key does not identify an element in this collection value.

Terraform state after apply

$ terraform state show aws_acm_certificate.my_cert
# aws_acm_certificate.my_cert:
resource "aws_acm_certificate" "my_cert" {
    arn                       = "arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid"
    domain_validation_options = []
    id                        = "arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid"
    subject_alternative_names = []
    validation_emails         = []
    validation_method         = "NONE"
}

Terraform state after refresh

$ terraform refresh --target aws_acm_certificate.my_cert
aws_acm_certificate.my_cert: Refreshing state... [id=arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid]
$ terraform state show aws_acm_certificate.my_cert
# aws_acm_certificate.my_cert:
resource "aws_acm_certificate" "my_cert" {
    arn                       = "arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid"
    domain_name               = "my.domain.com"
    domain_validation_options = [
        {
            domain_name           = "my.domain.com"
            resource_record_name  = "_9ebd28a65e2e566debd548f2ece37969.my.domain.com."
            resource_record_type  = "CNAME"
            resource_record_value = "_d6bd37f616bbeebf692810a7758c448d.duyqrilejt.acm-validations.aws."
        },
    ]
    id                        = "arn:aws:acm:eu-west-2:my-account:certificate/my-cert-uuid"
    subject_alternative_names = []
    tags                      = {}
    validation_emails         = []
    validation_method         = "DNS"
}

Steps to Reproduce

1. terraform apply
2. terraform state show aws_acm_certificate.my_cert
3. terraform refresh --target aws_acm_certificate.my_cert
4. terraform state show aws_acm_certificate.my_cert

References

• #0000

[kherock]

I'm also still experiencing this with provider v2.25.0. This sounds like it was supposed to be fixed in v2.23.0 with #9598, but I can't get anything to work.

[katherinealbany]

I'm also still experiencing this with provider v2.26.0

[andremichi]

I'm also having this issue with provider v2.26.0.

[roylines]

Me too

[dvishniakov]

I've had a similar issue, but didn't bother to try to execute refresh command.
In my case, AWS domain whitelisting was missing for the specific domain in the specific account.

If you've not done that already, it is as simple as filing a support ticket for Certificate manager, Whitelisting domain. Just mention which one there. It might not be needed to wait for the Support response. In my case, they've acted much faster than responding in the ticket. Just try from time to time.

Other symptoms of this issue:

• Validation method always is saved as NONE (even though AWS API returns 'DNS' when you turn on TF debug log)
• Validation options are empty (even though AWS API returns them when you turn on TF debug log)
• You don't see the "Create record in Route 53" button in AWS Console when you 1) unfold the certificate 2) unfold the domain in that certificate
• Certificate status transitions to FAILED in a few seconds.

I've first experienced this issue in NEW account around Feb-March 2019. Since this issue is created in August - whitelisting could be the case. Yesterday it happened for me in an old account, which already had this for a few years: imported wildcard cert, delegated DNS zone, many and frequently changing DNS records for the domain. I was naive to think this whitelisting stuff will not affect grandfathered accounts.

Additional info from AWS Support:
Under some circumstances, the console's Create record in Route 53 button may not be available when you expect it. If this happens, check for the following possible causes.

You are not using Route 53 as your DNS provider.
You are logged into ACM and Route 53 through different accounts.
You lack IAM permissions to create records in a zone hosted by Route 53.
You or someone else has already validated the domain.

[nickdgriffin]

Having something very similar on 2.47 - my difference being that I'm updating an existing certificate resource to have a new SAN, so the "Invalid index" references the previous count of domain_validation_options as it has the old state.

It feels like there needs to be a check before this return statement after the certificate is requested that makes sure the "complete" resource is returned, with the expected fields, with a suitable retry on it: https://github.com/terraform-providers/terraform-provider-aws/blob/b7592c0b08b8ed2021ae35d32b6a3b655e09ef5d/aws/resource_aws_acm_certificate.go#L210-L219

[stekern]

I'm encountering the same issue when I try to add a domain to subject_alternative_names (which initially was set to an empty list). Here's a code example, and steps to reproduce:

1. Run terraform apply without setting var.external_domain
2. Set var.external_domain (thus modifying subject_alternative_names) and run terraform apply

I get a Error: Invalid index error on step 2:

Error: Invalid index

  on ../../../terraform-aws-static-site/main.tf line 46, in resource "aws_route53_record" "certificate_validation":
  46:   name            = aws_acm_certificate.certificate.domain_validation_options[count.index].resource_record_name
    |----------------
    | aws_acm_certificate.certificate.domain_validation_options is list of object with 1 element
    | count.index is 1

The given key does not identify an element in this collection value.

locals {
  domains = var.external_domain == "" ? [var.site_name] : [var.site_name, var.external_domain]
}

data "aws_route53_zone" "main" {
  name = var.hosted_zone_name
}

resource "aws_route53_zone" "external" {
  count = var.external_domain == "" ? 0 : 1
  name  = var.external_domain
  tags  = var.tags
}


resource "aws_acm_certificate" "certificate" {
  domain_name               = var.site_name
  subject_alternative_names = var.external_domain == "" ? [] : [var.external_domain]
  validation_method         = "DNS"
  provider                  = aws.virginia
  tags                      = var.tags

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "certificate_validation" {
  depends_on      = [aws_route53_zone.external, aws_acm_certificate.certificate]
  count           = length(local.domains)
  name            = aws_acm_certificate.certificate.domain_validation_options[count.index].resource_record_name
  type            = aws_acm_certificate.certificate.domain_validation_options[count.index].resource_record_type
  zone_id         = aws_acm_certificate.certificate.domain_validation_options[count.index].domain_name == var.site_name ? data.aws_route53_zone.main.zone_id : aws_route53_zone.external.0.zone_id
  records         = [aws_acm_certificate.certificate.domain_validation_options[count.index].resource_record_value]
  ttl             = 60
  allow_overwrite = true
}

[chrismillah]

Also experiencing this with creating aws ACM cert with SANS, i've had to previously work around by creating the cert first then uncommenting my SAN validation and rerunning it.

I've even tried adding some depends_on to wait for the cert to be created first, no dice..

What is the proper approach here?

resource "aws_acm_certificate" "cert" {
  domain_name               = "XXX.com"
  validation_method         = "DNS"
  subject_alternative_names = ["XXX1.com", "XXX2.com"]
  tags                      = local.common_tags

  lifecycle {
    create_before_destroy = false
  }
}

resource "aws_route53_record" "cert_validation" {
  name            = aws_acm_certificate.cert.domain_validation_options.0.resource_record_name
  type            = aws_acm_certificate.cert.domain_validation_options.0.resource_record_type
  zone_id         = "XXXX"
  records         = [aws_acm_certificate.cert.domain_validation_options.0.resource_record_value]
  ttl             = 60
  allow_overwrite = true
}

resource "aws_route53_record" "app_cert_validation_san" {
  name       = aws_acm_certificate.cert.domain_validation_options.1.resource_record_name
  type       = aws_acm_certificate.cert.domain_validation_options.1.resource_record_type
  zone_id    = "XXX"
  records    = [aws_acm_certificate.cert.domain_validation_options.1.resource_record_value]
  ttl        = 60
  depends_on = [aws_acm_certificate.cert]
}


resource "aws_route53_record" "app_cert_validation_san_2" {
  name       = aws_acm_certificate.cert.domain_validation_options.2.resource_record_name
  type       = aws_acm_certificate.cert.domain_validation_options.2.resource_record_type
  zone_id    = "XXX"
  records    = [aws_acm_certificate.cert.domain_validation_options.2.resource_record_value]
  ttl        = 60
  depends_on = [aws_acm_certificate.cert]
}

resource "aws_acm_certificate_validation" "cert" {
  certificate_arn = aws_acm_certificate.cert.arn
  validation_record_fqdns = [
    aws_route53_record.cert_validation.fqdn,
    aws_route53_record.app_cert_validation_san,
    aws_route53_record.app_cert_validation_san_2
  ]

}

Error: Invalid index

on certificates.tf line 23, in resource "aws_route53_record" "app_cert_validation_san":
23: name = aws_acm_certificate.cert.domain_validation_options.1.resource_record_name
|----------------
| aws_acm_certificate.cert.domain_validation_options is list of object with 1 element

The given key does not identify an element in this collection value.

[stekern]

I think I might have discovered a temporary fix using Terraform's try lookup function, but it feels hacky. The way I see it, the worst case scenario of using try lookup in this way is duplicate validation records being created (and I assume records within a zone needs to be unique, so Terraform will in this case spit out an error). In practice, however, the correct values are always being used (I've experimented with adding and removing entries from var.domain_zones -- using domains spanning different hosted zones) and I haven't encountered any errors yet.

Is this a viable fix until the redesign is finished, or are there any critical gotcha's I should be aware of here?

# module/main.tf
locals {
  # var.domain_zones = {
  #   "domain.com" = "<hosted-zone-id>"
  #   "domain.net"   = "<hosted-zone-id>"
  #   "my.domain.com" = "<hosted-zone-id>"
  # }
  domains = sort(keys(var.domain_zones))
  validation_options_by_domain_name = {
    for opt in aws_acm_certificate.cert_website.domain_validation_options : opt.domain_name => merge(opt, {
      # Fallback value will be used when domain_validation_options references 
      # a domain_name that has been removed from var.domain_zones
      zone_id = lookup(var.domain_zones, opt.domain_name, keys(var.domain_zones)[0])
    })
  }
}

resource "aws_acm_certificate" "this" {
  domain_name               = local.domains[0]
  validation_method         = "DNS"
  subject_alternative_names = slice(local.domains, 1, length(local.domains))
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "this" {
  # Fallback values will be used when domain_validation_options is not up-to-date
  depends_on      = [aws_acm_certificate.cert_website]
  for_each        = var.domain_zones
  name            = lookup(local.validation_options_by_domain_name, each.key, values(local.validation_options_by_domain_name)[0]).resource_record_name
  type            = lookup(local.validation_options_by_domain_name, each.key, values(local.validation_options_by_domain_name)[0]).resource_record_type
  zone_id         = lookup(local.validation_options_by_domain_name, each.key, values(local.validation_options_by_domain_name)[0]).zone_id
  records         = [lookup(local.validation_options_by_domain_name, each.key, values(local.validation_options_by_domain_name)[0]).resource_record_value]
  ttl             = 60
  allow_overwrite = true
}

[thomaspicquet]

Thanks @stekern, I was able to use your method with a cloudflare record:

locals {
    validation_options_by_domain_name = {
        for opt in aws_acm_certificate._.domain_validation_options : opt.domain_name => opt
    }
}
resource "cloudflare_record" "validation" {
    domain = var.cf_domain
    name   = replace(lookup(local.validation_options_by_domain_name, var.cf_domain, values(local.validation_options_by_domain_name)[0]).resource_record_name, format(".%s.", var.cf_domain), "")
    type   = lookup(local.validation_options_by_domain_name, var.cf_domain, values(local.validation_options_by_domain_name)[0]).resource_record_type
    value  = lookup(local.validation_options_by_domain_name, var.cf_domain, values(local.validation_options_by_domain_name)[0]).resource_record_value
    ttl    = 120
}