terraform updates aws_cloudfront_distribution every time over and over again

[BerndWessels]

Hi there,

Terraform Version

Terraform v0.6.16

Affected Resource(s)

• aws_cloudfront_distribution

It may be an issue with Terraform's core.

Terraform Configuration Files

/**
 * AWS region and credentials.
 */
provider "aws" {
  region = "${var.aws_region}"
  access_key = "${var.aws_access_key}"
  secret_key = "${var.aws_secret_key}"
}
/**
 * Set of nameservers used for the platform domain.
 */
resource "aws_route53_delegation_set" "PlatformDelegationSet" {
  reference_name = "${var.environment}"
}
/**
 * Hosted zone used for the platform domain.
 */
resource "aws_route53_zone" "PlatformHostedZone" {
  name = "${var.platform_domain}"
  delegation_set_id = "${aws_route53_delegation_set.PlatformDelegationSet.id}"
}
/**
 * CNAME record used by the email provider to verfify the domain ownership.
 */
resource "aws_route53_record" "PlatformMailVerificationRecord" {
  zone_id = "${aws_route53_zone.PlatformHostedZone.zone_id}"
  name = "${var.email_domain_verification_cname_name}"
  type = "CNAME"
  ttl = "300"
  records = [
    "${var.email_domain_verification_cname_value}"]
}
/**
 * MX record pointing to the email provider.
 */
resource "aws_route53_record" "PlatformMailMxRecord" {
  zone_id = "${aws_route53_zone.PlatformHostedZone.zone_id}"
  name = "${aws_route53_zone.PlatformHostedZone.name}"
  type = "MX"
  ttl = "300"
  records = [
    "${var.email_mx_record_values.1}",
    "${var.email_mx_record_values.2}"]
}
/**
 * S3 bucket hosting the platform website.
 */
resource "aws_s3_bucket" "PlatformWebsiteBucket" {
  bucket = "${var.platform_domain}"
  acl = "public-read"
  policy = "${file("./Websites/Marketing/BucketPolicy.json")}"
  website {
    index_document = "index.html"
    routing_rules = "${file("./Websites/Marketing/RoutingRules.json")}"
  }
}
/**
 * CloudFront distribution of the platform website.
 */
resource "aws_cloudfront_distribution" "PlatformWebsiteDistribution" {
  aliases = [
    "${var.platform_domain}",
    "www.${var.platform_domain}"]
  comment = "Managed by Terraform"
  default_cache_behavior {
    allowed_methods = [
      "GET",
      "HEAD"]
    cached_methods = [
      "GET",
      "HEAD"]
    compress = true
    default_ttl = 30
    forwarded_values {
      cookies {
        forward = "none"
      }
      query_string = false
    }
    max_ttl = 86400
    min_ttl = 0
    target_origin_id = "S3-${var.platform_domain}"
    viewer_protocol_policy = "allow-all"
  }
  default_root_object = "index.html"
  enabled = true
  origin {
    domain_name = "${aws_s3_bucket.PlatformWebsiteBucket.bucket}.s3.amazonaws.com"
    origin_id = "S3-${var.platform_domain}"
    s3_origin_config {
    }
  }
  price_class = "PriceClass_All"
  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }
  viewer_certificate {
    acm_certificate_arn = "${var.platform_domain_certificate_arn}"
    minimum_protocol_version = "TLSv1"
    ssl_support_method = "sni-only"
  }
}
/**
 * Records routing to the CloudFront distribution of the platform website.
 */
resource "aws_route53_record" "PlatformWebsiteRecord1" {
  zone_id = "${aws_route53_zone.PlatformHostedZone.zone_id}"
  name = "${aws_route53_zone.PlatformHostedZone.name}"
  type = "A"
  alias {
    name = "${aws_cloudfront_distribution.PlatformWebsiteDistribution.domain_name}"
    zone_id = "${aws_cloudfront_distribution.PlatformWebsiteDistribution.hosted_zone_id}"
    evaluate_target_health = false
  }
}
resource "aws_route53_record" "PlatformWebsiteRecord2" {
  zone_id = "${aws_route53_zone.PlatformHostedZone.zone_id}"
  name = "www.${aws_route53_zone.PlatformHostedZone.name}"
  type = "A"
  alias {
    name = "${aws_cloudfront_distribution.PlatformWebsiteDistribution.domain_name}"
    zone_id = "${aws_cloudfront_distribution.PlatformWebsiteDistribution.hosted_zone_id}"
    evaluate_target_health = false
  }
}

Debug Output

terraform plan and terraform apply show this after the first successful terraform apply (which should not be the case):

~ aws_cloudfront_distribution.PlatformWebsiteDistribution
    origin.1479266234.custom_header.#:                                    "0" => "0"
    origin.1479266234.custom_origin_config.#:                             "0" => "0"
    origin.1479266234.domain_name:                                        "" => "mydomain.nz.s3.amazonaws.com"
    origin.1479266234.origin_id:                                          "" => "S3-mydomain.nz"
    origin.1479266234.origin_path:                                        "" => ""
    origin.1479266234.s3_origin_config.#:                                 "0" => "1"
    origin.1479266234.s3_origin_config.2547889144.origin_access_identity: "" => ""
    origin.4020884933.custom_header.#:                                    "0" => "0"
    origin.4020884933.custom_origin_config.#:                             "0" => "0"
    origin.4020884933.domain_name:                                        "mydomain.nz.s3.amazonaws.com" => ""
    origin.4020884933.origin_id:                                          "S3-mydomain.nz" => ""
    origin.4020884933.origin_path:                                        "" => ""
    origin.4020884933.s3_origin_config.#:                                 "0" => "0"

Expected Behavior

The first time running terraform apply it should create the aws_cloudfront_distribution.
The next time running terraform apply it should NOT update the aws_cloudfront_distribution.

Actual Behavior

The first time running terraform apply it creates the aws_cloudfront_distribution.
The next time running terraform apply it UPDATES the aws_cloudfront_distribution.

This happens even if there is no change at all to the Configuration Files :(

Steps to Reproduce

1. terraform apply
2. wait until successfully finished
3. terraform apply
4. wait until successfully finished
   

   Important Factoids

So in other words:

Terraform successfully creates the aws_cloudfront_distribution.
But the next time running terraform apply it somehow believes there are changes and updates the Cloudfront Distribution in AWS again.
This is unnecessary and causes the Distribution to deploy again which always takes a long time.

I think this is a really serious issue.

[ksatirli]

Can confirm this is also present on 0.7.0; actual behavior is identical.

Happy to provide my TF resources for this as well, but @BerndWessels already outlined everything perfectly.

[gunzy83]

Confirming that I have run into the same issue with 0.7.0 as well. I see the same behaviour in the output except the numbers after origin. are different. It does not matter if I change this in terraform.tfstate, the numbers remain static and I am yet to work out where they come from and why it looks like an update needs to happen on each run.

[gunzy83]

If you specify a origin_access_identity inside s3_origin_config it works as expected (idempotent). It seems that the bug only happens if the origin_access_identity is empty or not defined.

[BerndWessels]

Awesome, thank you so much @gunzy83 to figure that out. I suppose that is rather a workaround than a fix but it makes me totally happy.
I think it would be good if that could make it into the official terraform cloudfront docs as a note.

[gunzy83]

@BerndWessels No worries. Yeah it is a workaround so as @erutherford said this is definitely a bug and needs to be resolved (I need to learn Go so I can make a PR).

[erutherford]

Sorry @gunzy83 , my comment was incorrect, I was mixing up custom_origin_config and s3_origin_config. I should have updated my comment instead of deleting it, sorry about that. It's odd though, I didn't start seeing this bug until I added custom_origin_config and I'm not even specifying s3_origin_config. Let me try adding the work around you mentioned and see if that fixes it.

resource "aws_cloudfront_distribution" "bucket_cloudfront"{
  origin {
    domain_name = "${aws_s3_bucket.bucket.id}.${lookup(var.aws_s3_endpoints, var.aws_region)}"
    origin_id = "${aws_s3_bucket.bucket.id}"

    custom_origin_config {
      http_port = 80
      https_port = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols = ["TLSv1", "TLSv1.1", "TLSv1.2"]
    }
  }

[erutherford]

I think my issue is a different bug :/ going to open a different ticket

[gunzy83]

Ahh sorry @erutherford, didn't parse the first part of your comment properly, my bad. This is odd because I have not seen the issue with custom_origin_config but only with s3_origin_config without origin_access_identity.

[erutherford]

@gunzy83, I'm batting 0 for 2, after some more testing it looks like it was an issue with the viewer_certificate config section, I just didn't see the issue until after the custom_origin_config change. I created an issue for it, but I have a not so good work around for now.

[nungster]

I too can confirm it is happening with Terraform 0.7.0.

Happy to provide my TF resources for this as well, but @BerndWessels already outlined everything perfectly.

This is an issue since cloudfront take quite a while to update once a change is initiated.

[BabakMN]

Can confirm same issue is present here on v0.7.0. What's the current workaround?

[llarsson]

The current workaround is as @gunzy83 found out, as far as I can tell. Throw something like this in, and your CloudFront seems to stabilize:

resource "aws_cloudfront_origin_access_identity" "origin_access_identity" {
  comment = "Terraform issue 7930 workaround"
}

Of course you also need to refer to it in your distribution definition.

[vancluever]

@BerndWessels, I noticed you are providing an empty s3_origin_config. This should work without providing this. In fact, the first acceptance test uses the below config:

resource "aws_cloudfront_distribution" "s3_distribution" {
  origin {
    domain_name = "${aws_s3_bucket.s3_bucket.id}"
    origin_id = "myS3Origin"
  }
  enabled = true
  default_root_object = "index.html"
  logging_config {
    include_cookies = false
    bucket = "mylogs.${var.rand_id}.s3.amazonaws.com"
    prefix = "myprefix"
  }
  aliases = [ "mysite.${var.rand_id}.example.com", "yoursite.${var.rand_id}.example.com" ]
  default_cache_behavior {
    allowed_methods = [ "DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT" ]
    cached_methods = [ "GET", "HEAD" ]
    target_origin_id = "myS3Origin"
    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }
    viewer_protocol_policy = "allow-all"
    min_ttl = 0
    default_ttl = 3600
    max_ttl = 86400
  }
  price_class = "PriceClass_200"
  restrictions {
    geo_restriction {
      restriction_type = "whitelist"
      locations = [ "US", "CA", "GB", "DE" ]
    }
  }
  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

Have you tried this without the empty s3_origin_config structure?

I agree that it's not intuitive that the structure can be added empty - I will work on a fix that ensures both that and the custom_origin_config structures are validated.

PS: I've verified that the above config does create a origin of type S3.

[vancluever]

Everyone,

This actually may be due to an incorrect default for origin_access_identity. Prelim tests to change this to a required value seem to be ok, I will do more testing before putting in a PR.

Will update when it's ready!

[llarsson]

@vancluever while that may solve the issue in Terraform, it does make Terraform impose a restriction that is not present in AWS. Origin access identity is optional in AWS.

[vancluever]

@llarsson sorry if there was some confusion in my reply - what I'm saying is that it's not required. If you check the pasted config and my message I mention that if you do not specify s3_origin_config, the distribution config will not have an origin access identity configured. The bug is probably happening because there is an empty default for origin_access_identity incorrectly configured, which gets written to state and hashed upon the second run, causing the spurious diff when s3_origin_config is declared as an empty block.

Again, if you don't want an origin, don't include s3_origin_config at all. As long as you don't include the block the schema constraints for that sub-resource (ie: the Required: true attribute in origin_access_identity) won't fire.

I'm just reviewing this right now and playing with acceptance tests to make sure that this indeed fixes the issue and will be putting in the PR after that.

[Dock1100]

Can confirm the same issue on terraform v0.11.14 (for now I can't migrate to v12).
Suggested workaround didn't help.

resource "aws_cloudfront_origin_access_identity" "origin_access_identity" {
  comment = "Terraform issue 7930 workaround"
}

relevant configuration blocks of resource "aws_cloudfront_distribution":

  origin {
    domain_name = "${var.primary_s3_origin_domain_name}"
    origin_id   = "${local.primary_s3_origin_id}"

    s3_origin_config {
      origin_access_identity = ""
    }
  }

  origin {
    domain_name = "${var.primary_backend_origin_domain_name}"
    origin_id   = "${local.primary_backend_origin_id}"

    custom_origin_config {
      http_port = 80
      https_port = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols = ["TLSv1", ] 
      origin_keepalive_timeout = 5
      origin_read_timeout = 60
    }
  }

Terraform output (it tries to "recreate" s3 origin, and "update" custom origin):

Terraform will perform the following actions:

  ~ module.cloudfront.aws_cloudfront_distribution.b2c_front_and_alb
      origin.3631822842.custom_header.#:                                          "0" => "0"
      origin.3631822842.custom_origin_config.#:                                   "0" => "0"
      origin.3631822842.domain_name:                                              "" => "staging.example.com.s3.amazonaws.com"
      origin.3631822842.origin_id:                                                "" => "S3-staging.example.com"
      origin.3631822842.origin_path:                                              "" => ""
      origin.3631822842.s3_origin_config.#:                                       "0" => "1"
      origin.3631822842.s3_origin_config.2547889144.origin_access_identity:       "" => ""
      origin.3674543374.custom_header.#:                                          "0" => "0"
      origin.3674543374.custom_origin_config.#:                                   "1" => "1"
      origin.3674543374.custom_origin_config.2037592804.http_port:                "80" => "80"
      origin.3674543374.custom_origin_config.2037592804.https_port:               "443" => "443"
      origin.3674543374.custom_origin_config.2037592804.origin_keepalive_timeout: "5" => "5"
      origin.3674543374.custom_origin_config.2037592804.origin_protocol_policy:   "https-only" => "https-only"
      origin.3674543374.custom_origin_config.2037592804.origin_read_timeout:      "60" => "60"
      origin.3674543374.custom_origin_config.2037592804.origin_ssl_protocols.#:   "1" => "1"
      origin.3674543374.custom_origin_config.2037592804.origin_ssl_protocols.0:   "TLSv1" => "TLSv1"
      origin.3674543374.domain_name:                                              "alb-staging-1656636550.us-east-1.elb.amazonaws.com" => "alb-staging-1656636550.us-east-1.elb.amazonaws.com"
      origin.3674543374.origin_id:                                                "ELB-alb-staging" => "ELB-alb-staging"
      origin.3674543374.origin_path:                                              "" => ""
      origin.3674543374.s3_origin_config.#:                                       "0" => "0"
      origin.3893913461.custom_header.#:                                          "0" => "0"
      origin.3893913461.custom_origin_config.#:                                   "0" => "0"
      origin.3893913461.domain_name:                                              "staging.example.com.s3.amazonaws.com" => ""
      origin.3893913461.origin_id:                                                "S3-staging.example.com" => ""
      origin.3893913461.origin_path:                                              "" => ""
      origin.3893913461.s3_origin_config.#:                                       "0" => "0"

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.