Terraform-provider-aws: aws_route53_health_check does not syncronize configuration after an apply

[grayaii]

Terraform Version

Terraform v0.10.7
aws provider: 1.9.0

Affected Resource(s)

Please list the resources as a list, for example:

• aws_route53_health_check

Terraform Configuration Files

resource "aws_route53_record" "walstream-secondary" {
  name = "walstream.ccodb.foobar.com"
  zone_id = "${data.terraform_remote_state.route53.foobar_zone_id}"
  type = "A"
  set_identifier = "SECONDARY"
  failover_routing_policy = {
       type = "SECONDARY"
   }
  alias {
    name = "write.ccodb.foobar.com"
    zone_id = "${data.terraform_remote_state.route53.foobar_zone_id}"
    evaluate_target_health = true
  }
}

resource "aws_route53_record" "walstream" {
  lifecycle { create_before_destroy = true }
  name    = "walstream.ccodb.foobar.com"
  zone_id = "${data.terraform_remote_state.route53.foobar_zone_id}"
  type = "A"
  set_identifier = "PRIMARY"
  health_check_id = "${aws_route53_health_check.walstream_health_check.id}"
  failover_routing_policy = {
       type = "PRIMARY"
   }
  records = ["${var.failover_ip}"]
  ttl     = "60"
}

resource "aws_cloudwatch_metric_alarm" "failover_instance_health" {
    alarm_name          = "${var.environment}-failover-instance-health"
    alarm_description   = "The healthyness of the failover instance"
    namespace           = "AWS/EC2"
    metric_name         = "StatusCheckFailed"
    dimensions {
      InstanceId = "${var.failover_id}"
    }
    statistic           = "Average"
    period              = "60"
    evaluation_periods  = "1"
    comparison_operator = "GreaterThanOrEqualToThreshold"
    threshold           = "1"
    actions_enabled     = "True"
    # Trigger Alarm if data is missing:
    treat_missing_data  = "breaching"
    # Alarm when triggered:
    alarm_actions       = ["${data.terraform_remote_state.vpc.db_topic_arn}"]
    ok_actions          = ["${data.terraform_remote_state.vpc.db_topic_arn}"]
    actions_enabled     = "${var.alerting_enabled}"
}

resource "aws_route53_health_check" "walstream_health_check" {
  type                            = "CLOUDWATCH_METRIC"
  cloudwatch_alarm_name           = "${aws_cloudwatch_metric_alarm.failover_instance_health.alarm_name}"
  cloudwatch_alarm_region         = "${var.region}"
  insufficient_data_health_status = "Unhealthy"
  tags = {
    Name = "${var.environment}-failover-instance-health"
  }
}

Expected Behavior

We have a aws_route53_health_check that is based off an aws_cloudwatch_metric_alarm.
The aws_cloudwatch_metric_alarm checks the healthiness of an EC2 instance.
When we update the instance-id of the aws_cloudwatch_metric_alarm, we would expect the aws_route53_health_check health check to start using the updated configuration of the alarm.

Actual Behavior

We update the the instance-id of the aws_cloudwatch_metric_alarm but the aws_route53_health_check still uses the OLD instance id. You have to manually go to the AWS console and click on "Synchronize Configuration" button:

[jmaitrehenry]

We have the same problem with Terraform v0.11.7 too.
We lost a lot of time to understand why our alarm was not updated when TF say Modifications complete.

Actually, every time we deploy a new ASG/ALB, the alarm is triggered and it's the alarm for region failover. So every time, we need to terraform destroy and apply if we want to have the new alarm version without going to AWS console.

[JeremieCharest]

Same thing for me! TF output confirmed modification but in reality it's pending and waiting for manual approval in AWS :

aws_cloudwatch_metric_alarm.pxy_alb_healthy_host_count: Modifying... (ID: XXXX-us-east-1-healthy-pxy-AlbCriticalHealthyHostCount)
  dimensions.LoadBalancer: "app/XXXX/0d5AAAAA50d507591" => "app/XXXX/916YYYYYf2d8c7b7"
...
aws_cloudwatch_metric_alarm.pxy_alb_healthy_host_count: Modifications complete after 0s (ID: XXXX-us-east-1-healthy-pxy-AlbCriticalHealthyHostCount)

Terraform 0.11.7
AWS provider: 1.21.0

[jmaitrehenry]

From AWS documentation

To update a health check programmatically, you can use the UpdateHealthCheck API. Just specify the current values for AlarmIdentifier and Region, and Route 53 will get the latest settings from CloudWatch. For more information, see UpdateHealthCheck in the Amazon Route 53 API Reference.

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/health-checks-updating-cloudwatch-alarm-settings.html

And actually, Terraform will not set the AlarmIdentifier on update if the alarm_name or region didn't change:

    if d.HasChange("cloudwatch_alarm_name") || d.HasChange("cloudwatch_alarm_region") {
        cloudwatchAlarm := &route53.AlarmIdentifier{
            Name:   aws.String(d.Get("cloudwatch_alarm_name").(string)),
            Region: aws.String(d.Get("cloudwatch_alarm_region").(string)),
        }

        updateHealthCheck.AlarmIdentifier = cloudwatchAlarm
    }

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_route53_health_check.go#L178

Same for the region:

    if d.HasChange("regions") {
        updateHealthCheck.Regions = expandStringList(d.Get("regions").(*schema.Set).List())
    }

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_route53_health_check.go#L195

I will try a patch to see if it fix the problem.

[mutt13y]

This is still a problem with terraform 0.12.20, AWS 2.33