Terraform: AWS autoscaling group notification issue

[madhurranjan]

Facing the following issue while trying to set notification in SNS for ASG. I've a previously created SNS and want all my ASGs to post to that SNS. Below is the output:

* aws_autoscaling_notification.notifications: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

`    Terraform Version: 0.6.13
    Resource ID: aws_autoscaling_notification.notifications
    Mismatch reason: attribute mismatch: group_names.60741579
    Diff One (usually from plan): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"topic_arn":*terraform.ResourceAttrDiff{Old:"", New:"arn:aws:sns:us-east-1:<account>:asgalerts", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}, "group_names.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "group_names.60741579":*terraform.ResourceAttrDiff{Old:"", New:"${element(aws_launch_configuration.config.*.name,count.index)}", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.#":*terraform.ResourceAttrDiff{Old:"", New:"3", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.3116970575":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_TERMINATE", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.4098158487":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCH", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.753791182":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCH_ERROR", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}}, Destroy:false, DestroyTainted:false}
    Diff Two (usually from apply): *terraform.InstanceDiff{Attributes:map[string]*terraform.ResourceAttrDiff{"group_names.#":*terraform.ResourceAttrDiff{Old:"", New:"1", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "group_names.706393881":*terraform.ResourceAttrDiff{Old:"", New:"test_nq3uqqh5ixvgyncha67vn7mryxu", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.#":*terraform.ResourceAttrDiff{Old:"", New:"3", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.3116970575":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_TERMINATE", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.4098158487":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCH", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "notifications.753791182":*terraform.ResourceAttrDiff{Old:"", New:"autoscaling:EC2_INSTANCE_LAUNCH_ERROR", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Type:0x0}, "topic_arn":*terraform.ResourceAttrDiff{Old:"", New:"arn:aws:sns:us-east-1:<account id>:asgalerts", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Type:0x0}}, Destroy:false, DestroyTainted:false}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.

Below is the snippet for notification:

resource "aws_autoscaling_notification" "notifications" {
  count = "${var.enabled}"
  group_names = [
    "${element(aws_autoscaling_group.group_release.*.name,count.index)}"
  ]
  notifications  = [
    "autoscaling:EC2_INSTANCE_LAUNCH",
    "autoscaling:EC2_INSTANCE_TERMINATE",
    "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
  ]
  topic_arn = "${var.asg_topic_arn}"

}

[dblooman]

Is this something that is likely to be fixed soon, my TF runs fail every time I rebuild an AMO, which is daily.

[ghost]

I've never looked at coding in go before so I'm a little hesitant to speak up, but could this issue have something to do with aws-sdk-go expecting an AutoScalingGroupName as a string, while terraform is accepting an array/set of autoscaling group names?

https://github.com/aws/aws-sdk-go/blob/master/service/autoscaling/api.go#L6024
AutoScalingGroupName *string min:"1" type:"string" required:"true"

https://github.com/hashicorp/terraform/blob/master/builtin/providers/aws/resource_aws_autoscaling_notification.go#L20-L40
"group_names": &schema.Schema{
Type: schema.TypeSet,
Required: true,
Elem: &schema.Schema{Type: schema.TypeString},
Set: schema.HashString,
},

[donnoman]

my aws_autoscaling_notification is trying to change an unrelated notification.

lh-wordpress-staging was manually created, it has nothing to do with this terraform plan.
None of the other objects have confused themselves with other pre-existing objects for lh-wordpress-staging.

resource "aws_autoscaling_notification" "default" {
  group_names = ["${aws_autoscaling_group.default.name}"]
  notifications  = [
    "autoscaling:EC2_INSTANCE_LAUNCH",
    "autoscaling:EC2_INSTANCE_TERMINATE",
    "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
  ]
  topic_arn = "${var.aws_sns_topic.default}"
}

~ aws_autoscaling_notification.default
    group_names.727904493:    "" => "lh-wordpress-production"
    group_names.840576081:    "lh-wordpress-staging" => ""
    notifications.#:          "4" => "3"
    notifications.3116970575: "autoscaling:EC2_INSTANCE_TERMINATE" => "autoscaling:EC2_INSTANCE_TERMINATE"
    notifications.3298075921: "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" => ""
    notifications.4098158487: "autoscaling:EC2_INSTANCE_LAUNCH" => "autoscaling:EC2_INSTANCE_LAUNCH"
    notifications.753791182:  "autoscaling:EC2_INSTANCE_LAUNCH_ERROR" => "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"

which is what it precisely did when I did the apply

aws_autoscaling_notification.default: Modifying...
  group_names.727904493:    "" => "lh-wordpress-production"
  group_names.840576081:    "lh-wordpress-staging" => ""
  notifications.#:          "4" => "3"
  notifications.3116970575: "autoscaling:EC2_INSTANCE_TERMINATE" => "autoscaling:EC2_INSTANCE_TERMINATE"
  notifications.3298075921: "autoscaling:EC2_INSTANCE_TERMINATE_ERROR" => ""
  notifications.4098158487: "autoscaling:EC2_INSTANCE_LAUNCH" => "autoscaling:EC2_INSTANCE_LAUNCH"
  notifications.753791182:  "autoscaling:EC2_INSTANCE_LAUNCH_ERROR" => "autoscaling:EC2_INSTANCE_LAUNCH_ERROR"
aws_autoscaling_notification.default: Modifications complete

And now my lh-wordpress-staging ASG has no notification policy.

› terraform --version
Terraform v0.6.16

[roderickrandolph]

I'm experiencing this issue as well. The bug seems to rear its head when the number of ASG notifications for a single 'topic_arn' exceeds a threshold -- possibly a paging issue when terraform tries to match the tf state against the results the AWS API returns.

When the threshold is reached it appears this resource removes the 'topic_arn' notification for the last ASG group name in the AWS results (often unrelated to the current / target ASG group name but it shares the same 'topic_arn') and then creates an ASG notification for the target ASG group name. The problem repeats itself on the next terraform plan for a different ASG group name but with the same 'topic_arn'.

[baskaran-md]

The issue happens when there is a double referencing of the variables.

resource "aws_autoscaling_notification" "notifications" {
  group_names = [
    "${aws_autoscaling_group. asg.name}"
  ]
  notifications  = [
    "autoscaling:EC2_INSTANCE_LAUNCH_ERROR",
    "autoscaling:EC2_INSTANCE_TERMINATE",
    "autoscaling:EC2_INSTANCE_TERMINATE_ERROR"
  ]
  topic_arn = "${var.aws_sns_topic.default}"
}

resource "aws_autoscaling_group" "asg" {
  name                      = "${element(aws_launch_configuration.lc.*.name,count.index)}"
  launch_configuration      = "${element(aws_launch_configuration.lc.*.id,count.index)}"
  . . . . .
}

resource "aws_launch_configuration" "lc" {
  count = "${var.count}"
  name_prefix                 = "${var.application}_${var.release}_"
  . . . . .
}

When the terraform plan is made, the group_name is assumed to be a string with those variables instead of a "<computed>"

 group_names.506827717: "${element(aws_launch_configuration.lc.*.name,count.index)}"

However, on terraform apply this variables are computed to a string (like tomcat_123__20160919204150034123567abc). It seems like terraform fails when it tries a diff between the planned value and the computed one.

Workaround:
In aws_autoscaling_notification, use id of AutoScaling group, instead of the name

  group_names = [
    "${aws_autoscaling_group.asg.id}"
  ]

On doing this, when terraform plan is made, the group_name is shown as "<computed>" and so the diff on terraform apply takes care of this value.

[roderickrandolph]

Switching from "name" to "id" did not fix the bug for me. The values of "id" and "name" are the same.

[mitchellh]

This should be fixed in the latest version of Terraform.

Its hard to be 100% sure but we fixed many issues related to not showing properly (which triggered this). Please give it a try and if it is still happening open a new issue. thanks!

[roderickrandolph]

this issue still exists in 0.8 - not sure why the issue was closed before anyone here confirmed it was fixed.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.