Terraform-provider-aws: Incorrectly destroying aws_autoscaling_group resource that isn't marked deposed

[hashibot]

_This issue was originally opened by @blindMess as hashicorp/terraform#16738. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Terraform Version

Terraform v0.11.0

Terraform Configuration Files

resource "aws_launch_configuration" "asg-server-as-conf" {
    name_prefix = "asg-"
        associate_public_ip_address = true
        iam_instance_profile = "${aws_iam_instance_profile.asg-profile.name}"
        image_id = "${lookup(var.my-ami-asg, var.region)}"
        instance_type = "t2.medium"
        security_groups = [
            "${aws_security_group.asg-server.id}"
        ]
    lifecycle {
        create_before_destroy = true
    }
}

resource "aws_autoscaling_group" "asg-server-as-group" {
    name = "${var.environment}-${aws_launch_configuration.asg-server-as-conf.name}"
    launch_configuration = "${aws_launch_configuration.asg-server-as-conf.name}"
    vpc_zone_identifier = [
            "${aws_subnet.asg-a.id}",
            "${aws_subnet.asg-b.id}",
            "${aws_subnet.asg-c.id}"
    ]
    wait_for_elb_capacity = "${var.default-instance-number}"
    wait_for_capacity_timeout = "20m"
    max_size = 4
    min_size = "${var.default-instance-number}"
    enabled_metrics = [
        "GroupMinSize",
        "GroupMaxSize",
        "GroupDesiredCapacity",
        "GroupInServiceInstances",
        "GroupPendingInstances",
        "GroupStandbyInstances",
        "GroupTerminatingInstances",
        "GroupTotalInstances"
    ]
    termination_policies = ["OldestInstance"]
    health_check_grace_period = 900
    health_check_type = "ELB"
    load_balancers = [
        "${aws_elb.asg-elb.name}"
    ]
    lifecycle {
        create_before_destroy = true
    }
}

Debug Output

Crash Output

Expected Behavior

A new aws_autoscaling_group is created and once it's instances are healthy on the ELB, the old aws_autoscaling_group is destroyed.

Actual Behavior

A new aws_autoscaling_group is created and once it's instances are healthy on the ELB, both the old and the new aws_autoscaling_group is destroyed.

Steps to Reproduce

1. Run terraform apply to create the launch configuration and autoscaling group
2. Update the image_id in the launch configuration and ensure that the ELB healthcheck for the autoscaling group fails.
3. Run terraform apply which should time out because of wait_for_elb_capacity.
4. Run terraform apply again, which will throw an error saying that the aws_autoscaling_group already exists.
5. On the AWS console, delete the aws_autoscaling_group referenced in the error.
6. Run terraform apply again

Important Factoids

After step 3 in reproducing, the state file will show that the new aws_autoscaling_group is primary, and the old aws_autoscaling_group is deposed.

References

Same scenario is mentioned here https://github.com/hashicorp/terraform/issues/11459 by user @artburkart but the issue was closed.

[artburkart]

Here's what I do to get around it:

#!/bin/bash

terraform plan -out plan.out &> plan.txt
if grep "aws_launch_configuration.asg-server-as-conf (deposed)" < plan.txt; then
    terraform taint aws_launch_configuration.asg-server-as-conf
    terraform plan -out plan.out
fi
terraform apply plan.out

[samueltb]

This got me as well. During the first application, I had an error from AWS on service limits (not ELB errors). I ended up in the same state described and then made the mistake of manually deleting the ASG.

[tamsky]

IMHO, the root-cause of this issue is human. Terraform is not at fault.

The claimed observed behavior:

Actual Behavior

A new aws_autoscaling_group is created and once it's instances are healthy on the ELB, both the old and the new aws_autoscaling_group is destroyed.

simply reflects the manual action disclosed by the bug author:

Steps to Reproduce

1. On the AWS console, delete the aws_autoscaling_group referenced in the error.

This manual action is the direct cause of the observed undesired behavior.

The error observed (not quoted in the original bug, but added here for clarity) is:
Error creating AutoScaling Group: AlreadyExists: AutoScalingGroup by this name already exists

This means the current ASG name is primary, is not tainted and is not deposed.

---

In the terraform config above, we see:

resource "aws_autoscaling_group" "asg-server-as-group" {
    name = "${var.environment}-${aws_launch_configuration.asg-server-as-conf.name}"

This indicates that each new LC causes a new ASG to be created (because the LC is using name_prefix, and the resulting ${LC.name} values should be unique over time.)

Therefore, if one reacts to the AlreadyExists error by choosing one of the following actions,
terraform should perform as expected on the next plan/apply cycle:

1. Taint the LC: terraform taint aws_launch_configuration.asg-server-as-conf
2. Manually manipulate terraform state so a new LC will be generated:

terraform state mv \
    aws_launch_configuration.asg-server-as-conf \
    aws_launch_configuration.asg-server-as-conf-failed

1. Use the AWS console to delete the aws_launch_configuration (LC) resource that has the conflicting ASG name -- this is a riskier option as it leaves the ASG in a state such that it is not able to launch new instances.

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!