Terraform-provider-aws: AWS security groups not being destroyed

[szczad]

Hi there,

Terraform Version

Terraform v0.11.0

• provider.aws v1.3.1

Affected Resource(s)

Please list the resources as a list, for example:

• aws_security_group
  (probably more)

Terraform Configuration Files

Changed "name" from "all-zabbix" to "all-zabbix-test"

resource "aws_security_group" "all-zabbix" {
  vpc_id                 = "${aws_vpc.infra.id}"
  name                   = "all-zabbix-test"
  description            = "Zabbix"
}

resource "aws_security_group_rule" "all-zabbix-in-tcp-10051" {
  security_group_id = "${aws_security_group.all-zabbix.id}"
  description       = "Zabbix internal communication"
  type              = "ingress"
  protocol          = "tcp"

  from_port = 10051
  to_port   = 10051
  self      = true
}

Debug Output

aws_security_group_rule.all-zabbix-in-tcp-10051: Destroying... (ID: sgrule-1234567890)
aws_security_group_rule.all-zabbix-in-tcp-10051: Destruction complete after 1s
aws_security_group.all-zabbix: Destroying... (ID: sg-12345678)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 10s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 20s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 30s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 40s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 50s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m0s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m10s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m20s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m30s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m40s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 1m50s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m0s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m10s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m20s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m30s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m40s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 2m50s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m0s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m10s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m20s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m30s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m40s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 3m50s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m0s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m10s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m20s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m30s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m40s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 4m50s elapsed)
aws_security_group.all-zabbix: Still destroying... (ID: sg-12345678, 5m0s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* aws_security_group.all-zabbix (destroy): 1 error(s) occurred:

* aws_security_group.all-zabbix: DependencyViolation: resource sg-12345678 has a dependent object
        status code: 400, request id: abcdefg-dead-beef-dead-abcdefg123456

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Expected Behavior

• remove SG's rules
• remove old SG from instances/interfaces
• delete old SG
• create SG with new values
• adds new SG to instances/interfaces

Actual Behavior

• remove SG's rules
• delete old SG
• times out during deletion leaving SG without rules

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. change security group name to force resource recreation
2. terraform apply

[mzupan]

is the SG attached to anything like a ELB or instance?

[szczad]

Yeah. It's attached to one instance managed by Terraform. If I manually remove association everything goes well, but terraform does not do that on its own.

[apparentlymart]

Hi @szczad! Sorry for this limitation.

At the moment Terraform doesn't have any mechanism to deal with these "enforced dependencies" in the underlying service, and indeed it affects a number of resources, such as what we see in #646, #151, #2201.

The problem is that this requires some extra coordination between operations -- creating multiple coordinated steps as you mentioned -- which Terraform's provider model isn't currently able to represent. In certain cases such dependencies _could_ be represented in principle -- for example, in this case where Terraform is managing both the security group and the other resources that belong to it -- while in other cases Terraform can't "see" the dependency at all, because e.g. the EC2 instances are being created implicitly by an aws_autoscaling_group.

We would like to find a way to address this limitation in the long run, for certain situations at least, but at this time we do not have a suitable design figured out to deal with it, and our current Terraform Core development focus is elsewhere. For now it is, as you noted, required to manually break the dependency somehow before making these changes, which is definitely not ideal.

In principle we could improve the behavior here by at least producing a helpful error when this situation arises. Unfortunately the long-running polling behavior you saw here was introduced to work around a different problem: network interfaces tend to live for a few minutes after their associated resources are destroyed, acting as dependencies on the security group that aren't reflected in the API at all. Terraform therefore retries here so that it can wait until VPC has finished deallocating the network interface before proceeding, rather than failing with a hard error in that case.

[szczad]

Hmm... That sounds complicated a lot. But thanks for detailed explanation.
It looks like I have to stick to managing our SGs manually. Lucky those who split their configuration into smaller pieces. :-)

[FrenchBen]

I often run into this exact issue, which is quite annoying, especially when I'm trying to destroy -force.

Having a dependency violation come up, when I'm trying to get rid of my resources, is not expected behavior.
I'm asking for them to be destroyed (deleted), why are we even evaluating dependencies?

Does this mean that the -force flag is only useful to prevent a user from having any input?

[eriksw]

Has 0.12 introduced anything that'd allow for a fix to this? Having to manually alter >80 security groups via the console is not a pleasant experience.

[rehevkor5]

Also related (SG with RDS, SG with VPCE): https://github.com/terraform-providers/terraform-provider-aws/issues/9692

[linuxman79]

This is also manifest by removing a security group AND its association with any instances. In this case, the dependency is broken by the change to the instance, but terraform is trying to delete the security group BEFORE modifying the instance. Just another manifestation of the same issue.

[pneigel-ca]

I experience this problem with the following resources:

• aws_launch_template
• aws_autoscaling_group
• aws_security_group

I am using:
Terraform v0.12.20
provider.aws v2.49.0

When I attempt to destroy the resources, the security group hangs for several minutes. When investigating in the UI, it appears there is a network interface dependency which prevents it's deletion from the AWS console. If I delete the network interface manually, the terraform deletion succeeds very quickly. If I do not, ultimately the operation fails similarly.

Error: Error deleting security group: DependencyViolation: resource sg-some-resource0 has a dependent object status code: 400, request id: some-id

[perelin]

Same issue here. When I try to apply a change to a SG that forces replacement:

Error: Error deleting security group: DependencyViolation: resource sg-0deddff8230759a10 has a dependent object
    status code: 400, request id: 451d4b6f-4d10-4a62-a70e-cfefce3cbd3c

[brandon-fryslie]

What are the workarounds for this? We're evaluating Terraform right now to potentially switch from using Ansible for provisioning cloud infrastructure, but this seems like a pretty glaring omission.

Is there really no way to tell Terraform to remove the SG from places it is used before updating it? I can't imagine heavy users of Terraform are continuing to make manual configuration changes. Thanks for your help

[DaveHewy]

To add to this thread. I often find upon a second run of the destroy command the previously endlessly hanging security_group deletion task will immediately delete. Perhaps suggesting Terraform needs to perform some sort of re-calculation at each retry attempt?

[johnlabarge]

I'm getting this as well. Are there any workarounds?

[weibeld]

This issue occurs when renaming a security group in Terraform and also updating the aws_instance resource to reference the security group by its new name.

Terraform tries to do the following:

1. Destroy existing security group
2. Create new security group
3. Modify EC2 instance (disassociate old SG and associate new SG)

This causes _(1)_ to hang because the AWS API prevents deleting a SG that's still associated with an instance. _(2)_ succeeds, and _(3)_ is never executed.

The correct order of actions should be:

1. Create new security group
2. Modify EC2 instance (disassociate old SG and associate new SG)
3. Destroy existing security group

I didn't find any working solutions, except during the hanging going to the AWS Management Console _Actions > Networking > Change Security Groups_, disassociating the old security group, and associating the new security group.

[gchek]

Hitting the same:
if the SG of an EC2 is added to the main VPC SG, terraform can not destroy.
need to manually remove the main SG rules and remove the EC2 SG to succeed. Painful.

[BcTpe4HbIu]

With create_before_destroy = true in SG it works.

resource "aws_security_group" "intercluster" {
  name   = "some sg"
  vpc_id = var.vpc_id

  lifecycle {
    create_before_destroy = true
  }
<snip>

[bengiddins]

lifecycle {
create_before_destroy = true
}

Thanks for this! I'm watching a SG try to delete for over 60 minutes now - painstakingly went through discrete steps of creating a new group, assigning the new group, deleting the old group - oh wait, I tested manually adding a Lambda function to the SG and removing it, but now the SG is still attached to those network interfaces and won't delete ಠ_ಠ

aws_security_group.this: Still destroying... [id=sg-0920404d643c46xxx, 1h2m51s elapsed]
aws_security_group.this: Still destroying... [id=sg-0920404d643c46xxx, 1h3m1s elapsed]
aws_security_group.this: Still destroying... [id=sg-0920404d643c46xxx, 1h3m11s elapsed]

[gchek]

create_before_destroy = true doesn't help. To destroy my SG, i need to MANUALLY remove the other SG from the rules.
Simple test code available here

Terraform v0.12.26

• provider.aws v2.65.0

The issue is the the default SG is detroyed BEFORE the EC2 SG is and in fact it's not true - the default SG is still there even if Terraform says "destroyed" (see 2nd line below)

module.VPCs.aws_default_security_group.default: Destroying... [id=sg-02057153bb443e13d]
module.VPCs.aws_default_security_group.default: Destruction complete after 0s
module.VPCs.aws_security_group.GC-SG-VPC-test: Destroying... [id=sg-02a06934c19a8efaa]

The GC-SG-VPC-test security group is part the the default SG rules !!!

[Roxyrob]

Thanks @BcTpe4HbIu

  lifecycle {
    create_before_destroy = true
  }

worked in my case.