Terraform: Can't delete IAM role because policies are attached

[rhettg]

I renamed a policy and when attempting to apply it, failure:

Error applying plan:

6 error(s) occurred:

* Error deleting IAM Role SecretsStagingReadAccess: DeleteConflict: Cannot delete entity, must detach all policies first.
    status code: 409, request id: [d1a50611-2c10-11e5-9a88-316fd3525d8b]
* aws_iam_policy_attachment.secrets-read-stage: diffs didn't match during apply. This is a bug with Terraform and should be reported.
* Error deleting IAM Role SecretsProductionReadAccess: DeleteConflict: Cannot delete entity, must detach all policies first.
    status code: 409, request id: [d1b5cf2c-2c10-11e5-b08f-8197d6e94b96]
* aws_iam_policy_attachment.secrets-read-prod: diffs didn't match during apply. This is a bug with Terraform and should be reported.
* Error deleting IAM Role SecretsDevelopmentReadAccess: DeleteConflict: Cannot delete entity, must detach all policies first.
    status code: 409, request id: [d1baff1b-2c10-11e5-9a88-316fd3525d8b]
* aws_iam_policy_attachment.secrets-read-dev: diffs didn't match during apply. This is a bug with Terraform and should be reported.

I don't think there is anything special about my configuration, but it looks something like:

resource "aws_iam_policy" "secrets-full" {
  name = "SecretsFullAccess"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${aws_s3_bucket.secrets-prod.id}/*"
    },
    {
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${aws_s3_bucket.secrets-stage.id}/*"
    },
    {
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::${aws_s3_bucket.secrets-dev.id}/*"
    }
  ]
}
EOF
}

resource "aws_iam_role" "secrets-full" {
    name = "secrets-full"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "secrets-full" {
  name = "secrets-full-attachement"
  roles = ["${aws_iam_role.secrets-full.name}"]
  policy_arn = "${aws_iam_policy.secrets-full.arn}"
}

The error occurred after changing the role name.

[pgray]

I'm working on a fix for this and I should be able to fix #2818 with the same logic... I think.

[catsby]

@GrayCoder thanks for volunteering! Are you still working on this? Apologies if you've already submitted it, I'm very behind :frowning:

[pgray]

I haven't gotten a chance to fully figure out how to do this yet. It seems like changing a user, role, or group name causes a deletion and recreation of that resource, but we'd need to trigger the detachment of the policy via that change.

[simpsora]

Is there any update on this one? It's a bit of a blocker to use TF in a production system if one must delete attached policies from roles before updating them.

[lestephane]

I'm also experiencing this problem. Guess I'd better be idiomatic with my resource names so I don't have to rename them for the time being.

[kamoljan]

+1

[pawelprazak]

Still experiencing this with Terraform v0.7.5, but with a manually (from AWS console) attached policy, is it an expected behavior?

[mrwacky42]

See also the suggestion in this comment by @tamsky.. (Since #2957 is now closed)

[afoldesi-gds]

Same here. Any updates?

[owenmorgan]

also seeing this.

[dcosson]

It seems like there's a separate issue than just the renaming case - even trying to delete both a role and its policy attachment in the same run will sometimes fail if terraform decides to delete the role first.

I don't quite understand this behavior, because my policy attachment references the role. During creation, terraform always uses the dependency graph and creates the thing being referred to (iam role) before creating the thing that depends on it (policy attachment).

Why don't deletions use the same dependency graph logic, and delete things in reverse order of how they were created?

(On top of this, to solve the specific case above it seems another feature would be needed to then "propagate recreations" for some dependency relationships such as policy attachment -> role during a role -/+).

[gtmtech]

Are there any concrete plans to solve this and #2957 ? Without it it is not possible to create an immutable infra workflow that works completely within the terraform toolset, and I note both are unsolved after a year and a half.

[pawelprazak]

sadly, we were forced to stop using terraform, because it had too many
issues preventing us from creating immutable infrastructure scripts

On Wed, 1 Mar 2017, 12:04 Geoff Meakin, [e-mail] wrote:

Are there any concrete plans to solve this and #2957
https://github.com/hashicorp/terraform/issues/2957 ? Without it it is not
possible to create an immutable infra workflow that works completely within
the terraform toolset, and I note both are unsolved after a year and a half.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform/issues/2761#issuecomment-283311279,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAC-6ts5jStzl94vEUDsnoWe7gMUdrzTks5rhVC6gaJpZM4FaTtP
.

[ronniemu]

This is a big problem for us with aws_ebs_volume and aws_volume_attachment and has been long running with no updates in respect to resolution. Anyone from Hashicorp to give a definitive plan, this was raised in 2015.

[jbares83]

This issue is also causing me problem in the automatic maintenance process of the instance.

2 error(s) occurred:

• module.XXX.aws_volume_attachment.XXX-data-attachment (destroy): 1 error(s) occurred:

• aws_volume_attachment.XXXX-data-attachment: Error waiting for Volume (vol-XXXX) to detach from Instance: i-XXXX

The only workaround is to do it manually on the Amazon console.

Do you have any plan to resolve this critical conflict?

[andreynpetrov]

Faced the same issue with terraform version 0.9.0.

• module.aws.aws_volume_attachment.im_xvdh (destroy): 1 error(s) occurred:

• aws_volume_attachment.im_xvdh: Error waiting for Volume (vol-0e2cee0e22466bac9) to detach from Instance: i-0f9b6a08597308406

• module.aws.aws_internet_gateway.terraform-apcera (destroy): 1 error(s) occurred:

• aws_internet_gateway.terraform-apcera: Error waiting for internet gateway (igw-e424e982) to detach: couldn't find resource (31 retries)

• module.aws.aws_volume_attachment.central_xvdh (destroy): 1 error(s) occurred:

• aws_volume_attachment.central_xvdh: Error waiting for Volume (vol-0cb84b1d93abdee5a) to detach from Instance: i-05e792bd7ae0173ad

[dvishniakov]

The same in 0.9.3

[pgray]

Try the newer resources:
iam_{role,user,group}_policy_attachment
https://www.terraform.io/docs/providers/aws/r/iam_role_policy_attachment.html

[dooreelko]

@pgray
all my policies are using the *_policy_attachment (just like the original report), same problem. actually this could be the root cause

[radeksimko]

Hi folks,
firstly sorry for the long silence here.

Secondly the root cause of the problem here is ordering of delete/create operation for the aws_iam_role being "renamed" (in quotes because one can't just rename a role, so it's being recreated under a new name).

When a field is marked as ForceNew in the internal schema then Terraform first destroys the old resource and then creates the new one and updates references (by default). This is a default behaviour because sometimes the reason for recreation isn't a unique name and in such cases we can't just create the new one first, because it would conflict with the existing name.

The easiest and cleanest solution for now is just to add this block of code to the role:

lifecycle {
  create_before_destroy = true
}

which reverses the ordering of delete/create operations and avoids the problem described in this issue altogether.

https://www.terraform.io/docs/configuration/resources.html#lifecycle

I think it would make sense in many cases (like here), to make this behaviour default one, but only under certain conditions - e.g. when recreation is triggered by a field like name (i.e. we actually _can_ guarantee that 2+ resources with a field of the same value can exist). For that reason I'm relabelling this issue to core as we'll need to modify the schema to support such behaviour.

[jogster]

I already use this in all the resources

lifecycle {
create_before_destroy = true
}

My Solution:

Manually run:
aws iam remove-role-from-instance-profile --instance-profile-name ${key} --role-name ${key}
aws iam delete-instance-profile --instance-profile-name ${key}
aws iam delete-role --role-name ${key}

and then remove the resources from my terraform.tfstate file

Not a great solution, but allowed me to destroy and rebuild these resources without having to completely teardown the complete script.

Would love to know if there is a more elegant solution.

[mrwacky42]

@jogster - at the very least, you can change your solution to use terraform state rm <the resource> instead of having to mess with terraform.tfstate file.

[jogster]

Thanks for the quick response, just tried that -- works a treat, I have added it to my script. I think this is workable for me now.

[ekristen]

I am seeing this too, if a role has an attached document and needs to get re-created, it'll error because terraform doesn't remove the attached policies.

[matikumra69]

I am new to Terraform and getting this problem while destroying

aws_volume_attachment.ebs_att: Error waiting for Volume (vol-06b32591d86b6d95e) to detach from Instance: i-04528452d805001c5

Any help will be appreciated. Thanks.

[b-dean]

@matikumra69 With aws_volume_attachment this was originally discussed on #2957 and there is a workaround in the comments there.

[houqp]

The workaround in 2957 doesn't work for latest terraform anymore. I am at v0.10.7.

[skehlet]

There's a force_detach_policies argument on aws_iam_role, set it to true so this doesn't happen in the future. But if you're having this problem now, adding it won't help. Expanding on @jogster's workaround above, you can manually get past it with:

aws iam list-attached-role-policies  --role-name <your-role-name>

to get the arn of the offending attached policy, then:

aws iam detach-role-policy  --role-name <your-role-name> --policy-arn <the-arn-from-above>

then

aws iam delete-role --role-name <your-role-name>

and then terraform plan, terraform apply should work. Or something similar, it took a couple tries to get past this.

[Xyrion]

Any update on when this will be resolved? From what i can see this has been an issue since 2015. How can it take this long to resolve such a critical issue. Workarounds are not a solution. Most workarounds don't even work.

[hashibot[bot]]

This issue has been automatically migrated to terraform-providers/terraform-provider-aws#5417 because it looks like an issue with that provider. If you believe this is _not_ an issue with the provider, please reply to terraform-providers/terraform-provider-aws#5417.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.