Terraform: Resources are overwritten in independent environments.

[fubarbaz]

I am using policy attachments like the following in two ways. One with -test, and one with -production with two different tfstate files, etc.

resource "aws_iam_policy_attachment" "AmazonEC2ContainerServiceRole-test" {
  name = "AmazonEC2ContainerServiceRole-test"
  roles = ["${aws_iam_role.ecsServiceRole.name}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

resource "aws_iam_policy_attachment" "AmazonEC2ContainerServiceFullAccess-test" {
  name = "AmazonEC2ContainerServiceFullAccess-test"
  roles = ["${aws_iam_role.ecsInstanceRole.name}"]
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerServiceFullAccess"
}

resource "aws_iam_policy_attachment" "AmazonKinesisAnalyticsFullAccess-test" {
  name = "AmazonKinesisAnalyticsFullAccess-test"
  roles = ["${aws_iam_role.ecsInstanceRole.name}"]
  policy_arn = "arn:aws:iam::aws:policy/AmazonKinesisAnalyticsFullAccess"
}

However, when I run terraform apply for the -production case, it tries to overwrite the configuration for the -test case. That's not what is supposed to happen. In the same way, if I run terraform apply for the -test case, it tries to delete the resources created for the -production case.

Just to be clear: you can imagine that there are multiple, independent main.tf files, each specifying a version of the resource. It's not the same resource, they just happen to point at the same policy_arn.

[fubarbaz]

So, to further simpify this: I want independent runs of terraform to be able to register role names to IAM > Policies > AmazonKinesisAnalyticsFullAccess.

[fubarbaz]

It appears that https://www.terraform.io/docs/providers/aws/r/iam_policy_attachment.html is just a bit vague. I'd argue that this terraform feature in its current form is not very useful. It should be obvious how to implement is in a different way given my comments above.

[catsby]

Hey @fubarbaz – aws_iam_policy_attachment is pretty far-reaching; it is recommended only 1 aws_iam_policy_attachment to manage the policies of any given role, group, or user. In your example above, I believe they are conflicting.

You mention a -test and -prod, are they in separate accounts? IS aws_iam_role.ecsInstanceRole.name the same IAM role for each, or does each env have it's own version of whatever role aws_iam_role.ecsInstanceRole.name represents?

Have you checked out https://www.terraform.io/docs/providers/aws/r/iam_role_policy_attachment.html ? Perhaps that will help you

[fubarbaz]

@catsby They are not conflicting. It's a Terraform limitation, as I explained above. If you do not understand the issue, then there is no point in trying to contribute to this issue. I already worked around it, and I don't consider it important enough to engage in further communication on this topic. So, this is a formal request to not send me any message on this topic ever again.

If @mitchellh doesn't understand how the solution look like (which would be doubtful), I could consider to write a more lengthy explanation as to how this should be fixed. @mitchellh is allowed to send me a message.

[mitchellh]

This would be a core bug but I don't 100% understand the case here (probably due to additional lack of knowledge of the full semantics of IAM policy attachment).

I understand you worked around it @fubarbaz so you may not have time to do this. But if you could: could you give us a config to repro this (minimal if possible, I think the above is probably close?) and just a few steps (as simple as 1. terraform apply 2. terraform apply -state-out=whatever etc.) just so I can very simply see whats going on.

Once I have that I can take a closer look.

[fubarbaz]

@mitchellh

#file name: a/main.tf
resource "aws_iam_policy_attachment" "AmazonEC2ContainerServiceRole-a" {
  name = "AmazonEC2ContainerServiceRole-a"
  roles = ["${aws_iam_role.ecsServiceRole.name}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

#file name: b/main.tf
resource "aws_iam_policy_attachment" "AmazonEC2ContainerServiceRole-b" {
  name = "AmazonEC2ContainerServiceRole-b"
  roles = ["${aws_iam_role.ecsServiceRole.name}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

I didn't test this, but I suspect this will produce a repro if you make sure the roles exist.

cd a && terraform apply && cd ../b && terraform apply

On a higher-level, what happens is that that resource which is different (e.g. also has a different name in Amazon), has an influence on other resources referencing the same policy_arn.

I hope this helps you track it down.

[mitchellh]

Thanks, I filled in some dots since it wasn't a ready-to-run repro so just let me know if I filled in any dots incorrectly.

After doing so, I ran your example and everything worked well for me:

A → terraform apply
aws_iam_role.ecsServiceRole: Creating...
  arn:                "" => "<computed>"
  assume_role_policy: "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"ec2.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:        "" => "<computed>"
  name:               "" => "fooRole"
  path:               "" => "/"
  unique_id:          "" => "<computed>"
aws_iam_role.ecsServiceRole: Creation complete
aws_iam_policy_attachment.AmazonEC2ContainerServiceRole-a: Creating...
  name:             "" => "AmazonEC2ContainerServiceRole-a"
  policy_arn:       "" => "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
  roles.#:          "" => "1"
  roles.1104478714: "" => "fooRole"
aws_iam_policy_attachment.AmazonEC2ContainerServiceRole-a: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
A → cd ../B
B → terraform apply
aws_iam_role.ecsServiceRole: Creating...
  arn:                "" => "<computed>"
  assume_role_policy: "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"ec2.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:        "" => "<computed>"
  name:               "" => "fooRole"
  path:               "" => "/"
  unique_id:          "" => "<computed>"
Error applying plan:

1 error(s) occurred:

* aws_iam_role.ecsServiceRole: Error creating IAM Role fooRole: EntityAlreadyExists: Role with name fooRole already exists.
    status code: 409, request id: 4ead8643-c153-11e6-ab80-7d6106f61ec2

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
B → terraform apply
aws_iam_role.ecsServiceRole: Creating...
  arn:                "" => "<computed>"
  assume_role_policy: "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"ec2.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:        "" => "<computed>"
  name:               "" => "barRole"
  path:               "" => "/"
  unique_id:          "" => "<computed>"
aws_iam_role.ecsServiceRole: Creation complete
aws_iam_policy_attachment.AmazonEC2ContainerServiceRole-b: Creating...
  name:            "" => "AmazonEC2ContainerServiceRole-b"
  policy_arn:      "" => "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
  roles.#:         "" => "1"
  roles.400562090: "" => "barRole"
aws_iam_policy_attachment.AmazonEC2ContainerServiceRole-b: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
B →

The only issue I ran into was that the policy names errored but that is an Amazon semantic: Policy names are a singleton and we send in exactly the name specified. I believe that is the correct behavior we don't attempt to circumvent or change semantics of the backend provider.

When inspecting the state in A vs. B, I saw differing IDs.

Is there something I'm missing?

[fubarbaz]

Now, in the Amazon Console inspect how fooRole and barRole look like, i.e. who has the AmazonEC2ContainerServiceRole. The right answer should be both. In the version I ran some time ago it was one.

If you see both then it has been fixed already, but I'd find that unlikely.

[dvelitchkov]

I am being bitten by this bug as well.

I have two separate terraform environments in the same AWS account.

Prod env:

resource "aws_iam_role" "mcube-ecs-inst-role-prod" {
    name = "mcubeECSInstanceRoleProd"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}


resource "aws_iam_policy_attachment" "mcube-ecs-inst-role-prod" {
    name = "mcube-ecs-inst-role-prod"
    roles = ["${aws_iam_role.mcube-ecs-inst-role-prod.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

resource "aws_iam_instance_profile" "mcube-inst-profile-prod" {
    name = "mcube-inst-profile-prod"
    roles = ["${aws_iam_role.mcube-ecs-inst-role-prod.name}"]
}

non-prod env:

resource "aws_iam_role" "mcube-ecs-inst-role" {
    name = "mcubeECSInstanceRole"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}


resource "aws_iam_instance_profile" "mcube-inst-profile" {
    name = "mcube-inst-profile"
    roles = ["${aws_iam_role.mcube-ecs-inst-role.name}"]
}

resource "aws_iam_policy_attachment" "mcube-ecs-inst-role" {
    name = "mcube-ecs-inst-role"
    roles = ["${aws_iam_role.mcube-ecs-inst-role.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

Doing a terraform apply from one environment will alway screw up the other environment by removing the other environment's policy attachments. Every time.

Running terraform 0.7.13

EDIT: I gave all my resources different names in an attempt to stop them from stepping on each other, even though they were already in completely separate terraform environments.

It looks like terraform sees the policy attachment as some kind exclusive action - it detaches the IAM Policy from all roles that are not explicitly specified in the aws_iam_policy_attachment, even ones that are managed by separate terraform environments.

[timothy-spencer]

I am also having this problem, and dvelitchkov's description of the problem in #5979 seems to be a nice simple way to describe the issue:

Basically, the aws_iam_policy_attachment resource is a "all-or-nothing" type resource for the entire AWS account. I can't even have separate aws_iam_policy_attachment resources in separate environments.

Not sure if that maps to how it's implemented in code, but that's how it looks to me too. Let's hope that it's just a simple "oh, we are doing a search improperly to get the id of the policy" here and not something more fundamental. :-)

[fubarbaz]

The status of this ticket is incorrect. We are not waiting for my response anymore, because the issue has been clarified by others already. Please remove the waiting-response label.

[ijin]

This problem essentially renders AWS's managed policies useless in the context of different environments (staging, prod, etc.).

For example, I want to attach AWSLambdaBasicExecutionRole (arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole) to different IAM roles in my staging and production environments defined by separate *.tf files (using modules). Since this resource is "all-or-nothing", only the last updated role gets attached to the policy, erasing all previous attachments.

The purpose of using AWS's managed policies is that the policy content might change or be updated over time, but that is all "managed" by AWS. That is why attaching it across users/groups/roles is preferred.

I think we should change this resource so that it appends attachments (not erase previous ones). Or at least have another resource that does.

I hope this makes it clear, @mitchellh

[LegNeato]

As a workaround, could you create a custom policy per-environment that simply extends the managed policy (with nothing!) and use that? It would then have a different ARN when calling aws_iam_policy_attachment multiple times and thus not overwrite. If that is possible, you get the benefits of the managed policy without hitting this issue.

[LondonAppDev]

I'm experiencing this same issue with Terraform 0.9.5.

[LondonAppDev]

OK so I managed to get this to work. Not sure if this is the same problem that others are having on this thread, but in-case it is, this is how I resolved it.

I was including the ${terraform.env} within the name for my aws_iam_policy_attachment like this:

resource "aws_iam_policy_attachment" "ecs-service-policy-attach" {
  name        = "${var.prefix}-${terraform.env}-ecs-service-policy"
  roles       = ["${aws_iam_role.ecs_role.name}"]
  policy_arn  = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

Turns out this was't necessary because I am simply using an existing AWS service role, so I changed it to this:

resource "aws_iam_policy_attachment" "ecs-service-policy-attach" {
  name        = "${var.prefix}-ecs-service-policy"
  roles       = ["${aws_iam_role.ecs_role.name}"]
  policy_arn  = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

To get this to work I had to comment out my aws_iam_policy_attachment blocks, apply on all envs (to remove the polocies).

Then I un-commented them and applied with the change above to re-create the policies and it worked.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.