Terraform: aws_iam_policy_attachment error when attaching amazon "managed" policies

[dansteen]

For the following config:

resource "aws_iam_role" "emr" {
    name = "matching-spark-emr-${var.env}"
    assume_role_policy = "${file("${path.module}/iam-policy-emr-trust-relationship.tpl")}"
}

# the generic role provided by amazon
resource "aws_iam_policy_attachment" "AmazonElasticMapReduceRole" {
  name = "AmazonElasticMapReduceRole"
  roles = [ "${aws_iam_role.emr.name}" ]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"
}

I Occasionally, get the following error:

* aws_iam_policy_attachment.AmazonElasticMapReduceRole: [WARN] Error updating user, role, or group list from IAM Policy Attachment AmazonElasticMapReduceRole:
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole was not found.
        status code: 404, request id: be6c2f10-f824-11e5-9241-eb1509222747

Note that I only seem to get this when attaching amazon "managed" policies to a role. A second run results in success. Also, as a note, the above code is in a module, and I call that module multiple times (despite the warning in the documentation about only using "aws_iam_policy_attachment" once per policy.

[jeekajoo]

Same here, using 0.6.14.
And also, a terraform plan is always showing changes for aws_iam_policy_attachment objects. Mine are provided by a home-made module.

plan:

~ module.ecs_gameserver.aws_iam_policy_attachment.ecs_role_policy
    roles.#:          "3" => "1"
    roles.1470799240: "ecs_instances_profile-foobar_stage-gameserver" => "ecs_instances_profile-foobar_stage-gameserver"
    roles.3133451130: "ecs_instances_profile-foobar_stage-main" => ""
    roles.695418250:  "ecs_instances_profile-foobar_stage-influxdb" => ""

~ module.ecs_gameserver.aws_iam_policy_attachment.ecs_services_policy
    roles.#:          "3" => "1"
    roles.1503848858: "ecs_services_role-foobar_stage-influxdb" => ""
    roles.2631330245: "ecs_services_role-foobar_stage-gameserver" => "ecs_services_role-foobar_stage-gameserver"
    roles.3765894190: "ecs_services_role-foobar_stage-main" => ""

~ module.ecs_influxdb.aws_iam_policy_attachment.ecs_role_policy
    roles.#:          "3" => "1"
    roles.1470799240: "ecs_instances_profile-foobar_stage-gameserver" => ""
    roles.3133451130: "ecs_instances_profile-foobar_stage-main" => ""
    roles.695418250:  "ecs_instances_profile-foobar_stage-influxdb" => "ecs_instances_profile-foobar_stage-influxdb"

~ module.ecs_influxdb.aws_iam_policy_attachment.ecs_services_policy
    roles.#:          "3" => "1"
    roles.1503848858: "ecs_services_role-foobar_stage-influxdb" => "ecs_services_role-foobar_stage-influxdb"
    roles.2631330245: "ecs_services_role-foobar_stage-gameserver" => ""
    roles.3765894190: "ecs_services_role-foobar_stage-main" => ""

~ module.ecs_main.aws_iam_policy_attachment.ecs_role_policy
    roles.#:          "3" => "1"
    roles.1470799240: "ecs_instances_profile-foobar_stage-gameserver" => ""
    roles.3133451130: "ecs_instances_profile-foobar_stage-main" => "ecs_instances_profile-foobar_stage-main"
    roles.695418250:  "ecs_instances_profile-foobar_stage-influxdb" => ""

~ module.ecs_main.aws_iam_policy_attachment.ecs_services_policy
    roles.#:          "3" => "1"
    roles.1503848858: "ecs_services_role-foobar_stage-influxdb" => ""
    roles.2631330245: "ecs_services_role-foobar_stage-gameserver" => ""
    roles.3765894190: "ecs_services_role-foobar_stage-main" => "ecs_services_role-foobar_stage-main"

apply:

Error applying plan:

4 error(s) occurred:

* aws_iam_policy_attachment.ecs_services_policy: [WARN] Error updating user, role, or group list from IAM Policy Attachment ecs_services_policy-foobar_stage-main:
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole was not found.
    status code: 404, request id: 3458bd20-fcbc-11e5-a2e0-67b29efeaa2f
* aws_iam_policy_attachment.ecs_role_policy: [WARN] Error updating user, role, or group list from IAM Policy Attachment ecs_instances_policy-foobar_stage-influxdb:
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role was not found.
    status code: 404, request id: 345a6a54-fcbc-11e5-aae8-1f0ddba5b8ec
* aws_iam_policy_attachment.ecs_role_policy: [WARN] Error updating user, role, or group list from IAM Policy Attachment ecs_instances_policy-foobar_stage-main:
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role was not found.
    status code: 404, request id: 3491f592-fcbc-11e5-8736-3b1470f43ab2
* aws_iam_policy_attachment.ecs_services_policy: [WARN] Error updating user, role, or group list from IAM Policy Attachment ecs_services_policy-foobar_stage-influxdb:
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole was not found.
    status code: 404, request id: 3493a36e-fcbc-11e5-becf-051c60f11c4a

[dansteen]

@jeekajoo do you call the aws_iam_policy_attachment resource more than once across your config? If you do, you will see changes each time you run a plan. Thats part of what the warning is in the docs...

[jeekajoo]

@dansteen I only use each aws_iam_policy_attachment once, in my module. I call this module 3 times.

resource "aws_iam_policy_attachment" "ecs_role_policy" {
    name = "ecs_instances_policy-${var.environment_name}-${var.cluster_type}"
    roles = ["${aws_iam_role.ecs_instances.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

resource "aws_iam_policy_attachment" "ecs_services_policy" {
    name = "ecs_services_policy-${var.environment_name}-${var.cluster_type}"
    roles = ["${aws_iam_role.ecs_services.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

Warnings/Errors disappear after 2 terraform apply.

[dansteen]

@jeekajoo each time the module is instantiated counts as a "call" of the aws_iam_policy_attachment resource, which means that you are calling it 3 times for each of those policies. This is the cause of the constant updates in your plan.

[jeekajoo]

@dansteen In my module, 2 different aws_iam_policy_attachment are attached to a 2 different roles.
aws_iam_policy_attachment are different accross each module calls because its name is variabilized.
Am I missing something?

Here is my module

resource "aws_ecs_cluster" "ecs" {
    name = "${var.environment_name}-${var.cluster_type}"
}

resource "aws_instance" "ecs" {
    ami = "${var.ami}"
    instance_type = "${var.instance_type}"
    iam_instance_profile = "${aws_iam_instance_profile.ecs.name}"
    key_name = "${var.key_name}"
    count = "${var.instance_count}"
    subnet_id = "${var.subnet_id}"
    associate_public_ip_address = true
    vpc_security_group_ids = ["${var.security_group_id}"]

    root_block_device {
        volume_type = "standard"
        volume_size = 50
    }

    connection {
        user = "admin"
    }

    #Instance tags
    tags {
        Name = "ecs-${var.environment_name}-${var.cluster_type}-${count.index}"
        Project = "${var.project_name}"
    }

    provisioner "remote-exec" {
        inline = [
            "echo ${var.instance_count} > /tmp/consul-server-count",
            "echo ${aws_instance.ecs.0.private_ip} > /tmp/consul-server-addr",
            "echo ${var.environment_name}-${var.cluster_type} > /tmp/ecs-cluster-name",
        ]
    }

    provisioner "remote-exec" {
        scripts = [
            "${path.module}/provisioner/grow_partition.sh",
            "${path.module}/provisioner/consul_install.sh",
            "${path.module}/provisioner/ecs_configure.sh",
            "${path.module}/provisioner/registrator_configure.sh",
            "${path.module}/provisioner/dnsmasq_restart.sh",
        ]
    }
}

# IAM Role, policy and instance profile for ECS cluster instances
resource "aws_iam_role" "ecs_instances" {
    name = "ecs_instances_profile-${var.environment_name}-${var.cluster_type}"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "ecs_role_policy" {
    name = "ecs_instances_policy-${var.environment_name}-${var.cluster_type}"
    roles = ["${aws_iam_role.ecs_instances.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

resource "aws_iam_instance_profile" "ecs" {
    name = "ecs_role-${var.environment_name}-${var.cluster_type}"
    roles = ["${aws_iam_role.ecs_instances.name}"]
}

# IAM Role and policy for ECS services
resource "aws_iam_role" "ecs_services" {
    name = "ecs_services_role-${var.environment_name}-${var.cluster_type}"
    assume_role_policy = <<EOF
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "ecs_services_policy" {
    name = "ecs_services_policy-${var.environment_name}-${var.cluster_type}"
    roles = ["${aws_iam_role.ecs_services.name}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

[dansteen]

@jeekajoo do you instantiate your module more than once?

[jeekajoo]

@dansteen Yes I did wrong. I got those IAM objects out of the module and I am reusing them accross 3 modules calls. Thank you Sir!

[jlambert121]

In my case, a 2nd run doesn't result in a success.

[CpuID]

Similar issue here with IAM Roles and aws_iam_role_policy_attachment. Tested on 0.7.0 and 0.7.2:

* aws_iam_role_policy_attachment.aws-ecs-describe-tags-ops-ci: [WARN] Error attaching policy arn:aws:iam::#############:policy/AWS-EC2-Describe-Tags to IAM Role ECS-Instance-Ops-CI: NoSuchEntity: Role ECS-Instance-Ops-CI was not found.
    status code: 404, request id: e288cfef-8693-11e6-b82a-45f985f17976
* aws_iam_role_policy_attachment.aws-ecs-clusterservicediscovery-ops-ci: [WARN] Error attaching policy arn:aws:iam::############:policy/AWS-ECS-ClusterServiceDiscovery to IAM Role ECS-Instance-Ops-CI: NoSuchEntity: Role ECS-Instance-Ops-CI was not found.
    status code: 404, request id: e288a808-8693-11e6-bf4f-7b2766e599ec
* aws_launch_configuration.ecs-ops-ci: Error creating launch configuration: ValidationError: You are not authorized to perform this operation.
    status code: 400, request id: e5c6e3d5-8693-11e6-a015-4fa4bc8b14f6

The 2 policy ARNs exist ahead-of-time in a separate .tfstate, due to reuse in different sections of our Terraform config topology.

.tf Logic:

# aws_iam_policy is defined in ../../ecs_iam/cluster_service_discovery.tf
resource "aws_iam_role_policy_attachment" "aws-ecs-clusterservicediscovery-ops-ci" {
  role = "ECS-Instance-Ops-CI"
  policy_arn = "arn:aws:iam::############:policy/AWS-ECS-ClusterServiceDiscovery"
}

# aws_iam_policy is defined in ../../ecs_iam/describe_tags.tf
resource "aws_iam_role_policy_attachment" "aws-ecs-describe-tags-ops-ci" {
  role = "ECS-Instance-Ops-CI"
  policy_arn = "arn:aws:iam::############:policy/AWS-EC2-Describe-Tags"
}

resource "aws_iam_role" "ecs-instance-ops-ci" {
    name = "ECS-Instance-Ops-CI"
    path = "/"
    assume_role_policy = <<EOF
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {"AWS": "*"},
            "Effect": "Allow",
            "Sid": ""
        }
    ]
}
EOF
}

Second run succeeds pretty much every time.

[ifuller1]

Hey guys - does anybody know any possible workarounds beyond creating multiple policies? Cheers.

[adamgotterer]

Having the same issue inside a module. Naming is all dynamic. Everything gets created properly, but something about aws_iam_policy_attachment causes the attachment to appear and then disappear on every apply.

I understand the warning in the docs. But should one use the aws_iam_policy_attachment in a module with the change that its run twice?

[adamgotterer]

I'm having the same issue even with all the policies broken out. Every time apply is run it modifies the state of the policy and every other time it just removes it entirely.

resource "aws_iam_policy_attachment" "ecs_public_instance_attach" {
  name = "public-instance-policy-attachment"
  roles = ["${module.ecs_public.instance_iam_profile_id}"]
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}
resource "aws_iam_policy_attachment" "ecs_public_service_attach" {
  name = "public-service-policy-attachment"
  roles = ["${module.ecs_public.service_iam_profile_id}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}
resource "aws_iam_policy_attachment" "ecs_private_instance_attach" {
  name = "private-instance-policy-attachment"
  roles = ["${module.ecs_private.instance_iam_profile_id}"]
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}
resource "aws_iam_policy_attachment" "ecs_private_service_attach" {
  name = "private-service-policy-attachment"
  roles = ["${module.ecs_private.service_iam_profile_id}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

~ aws_iam_policy_attachment.ecs_private_instance_attach
    roles.#:         "0" => "1"
    roles.453620460: "" => "ecs-production-private-instance-role"

~ aws_iam_policy_attachment.ecs_private_service_attach
    roles.#:          "0" => "1"
    roles.3670960167: "" => "ecs-production-private-service-role"

~ aws_iam_policy_attachment.ecs_public_instance_attach
    roles.#:          "0" => "1"
    roles.4033904570: "" => "ecs-production-public-instance-role"

~ aws_iam_policy_attachment.ecs_public_service_attach
    roles.#:          "0" => "1"
    roles.2905932258: "" => "ecs-production-public-service-role"

[adamgotterer]

Took me a few hours, but I finally understand what the docs mean. Hopefully this will help someone else:

The policy arn can only be called one time for ANY resource and ALL roles that need that policy need to be in a single resource. My above attempts turned into:

resource "aws_iam_policy_attachment" "ecs_instance" {
  name = "ecs-instance-policy-attachment"
  roles = [
    "${module.ecs_public.instance_iam_profile_id}",
    "${module.ecs_private.instance_iam_profile_id}"
  ]
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
}

resource "aws_iam_policy_attachment" "ecs_service" {
  name = "ecs-service-policy-attachment"
  roles = [
    "${module.ecs_private.service_iam_profile_id}",
    "${module.ecs_public.service_iam_profile_id}"
  ]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"
}

[ifuller1]

thanks @adamgotterer - so you should have one policy_attachment per policy and share the roles, makes perfect sense!

[adamgotterer]

Your welcome.

The issue (feature request!!) is what do you do if you need two modules that reference the same policy? You literally have to rip out the policy assignment and move it to the root level. Seems like something terraform should handle for you.

[clearly]

@adamgotterer You totally rule! Your explanation just saved a pretty big train wreck. Thanks!!!

[dvelitchkov]

This is awful, actually. I believe I'm seeing this issue here: #10500

Basically, the aws_iam_policy_attachment resource is a "all-or-nothing" type resource for the entire AWS account. I can't even have separate aws_iam_policy_attachment resources in separate environments.
Wow.

Worst part is, I don't believe there is any work-around for my situation.

[LegNeato]

I'm going to put how I have my stuff organized here in case it helps anyone 🔍

I hit this issue setting up CodeDeploy using managed policy ARNs and fixed it rather cleanly. The structure is general and isn't CodeDeploy-specific.

For others Googling, this is the error I was getting:

 aws_iam_policy_attachment.(name): [WARN] Error updating user, role, or group list from IAM Policy Attachment (name):
– NoSuchEntity: Policy arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole was not found.

So what I have is:

deploy module

• Handles the CodeDeploy configuration
• Does everything except aws_iam_policy_attachment on managed policy ARNs
• Exports the aws_iam_role service role it creates for CodeDeploy

frontend, backend and other service modules

• Sets up my app-specific clusters of machines, permissions, etc.
• Creates an aws_iam_role for the service that is used in the module's aws_iam_instance_profile
• Exports that aws_iam_role

deploy-workaround module

• Used because of this issue 😄
• Takes in service_role_names and instance_role_names
• Calls aws_iam_policy_attachment once, preventing this issue from manifesting

Basically:

# Give the service roles MANAGED policy/permissions for CodeDeploy.
resource "aws_iam_policy_attachment" "codedeploy-managed-attach" {
    name = "codedeploy-managed-attach"
    # Can't pass lists to modules, so we need to join and split.
    roles = ["${split(",", var.service_role_names)}"]
    policy_arn = "arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole"
}

# Give the instance roles MANAGED policy/permissions for CodeDeploy.
resource "aws_iam_policy_attachment" "codedeploy-managed-attach-instance" {
    name = "codedeploy-managed-attach-instance"
    roles = ["${split(",", var.instance_role_names)}"]
    policy_arn = "arn:aws:iam::aws:policy/AWSCodeDeployFullAccess"
}

main module

• Calls the service (frontend, backend, etc) modules
• Calls the deploy module once per service
• Calls the deploy-workaround module with the exported values
• Note: you could probably do some fancy looping and interpolation here if you want.

Basically:

module "backend" {
  source = "./backend"
  # ...
}

module "deploy_backend" {
   source = "./deploy"
   service_name = "backend"
   instance_role_name = "${module.backend.instance_role_name}",
}

module "frontend" {
   source = "./frontend"
   # ...
}

module "deploy_frontend" {
   source = "./deploy"
   service_name = "frontend"
   instance_role_name = "${module.frontend.instance_role_name}",
}

module "deploy-workaround" {
   source = "./deploy-workaround"
   # Can't pass lists to modules, so we need to join and split.
   service_role_names = "${join(",", list(
     "${module.deploy_frontend.service_role_name}",
     "${module.deploy_backend.service_role_name}",
   ))}"
   instance_role_names = "${join(",", list(
     "${module.frontend.instance_role_name}",
     "${module.backend.instance_role_name}",
   ))}"
}

Benefits

• Workaround for this issue is contained in one place
• deploy doesn't need to know anything about services or the deploy-workaround
• Services (frontend, backend, etc) don't need to know anything about this issue
• If this issue gets fixed you merely move the code from deploy-workaround into deploy
• No 404 when using managed profiles
• terraform apply is stable...the resources do not continually change

Feel free to ask me questions offline if something isn't clear.

[leftathome]

Wow, @LegNeato did a fantastic job breaking down this issue. Thank you so much for contributing that explanation. I just have one question for this audience:

In what circumstances would one want to use aws_iam_policy_attachment versus aws_(group|role|user)_attachment, resources which don't have this disclaimer on them?

This seems sort of similar to having an ebs_block_device declared as a property within aws_instance versus as an outboard aws_ebs_volume + aws_volume_attachment ... so would the circumstances be similar?

I suspect an experience-backed answer to this question would be of great interest to anyone guided here by search engines to this solution...

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.