Terraform: Instance Profile has no associated IAM Roles for some instances

[dpaq]

I think I'm seeing a race condition when using instance profiles.

In this case there were 5 instances (foo.0, foo.1, bar.0, bar.1, bar.2) that all use the same InstanceProfile. Two of them work fine but three of them claim the instance profile has no role associated:

 aws_route_table_association.testing: Creation complete
 aws_instance.bar.1: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.bar.2: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.foo.0: Error: 1 error(s) occurred:

 * Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::123123:instance-profile/test_bucket_access_instance_profile" has no associated IAM Roles
  status code: 400, request id: []
 aws_instance.bar.0: Provisioning with 'file'...
 aws_instance.foo.1: Provisioning with 'file'...

I'm wondering if this is an eventual consistency issue and this particular error launching should just include some retry logic. Obviously the InstanceProfile does have a Role since it works for some of the instances.

This is not repeatable regularly and just applying again gets me my missing instances.

A slightly redacted portion of the config:

{
resource "aws_instance" "foo" {
  ami = "${var.baseAMI}"
  instance_type = "${var.fooInstanceType}"
  count = "${var.numFooInstances}"
  key_name = "${var.keyName}"
  subnet_id = "${aws_subnet.foo.id}"
  vpc_security_group_ids = ["${aws_security_group.default.id}", "${aws_security_group.ssh.id}"]
  tags {
      Name = "${format("foo-%d", count.index)}"
  }
  iam_instance_profile = "${aws_iam_instance_profile.test_bucket_access_instance_profile.name}"
  root_block_device {
    delete_on_termination=true
  }
  provisioner "file" {
    connection {
      key_file = "${var.sshKeyFilename}"
      user = "${var.amiUsername}"
    }
    source = "mystuff"
    destination = "/home/${var.amiUsername}/mystuff"
  }
}

resource "aws_iam_instance_profile" "test_bucket_access_instance_profile" {
    name = "test_bucket_access_instance_profile"
    roles = ["${aws_iam_role.test_bucket_access_role.name}"]
}

resource "aws_iam_role" "test_bucket_access_role" {
  name = "test_bucket_access_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "test_bucket_access_policy" {
  name = "test_bucket_access_policy"
  role = "${aws_iam_role.test_bucket_access_role.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::${var.s3BucketName}"
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::${var.s3BucketName}/*"
    }
  ]
}
EOF
}

[dpaq]

This was seen with 0.5.3.

[josh-padnick]

We're also seeing this issue, however we _can_ deterministically reproduce it.

1. Launch our AWS resources via Terraform. Works fine.
2. Update the Launch Configuration for 3 different instances and terraform apply. This causes the following error message:

Error applying plan:

4 error(s) occurred:

* aws_instance.nat: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::941285601314:instance-profile/bryan-stephens-iam-nat-profile" has no associated IAM Roles
    status code: 400, request id: []
* aws_route_table.private-app-1: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 1 % var.num_nat_instances)}"
    status code: 400, request id: []
* aws_route_table.private-app-0: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 0 % var.num_nat_instances)}"
    status code: 400, request id: []
* aws_route_table.private-app-2: InvalidInstanceID.Malformed: Invalid id: "${element(split(",", module.nat.nat_instances_string), 2 % var.num_nat_instances)}"
    status code: 400, request id: []

In our case, the change to Launch Configurations that we made had nothing to do with the NAT instance IAM Instance Profile mentioned in the error.

Note that we are using at least 5 different nested modules. We're on terraform 0.6.3.

Correction: This is in fact intermittent, corroborating the eventual consistency issue that @dpaq suggested. Also, we were using 0.6.2. If this issue is resolved with 0.6.3 we'll report back as such.

[gdeconto]

we are also seeing this issue (as described by dpaq) when we deploy new AWS resources via terraform 0.6.3

[quater]

Encountering this issue with 0.6.3 as well. It appears to be an intermittent race condition. This problem is observed with an Terraform environment when created from scratch. Running the same terraform apply then results in a successful deployment.

[vancluever]

+1 on this, seeing it on S3 resources possibly, upon creation of the IAM resources needed for the principal the bucket creation is performed, and when the policy application is attempted it barfs.

[catsby]

@vancluever do you have a configuration that reproduces this? If so, can you please open another issue? As it stands, IAM propagation issues need to be handled in a resource-by-resource manner, for the time being.

This particular issue (#2660) has been resolved with #2037, so I'm going to close it.

Let me know if anyone is still hitting this

[vancluever]

@catsby check out my PR #4103 for the fixes I did for this - I will re-run this with the current upstream and paste the errors, with an sanitized config (I'll just append this stuff to the PR, if that's okay).

[errordeveloper]

I am seeing this with 0.6.10.

[ickymettle]

also still seeing behaviour occasionally in 0.6.12

[bmurtagh]

+1 with @ickymettle... Seeing it occasionally in 0.6.12.

[liquid-sky]

Encountered this in 0.6.14 on every run. Had to switch back to 0.6.8 for modules to work properly.

[pdecat]

Just encountered this issue with 0.6.16.

Retrying an apply worked it around.

[brikis98]

Seeing this in 0.6.16. When combined with #5200, it makes the aws_instance resource nearly unusable if you want to attach an aws_iam_role and aws_iam_instance_profile. First "I get a diffs didn't match" bug, then after reapplying I get a "has no associated IAM Roles" bug.

[brikis98]

@catsby Should this issue be reopened?

[tdeheurles]

@catsby
Hi, having the same issue Instance Profile has no associated IAM Roles.
I confirm that it's intermittent but raise really often. I need some terraform destroy/terraform apply to have it work.

The problem is that intermittent doesn't sound really pro :-/
Maybe it could be reopen.

[toddmichael]

Ran into this issue yesterday while creating a 4 node application stack. 1 out of 4 failed because of this issue. When the other nodes finished building, I re-ran terraform apply with no config changes and everything worked as expected. I tried again this morning with a single node, one that worked fine last night, and ran into it again. Same behavior: failed on first apply and then worked 30 seconds later when I re-ran. Definitely something awry here. If I discern a pattern, will follow up with more details. Thanks.

[tdeheurles]

Just to add an understanding of the issue: https://github.com/hashicorp/terraform/issues/4306.
It seems that AWS IAM are eventually consistent, so terraform have a feeback of IAM created and then we have an issue at the instance creation time.

[cmlad]

We are also seeing this on 0.6.16, for now our workaround is to add:

    provisioner "local-exec" {
      command = "sleep 90"
    }

[cmlad]

@catsby you are incorrect, #2037 fixes a slightly different problem. Here the issue is that IAM profile has propagated, but the role has not, and the error is:

* aws_instance.central: Error launching source instance: InvalidParameterValue: IAM Instance Profile "arn:aws:iam::198832413611:instance-profile/cmlad-sleep-general" has no associated IAM Roles
        status code: 400, request id: bb9f4992-4123-4a0a-8abe-f12560d71819

2037 fixes the case where the IAM profile has not propagated. The error there is:

 aws_instance.node_instance.0: Error launching source instance: InvalidParameterValue: Value (test-cloudera-hadoop-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
    status code: 400, request id:

The below retry line searches specifically for the Invalid IAM Instance Profile string which does not correspond to the error message in this issue.

if awsErr.Code() == "InvalidParameterValue" && strings.Contains(awsErr.Message(), "Invalid IAM Instance Profile") {

I will try to create a PR in the next few days. Hope you agree to reopen.

[sidhurana]

HI All.. faced same issue.. while creating e2 instance it was going through iam_instance_profile and was giving me different errors such as :- iam role already present, Invalid IAM Instance Profile name..

In my case the problem was instance type.. I was trying to launch m4.large instance in eu-west-1a and that availability zone was running full with m4.large instance types i.e you cannot launch more m4.large instances in same AZ for may be particular time period. As soon as i changed instance type, then error was gone and instance created..

IAM Instance Role creation is prior and parallel process while instance creation hence giving me strange/irrelevant IAM creation erros.

[realflash]

@sidhurana try https://github.com/terraform-providers/terraform-provider-aws/issues/838

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.