Terraform: TerraForm does not set Instance Profile ARN for IAM roles

[jeroenjacobs79]

Example given:

resource "aws_iam_role_policy" "instance_policy" {
    name = "${var.prefix}-${var.project_name}-policy"
    role = "${aws_iam_role.instance_role.id}"
    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}
EOF
}


resource "aws_iam_role" "instance_role" {
    name = "${var.prefix}-${var.project_name}-role"
    assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

Trying to associate this role with an instance fails with the following error:

 aws_instance.node_instance.0: Error launching source instance: InvalidParameterValue: Value (test-cloudera-hadoop-role) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
    status code: 400, request id:

When I create the IAM role from the AWS console, everything works fine. However, I noticed something odd:

The role created via web console has the Instance Profile ARN set.
With the role created via TerraForm, Instance Profile ARN remained empty.

You can easily verify this when you open the roles in the web console.

[stack72]

hi @jeroenjacobs1205 , can you tell me if you are creating something like this as well:

resource "aws_iam_instance_profile" "app-server" {
    name  = "app-server"
    path  = "/"
    roles = ["${aws_iam_role.app-server.name}"]
}

It is the aws_iam_instance_profile that then can be associated with the Instance

[jeroenjacobs79]

Partially solved. Some of the instances come up fine, others don't. I assume there is a need for a delay before I actually create instances based on that IAM profile. Is there a way to instruct TerraForm to wait like 20sec before creating instances based on that IAM profile?

[jeroenjacobs79]

To be more exact, I have

count=7

In my instance resource block. 5 of them went fine. 2 of them failed because of the IAM error.

Would be nice if I can do a delay before it starts creating those 7 instances.

[stack72]

it's a bit hacky, but you can use the null_resource provisioner:

https://terraform.io/docs/provisioners/null_resource.html

Then in your resource_aws_instance you can put a depends_on the null_provisioner

[jeroenjacobs79]

Might work, I'll give it a try. I was considering to add the local-exec provisioner to my IAM profile resource, and let it do a "sleep 10". Does 'local-exec' waits until the local action finishes, or does it return immediately?

[stack72]

@jeroenjacobs1205 did this work for you?

[jen20]

@jeroenjacobs1205 provisioners will complete before moving on. If I'm understanding your report correctly though I'd definitely consider this as a bug. Do you have a runnable example that you can share for us to investigate with? If not I think we have enough information to be able to construct a test reproducing this.

[jen20]

We should probably use the waiter from the SDK here: https://github.com/aws/aws-sdk-go/blob/ce51895e994693d65ab997ae48032bf13a9290b7/service/iam/waiters.go#L9

[catsby]

Are you still able to reproduce this with the example given? There is code (here) in the instance resource to account for this, I'd be interested in seeing logs or otherwise finding why that code isn't doing it's job.

For example, when I apply this config, I get this output:

2015/12/09 08:48:40 [DEBUG] Invalid IAM Instance Profile referenced, retrying...
2015/12/09 08:48:43 [DEBUG] Invalid IAM Instance Profile referenced, retrying...
2015/12/09 08:48:45 [DEBUG] Invalid IAM Instance Profile referenced, retrying...
2015/12/09 08:48:48 [INFO] Instance ID: i-e339513a
2015/12/09 08:48:48 [DEBUG] Waiting for instance (i-e339513a) to become running
2015/12/09 08:48:48 [DEBUG] Waiting for state to become: running

The instances boot with no issue. Is my example missing something? I'm running this on the latest version of Terraform v0.6.8

[jeroenjacobs79]

I did another TerraForm run, and I'm unable to reproduce the problem this time. Very weird, as I destroyed the entire environment to make sure all resources would be re-created. I'm going to close this for now. I don't think the TerraForm team can do anything when I cannot reproduce the problem in a reliable way.

[stack72]

FWIW, I actually saw someone complaining in the hangops slack org yesterday about IAM Instance Profile's being slow with cloudformation as well

Maybe there was some sort of an issue yesterday:

Is there something special about IAM::InstanceProfile roles that cause them to take longer than other resources to create via CloudFormation?
I'm still fairly new to CF, but it seems consistent across multiple runs.

[catsby]

Thank you for following up @jeroenjacobs1205 – please let us know if you do end up being able to reproduce this. As I mentioned, there's code to prevent this (and maybe better code in #4235!) and I would love to know if/why it wasn't working :smile:

[toddZilla0130]

Hello,

It doesn't look like this was ever resolved because it couldn't be reliably reproduced but I can reproduce this same problem exactly with Terraform 0.9.7. darwin x64

This HCL script should produce a facsimile of the ecsServiceRole which is required as part of an EC2 Container Service (it's attached to the Launch Config):

data "aws_iam_policy_document" "ecsinstancerole_assume_role_policy" {
  statement {
    actions = [ "sts:AssumeRole" ]
    principals {
      type = "Service"
      identifiers = [ "ec2.amazonaws.com" ]
    }
  }
}

resource "aws_iam_role" "core_instance_role" {
  name = "ecsInstanceRole"
  assume_role_policy = "${data.aws_iam_policy_document.ecsinstancerole_assume_role_policy.json}"
}

data "aws_iam_policy_document" "ecs_instance_role_document" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = ["*"]
  }

  statement {
    actions = [
      "ecs:CreateCluster",
      "ecs:DeregisterContainerInstance",
      "ecs:DiscoverPollEndpoint",
      "ecs:Poll",
      "ecs:RegisterContainerInstance",
      "ecs:StartTelemetrySession",
      "ecs:Submit*",
      "ecs:UpdateContainerInstancesState",
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }
}

resource "aws_iam_policy" "ecs_instance_role_policy" {
  name = "ecsInstancePolicy"
  policy = "${data.aws_iam_policy_document.ecs_instance_role_document.json}"
}

resource "aws_iam_role_policy_attachment" "ecs_instance_role" {
  role = "${aws_iam_role.core_instance_role.name}"
  policy_arn = "${aws_iam_policy.ecs_instance_role_policy.arn}"
}

Everything is created successfully except for the Instance Profile ARN so the subsequent Launch Config fails with this error:

Error applying plan:

1 error(s) occurred:

* aws_launch_configuration.default: 1 error(s) occurred:

* aws_launch_configuration.default: Error creating launch configuration: ValidationError: Invalid IamInstanceProfile: ecsInstanceRole
    status code: 400, request id: 3548f8a0-4b9e-11e7-8e15-fdddaabc8e58

There does not appear to be a readily available workaround within Terraform.

This might also be useful information: I am _not_ accessing the account directly; rather, I'm accessing it via sts:assume-role.

Thanks!

[jen20]

This might just be IAM's eventual consistency - if you run a subsequent apply does it work?

[toddZilla0130]

Sadly, no. Even after waiting and attempting to force it with this command line:

terraform apply -target=aws_iam_role.core_instance_role -refresh=true

has no effect.

[jen20]

Actually that's correct - you aren't creating an instance profile. The thing with the name ecsInstanceRole needs to be an iam_instance_profile rather than just a policy. The correct code here (from your example) is:

data "aws_iam_policy_document" "ecsinstancerole_assume_role_policy" {
  statement {
    actions = [ "sts:AssumeRole" ]
    principals {
      type = "Service"
      identifiers = [ "ec2.amazonaws.com" ]
    }
  }
}

data "aws_iam_policy_document" "ecs_instance_role_document" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = ["*"]
  }

  statement {
    actions = [
      "ecs:CreateCluster",
      "ecs:DeregisterContainerInstance",
      "ecs:DiscoverPollEndpoint",
      "ecs:Poll",
      "ecs:RegisterContainerInstance",
      "ecs:StartTelemetrySession",
      "ecs:Submit*",
      "ecs:UpdateContainerInstancesState",
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role" "core_instance_role" {
  name = "ecsInstanceRole"
  assume_role_policy = "${data.aws_iam_policy_document.ecsinstancerole_assume_role_policy.json}"
}

resource "aws_iam_role_policy" "core_instance" {
  name = "SomethingDescriptive"
  role = "${aws_iam_role.core_instance_role.id}"
  policy = "${aws_iam_policy_document.ecs_instance_role_document.id}"
}

resource "aws_iam_instance_profile" "core_instance" {
    name = "SomethingDescriptive"
    role = "${aws_iam_role.core_instance_role.id}"
}

Then make the launch configuration point at ${aws_iam_instance_profile.core_instance.id} and all should be well,

As an aside, I would strongly consider dropping type names from the end of resource names. They are never referred to independently of their type, so aws_iam_role.core_instance_role is unnecessarily noisy compared to aws_iam_role.core_instance.

[toddZilla0130]

Still having no luck with this. I made the name changes you suggested and my code now looks like this:

data "aws_iam_policy_document" "ecsinstance_assume_role" {
  statement {
    actions = [ "sts:AssumeRole" ]
    principals {
      type = "Service"
      identifiers = [ "ec2.amazonaws.com" ]
    }
  }
}

data "aws_iam_policy_document" "ecs_instance_role" {
  statement {
    actions = [
      "s3:Get*",
      "s3:List*"
    ]
    resources = ["*"]
  }

  statement {
    actions = [
      "ecs:CreateCluster",
      "ecs:DeregisterContainerInstance",
      "ecs:DiscoverPollEndpoint",
      "ecs:Poll",
      "ecs:RegisterContainerInstance",
      "ecs:StartTelemetrySession",
      "ecs:Submit*",
      "ecs:UpdateContainerInstancesState",
      "ecr:GetAuthorizationToken",
      "ecr:BatchCheckLayerAvailability",
      "ecr:GetDownloadUrlForLayer",
      "ecr:BatchGetImage",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role" "core_instance" {
  name = "SomethingSomethingDarkside"
  assume_role_policy = "${data.aws_iam_policy_document.ecsinstance_assume_role.json}"
}

resource "aws_iam_role_policy" "core_instance" {
  name = "SomethingDescriptive"
  role = "${aws_iam_role.core_instance.id}"
  policy = "${aws_iam_policy_document.ecs_instance_role_document.json}"
}

resource "aws_iam_instance_profile" "core_instance" {
  name = "ecsInstanceRole"
  role = "${aws_iam_role.core_instance_role.id}"
}

Still no Instance Profile ARN for the role however.

[toddZilla0130]

Wait, possibly a big NVM on this - I was editing code in one directory but running code in a different directory.

[toddZilla0130]

@jen20 Thanks for your assistance on this!

I did have to make one small change in aws_iam_role_policy: the policy needs to be "${data.aws_iam_policy_document.ecs_instance_role_document.json}"

While this creates an inline policy for the role and our MO is typically standalone policies, it's such a one-off inlining it is fine.

I'll also follow your advice on naming in the future.

Cheers!

[jen20]

I've updated the code for anyone coming to this in future - good catch! You can also do this via attachments - the important thing is the instance profile not the inlining. Hope this helped!

[toddZilla0130]

Now I'm seeing the error from this thread: https://github.com/hashicorp/terraform/issues/7198. Rerunning terraform apply doesn't help (as it seems to for some others).

Yay.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.