Terraform: Terraform does not use IAM Role for ECS Task as credential provider

[iwat]

Terraform Version

Terraform v0.7.3

Affected Resource(s)

• aws_alb_target_group
• aws_security_group
• a lot

This affects all AWS related command.

Terraform Configuration Files

resource "aws_security_group_rule" "demo_pri_ingress_vpn_service" {
    security_group_id = "${aws_security_group.demo_pri.id}"
    type = "ingress"

    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["${data.terraform_remote_state.infra.vpn-cidr_block}"]
}

Debug Output

https://gist.github.com/iwat/df0b0ebfe2f8db62adfd5953bfd6b92c

Panic Output

None

Expected Behavior

It should work by using IAM Role for ECS Task.
awscli works

Actual Behavior

It was using EC2 Instance Role which does not allow this action.

Error retrieving Target Group: AccessDenied: User: arn:aws:sts::872767853649:assumed-role/myrole/i-0223aeb98c19f2d0d

Steps to Reproduce

• Setup an EC2, do not provide any critical IAM action.
• Setup ECS task, provide required IAM action for testing.
• Try AWSCLI inside the running ECS task, it should work fine.
• Run terraform on AWS ECS Task.
  

  Important Factoids

None

References

• http://docs.aws.amazon.com/de_de/AmazonECS/latest/developerguide/task-iam-roles.html
• https://github.com/hashicorp/terraform/blob/v0.7.3/builtin/providers/aws/auth_helpers.go#L97
• https://github.com/hashicorp/terraform/blob/v0.7.3/builtin/providers/aws/auth_helpers.go#L128
• https://github.com/aws/aws-sdk-go/blob/v1.4.8/aws/defaults/defaults.go#L100

[bacoboy]

A little context to help the enhancement along. I ran into this while trying to run a terraform command from the new AWS CodeBuild service (which is running on an AWS hosted ECS cluster farm it seems).

In the newer versions of the AWS SDK, they've added one more "location" to scan for IAM keys to support IAM roles on docker containers.

The launched docker containers, if they have an IAM role, get passed an extra environment variable like this: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/v2/credentials/895e903e-0672-4c41-bdc8-ef0c3b37d178' This is a relative path to the ECS agent running on the EC2 instance which has been NATed via iptables to address http://169.254.170.2/. This means that the container can get its metadata (includes IAM keys) using that relative URL from the ECS agent like this:

$ curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{
    "AccessKeyId": "ACCESS_KEY_ID",
    "Expiration": "EXPIRATION_DATE",
    "RoleArn": "TASK_ROLE_ARN",
    "SecretAccessKey": "SECRET_ACCESS_KEY",
    "Token": "SECURITY_TOKEN_STRING"
}

Since all the SDK's have been updated to scan this location as well as the usual paths, it just seems to work like magic when you run AWS commands.

This sample, and all the gory details are from here

Hopefully this is enough to get somebody going on the terraform enhancement. Any takers?

UPDATE: Of course, now that I just typed all that, it seems that if you are using the aws-sdk-go package, you just need to update the dependency version. These seem to be the minimums

[imyousuf]

Faced the problem from within AWS CodeBuild with v0.8.6 as well.

[s0enke]

Facing the same problem. It seems that the Terraform AWS credential logic is custom and does not use the default
AWS provider chain (https://github.com/hashicorp/terraform/blob/351c6bed79abbb40e461d3f7d49fe4cf20bced41/builtin/providers/aws/auth_helpers.go#L94-L204). This custom chain does not include ECS task yet.

[s0enke]

I blogged about using Terraform within CodeBuild, which includes a workaround for this problem: https://www.ruempler.eu/2017/02/26/continuous-infrastructure-delivery-pipeline-aws-codepipeline-codebuild-terraform/

[danmandle]

This may be a little off-topic, but here's my workaround for use with Jenkins Pipelines (Groovy)

import groovy.json.JsonSlurperClassic

@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node {
  stage('AWS Creds Please'){
    def awsCredsJSON = sh(returnStdout: true, script: "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")

    def awsCreds = jsonParse(awsCredsJSON)

    env.AWS_ACCESS_KEY_ID = awsCreds.AccessKeyId
    env.AWS_SECRET_ACCESS_KEY = awsCreds.SecretAccessKey
  }
}

[apparentlymart]

Thanks for this feature request @iwat, and thanks to everyone else for the great info that followed.

Terraform is using the official Go SDK for AWS but is customizing the set of valid credential sources. To implement this I expect we would need to upgrade the SDK (assuming we didn't already do that for some other reason) and add one more credential provider to the list in the Terraform AWS provider.

Since this one is gated on the presence of an environment variable it should be safe to add without any unintended consequences for those not using ECS.

The Terraform team doesn't have any immediate plans to work on this but if someone else had the time or motivation we would love to review a PR!

[jekh]

I submitted PR #14199 for this. A couple questions in the PR, but hopefully this will help get terraform working with CodeBuild/ECS slaves.

[jch254]

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

[vinujan59]

I can confirm that terraform trying to assume the role of EC2's instead of ECS's

Below are some logs
2020/03/03 11:46:52 [INFO] ECS container credentials detected, RemoteCredProvider added to auth chain
2020/03/03 11:46:52 [INFO] AWS EC2 instance detected via default metadata API endpoint, EC2RoleProvider added to the auth chain
2020/03/03 11:46:53 [INFO] AWS Auth provider used: "EC2RoleProvider"

terraform version: 0.12.21

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.