Terraform: S3 backend config role_arn can't be assumed when running dockerized

[antonosmond]

Terraform Version

Terraform v0.9.3

Affected Resource(s)

• backend

Terraform Configuration Files

terraform = {
  required_version = "~> 0.9.3"

  backend "s3" {
    region     = "eu-west-1"
    bucket     = "anto-terraform-state"
    key        = "platform/network.tfstate"
    role_arn   = "arn:aws:iam::000000000000:role/terraform"
  }
}

Debug Output

Initializing the backend...

Error configuring the backend "s3": The role "arn:aws:iam::000000000000:role/terraform" cannot be assumed.

There are a number of possible causes of this - the most common are:
* The credentials used in order to assume the role are invalid
* The credentials do not have appropriate permission to assume the role
* The role ARN is not valid

Please update the configuration in your Terraform files to fix this error
then run this command again.

Expected Behavior

The backend role_arn is assumed and the backend is successfully initialized

Actual Behavior

The backend role_arn can't be assumed

Steps to Reproduce

1. Set AWS credentials ensuring they have access to assume the role e.g.
   export AWS_ACCESS_KEY_ID=**************
export AWS_SECRET_ACCESS_KEY=**************************

2. cd to directory with backend config

3. run terraform init (NOT dockerized) to ensure the credentials are set correctly and can assume the specified role - backend should be initialized successfully.

4. run the same initialization but dockerized, ensuring the same credentials are passed to the container and the backend config in the current directory is available e.g.

docker run -ti --rm \
  --volume "$(pwd):/terraform" \
  --workdir "/terraform" \
  -e "AWS_ACCESS_KEY_ID" \
  -e "AWS_SECRET_ACCESS_KEY" \
  "hashicorp/terraform:0.9.3" \
  init

[antonosmond]

I've also tried just launching the terraform container interactively e.g.

docker run -ti --rm \
  --volume "$(pwd):/terraform" \
  --workdir "/terraform" \
  -e "AWS_ACCESS_KEY_ID" \
  -e "AWS_SECRET_ACCESS_KEY" \
  --entrypoint sh \
  "hashicorp/terraform:0.9.3"

ensuring the backend config is available, setting the credentials and running terraform init and get the same error.

[jabley]

I'm wondering whether there is a time-based aspect to this issue?

I restarted Docker for Mac, rebooted Mac, and the problem was still apparent.

It disappeared the next day.

It intermittently reappears.

[xakraz]

Same issue with Terraform v0.10.8

[ggermis]

getting the same issue even when running non-dockerized

$ terraform version
Terraform v0.11.0

I have my ~/.aws/credentials file setup with several profiles. I use the default profile to assume roles on other accounts

[default]
aws_access_key_id = MY_KEY
aws_secret_access_key = MY_SECRET
region = eu-west-1

From the aws cli it works and I get my temporary credentials:

$ aws --profile default sts assume-role --role-arn arn:aws:iam::111111111111:role/terraform --role-session-name terraform
{
    "Credentials": {
        "AccessKeyId": "TEMP_KEY",
        "SecretAccessKey": "TEMP_SECRET",
        "SessionToken": "SESSION_TOKEN",
        "Expiration": "2017-11-30T10:13:15Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "ID:terraform",
        "Arn": "arn:aws:sts::111111111111:assumed-role/terraform/terraform"
    }
}

The only AWS_ variables set in my environment is:

$ env | grep AWS
AWS_PROFILE=default

My terraform backend configuration looks like:

terraform {
  required_version = ">= 0.11.0"
  backend "s3" {
    region          = "eu-west-1"
    bucket          = "terraform"
    key             = "dev/vpc.tfstate"
    encrypt         = "true"
    profile         = "default"
    role_arn        = "arn:aws:iam::111111111111:role/terraform"
    session_name    = "terraform"
  }
}

When trying to run a $ terraform init I get the "role cannot be assumed" error. Trace output for terraform init:

...
2017/11/30 10:15:04 [INFO] Building AWS region structure
2017/11/30 10:15:04 [INFO] Building AWS auth structure
2017/11/30 10:15:04 [INFO] Setting AWS metadata API timeout to 100ms
2017/11/30 10:15:04 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2017/11/30 10:15:04 [INFO] Attempting to AssumeRole arn:aws:iam::111111111111:role/terraform (SessionName: "terraform", ExternalId: "", Policy: "")
2017/11/30 10:15:04 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
Error initializing new backend:
2017/11/30 10:15:24 [DEBUG] plugin: waiting for all plugin processes to complete...
Error configuring the backend "s3": The role "arn:aws:iam::111111111111:role/terraform" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Please update the configuration in your Terraform files to fix this error
then run this command again.

My policy does not require MFA

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": [
            "arn:aws:iam::111111111111:role/terraform"
        ]
    }
}

I tried removing the .terraform subdirectory, and re-running $ terraform init but no change..

The weird thing is that it did work yesterday. I have setup and tore down my environment repeatedly the last few days... The code is in version control, and I haven't changed my ~/.aws/credentials and today it just stopped working

I did play around with installing boto3 and boto through pip, but am assuming this won't impact behaviour of terraform since it's a go application

Any insights into this issue would be much appreciated

[ggermis]

it turned out to be a network issue.. the strange thing is that the aws cli did work, while terraform didn't.. guess it's using a different network stack to do DNS resolving or something

[jsthibault]

Hi,

I have the same problem, did you find any solutions ?

it seems like, sometimes it worked and sometimes it's doesn't. It randomly works.

Regards,

[jvanwygerden]

Having the same issue - appears to be intermittent. Is there a known resolution for this issue

[ken5scal]

Hi, I'm experiencing the same issue

[yogeshdass]

It works great with Terraform version v0.13.3. Here is how I am doing it

terraform {
backend "s3" {
bucket = "terraform-states"
region = "ap-south-1"
profile = "terraform"
key = "prod/terraform.tfstate"
role_arn = "arn:aws:iam::037912942455:role/terraform"
session_name = "terraform"
}
}

profile "terraform" user has permission to just assume the terraform role

[ashkv92]

@yogeshdass Setting the profile corresponding to the role defined under .aws/config helped in my case. thanks for the pointer!!