Aws-vault: InvalidClientTokenId error when creating IAM role with vault credentials and terraform

[skeller88]

OS: 10.13.4 (High Sierra)
Terraform version: v0.11.7
provider.aws: v1.26.0

The problem:
When I attempt to create an IAM role via terraform while using vault credentials for a user called "terraform", a 403 error occurs.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault --debug exec terraform -- terraform apply --auto-approve 
2018/07/09 22:43:27 [keyring] Considering backends: [keychain file]
2018/07/09 22:43:27 Loading config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Parsing config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Looking for sessions for terraform
2018/07/09 22:43:27 Looking up all keys in keyring
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2018/07/09 22:43:27 [keyring] Found 3 results
2018/07/09 22:43:27 Session "terraform session (1531204691)" expires in 54m43.768571954s
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", account="terraform session (1531204691)", keychain="aws-vault.keychain"
2018/07/09 22:43:30 [keyring] Found item "aws-vault session for terraform"
2018/07/09 22:43:30 Using session ********************, expires in 54m40.363844684s
2018/07/09 22:43:30 Setting subprocess env: AWS_DEFAULT_REGION=us-west-2, AWS_REGION=us-west-2
2018/07/09 22:43:30 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2018/07/09 22:43:30 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"

Error: Error applying plan:

1 error(s) occurred:

* aws_iam_role.iam_for_lambda: 1 error(s) occurred:

* aws_iam_role.iam_for_lambda: Error creating IAM Role iam_for_lambda: InvalidClientTokenId: The security token included in the request is invalid
    status code: 403, request id: 3124a1fa-8404-11e8-b8c9-0f4f651e0f50

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Here are the terraform debug logs just before the error:

2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Accept-Encoding: gzip
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Action=GetRole&RoleName=iam_for_lambda&Version=2010-05-08
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetRole Details:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: HTTP/1.1 403 Forbidden
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Connection: close
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Length: 305
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Type: text/xml
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Date: Tue, 10 Jul 2018 05:46:53 GMT
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: X-Amzn-Requestid: a68ca1e8-8404-11e8-b6a6-e1ba4d916180
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">

I am able to successfully create the same IAM role via terraform by hardcoding the "terraform" user credentials into the provider definition in the .tf file.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply  --auto-approve

aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"
aws_iam_role.iam_for_lambda: Creation complete after 1s (ID: iam_for_lambda)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

tf file

provider "aws" {
  region     = "us-west-2"
}

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

~/.aws/config file

[profile terraform]
mfa_serial = arn:aws:iam::207283649878:user/terraform
region = us-west-2
source_profile = terraform

What I've tried

Creating other resources with the aws-vault credentials and terraform. I'm able to successfully create s3 buckets and a vpc.

resource "aws_s3_bucket" "terraform_test"{
  bucket = "terraform-test-1983451"
}

resource "aws_vpc" "test" {
  # https://serverfault.com/questions/630022/what-is-the-recommended-cidr-when-creating-vpc-on-aws
  # "...there is no harm in starting with a small prefix such as /16 because you can always create subnets."
  cidr_block = "11.0.0.0/16"

  # enabled by default, but just to be sure
  enable_dns_support   = true
  enable_dns_hostnames = true
}

It seems like the security token generated by aws-vault has different permissions than I'm expecting. But I'm not sure what additional debugging steps to take.

There's another issue that addresses security tokens, but I don't think the solution applies to me because the poster had two roles: https://github.com/99designs/aws-vault/issues/262

[FernandoMiguel]

Sounds like you are missing MFA

[skeller88]

Ok I added MFA and a role according to the Usage documentation. Now my .tf file looks like this:

provider "aws" {
  region     = "us-west-2"
  profile = "deployment_user"
  assume_role {
    role_arn     = "arn:aws:iam::[account-number]:role/deployment"
    session_name = "assume_deployment_role"
  }
}

And my ~/.aws/config file:

[profile deployment_user]
mfa_serial = arn:aws:iam::[account-number]:mfa/deployment_user
region = us-west-2
source_profile = deployment_user
role_arn = arn:aws:iam::[account-number]:role/deployment

I added the "deployment_user" access key and secret to the vault.

What policy does the "deployment_user" need to assume the "deployment" Role? I tried this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::[account-number]:role/deployment"
        }
    ]
}

But get an error:

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account-number]:mfa/deployment_user: [token-entered]

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[account-number]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

I copied role ARN from the "deployment" role via the AWS console, and I just added the AWS secret and key for "deployment_user" to the vault in a fresh bash session. So the user credentials must not have the right permissions. Any ideas?

[FernandoMiguel]

You seem to be miss understanding how aws-vault works.
Aws-vault assumes a role using an IAM account.
Your terraform then has a provider that runs under that role, or has a provider (or alias) to assume yet another role.
Here's an example
https://github.com/FernandoMiguel/kb/blob/master/PortoLinux/ET-2017Dez/terraform/main.tf

[skeller88]

I get that. I'm trying to use the "deployment_user" IAM account to assume
the "deployment" role. But I am not defining the "deployment_user" with the
right credentials to assume the "deployment" role.

On Mon, Jul 9, 2018 at 11:53 PM Fernando Miguel [e-mail]
wrote:

You seem to be miss understanding how aws-vault works.
Aws-vault assumes a role using an IAM account.
Your terraform then has a provider that runs under that role, or has a
provider (or alias) to assume yet another role.
Here's an example

https://github.com/FernandoMiguel/kb/blob/master/PortoLinux/ET-2017Dez/terraform/main.tf

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/99designs/aws-vault/issues/266#issuecomment-403719733,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQFAtjfJ9rGYPsNDhEz0RUhf5ud8feuks5uFE9RgaJpZM4VIzNo
.

[frezbo]

AWS requires to use an MFA to perform IAM operation with an assume-role and you should not specify the profile in the TF file.

[skeller88]

I'm using MFA. I tried again without the profile in the TF file

provider "aws" {
  region     = "us-west-2"
  assume_role {
    role_arn     = "arn:aws:iam::[account]:role/deployment"
    session_name = "terraform"
  }
}

and still get the error.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: 903223
aws-vault: error: Failed to get credentials for deployment_user: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code. 
    status code: 403, request id: afad2a3f-8463-11e8-a9fd-d188f32cdfae
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: 000725

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[account]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

[frezbo]

What does this give:

aws-vault exec deployment_user -- aws sts assume-role --role-arn <arn> --external-id <if there is one> --role-session-name dummy

Make sure to remove the creds when posting :wink:

[frezbo]

@skeller88 I just saw your .aws/config file. Two questions:

• The source_profile is the same as your current deployment_profile
• Assuming the file content was wrong, why are you again trying to assume the role in TF?. aws-vault already gives you the assumed creds.

[skeller88]

Can we DM? I'm on gmail at skeller88[at]gmail[dot]com

[skeller88]

What's the difference between "source_profile" and the first line of the ~/.aws/config file that has "profile" in it? I'm referring to

[profile deployment_user]
mfa_serial=arn:aws:iam::207283649878:mfa/deployment_user

Should I be giving the profile name a different name?

I executed the sts command:

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- aws sts assume-role --role-arn arn:aws:iam::[account]:role/deployment  --role-session-name dummy

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::[account]:assumed-role/deployment/[id] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[account]:role/deployment
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: [mfa-code]

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[acccount]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Assuming the file content was wrong, why are you again trying to assume the role in TF?. aws-vault already gives you the assumed creds.

I'm copying the format of the tf file in https://github.com/99designs/aws-vault/issues/262. Once the file content is right and I can properly assume the role with aws-vault, I'll want Terraform to assume that role as well.

Here's my understanding of the aws-vault process from the documentation:

• Create an IAM role with the desired AWS permissions. In my case, that role is "deployment_role". It has access to all the AWS resources I need to manipulate with Terraform.
• Create an IAM user. In my case, that user is "deployment". Enable MFA for the user.
• Create a policy that gives the IAM user the permission to assume the "deployment_role". In my case, that policy is "assume_deployment_role":

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::[account]:role/deployment"
    }
}

• Add a trust policy to the "deployment" Role that gives the "deployment_user" User permission to assume the role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::[account]:user/deployment_user"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}

• Add the "deployment_user" id and secret to the aws-vault. I did so under the name "deployment_user".
• Configure my ~/.aws/config profile to use the appropriate mfa role and assume the appropriate role

[profile deployment_user]
mfa_serial=arn:aws:iam::[account]:mfa/deployment_user
region=us-west-2
role_arn=arn:aws:iam::[account]:role/deployment

Do you see anything I'm doing wrong?

[frezbo]

All your understanding is perfectly right. The source profile is the profile credentials used or assuming the role. Sorry email is real hard to follow up

[skeller88]

Ok. How about telegram or discord? I'm pretty stuck on this and would love
to use the aws- vault as a tool.

On Tue, Jul 10, 2018 at 11:55 AM Noel Georgi [e-mail]
wrote:

All your understanding is perfectly right. The source profile is the
profile credentials used or assuming the role. Sorry email is real hard to
follow up

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/99designs/aws-vault/issues/266#issuecomment-403929595,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQFAgDNdKszs4HBX9XC63DZr0UDDGklks5uFPiFgaJpZM4VIzNo
.

[frezbo]

slack?

[skeller88]

Sure.
On Tue, Jul 10, 2018 at 10:18 PM Noel Georgi [e-mail]
wrote:

slack?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/99designs/aws-vault/issues/266#issuecomment-404047999,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQFAoRcxZTUz25l9H461L3Gads4ufJEks5uFYqsgaJpZM4VIzNo
.

[frezbo]

I'm available on the k8s and chef slack channels

[0xdabbad00]

From the error message

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::[account]:assumed-role/deployment/[id] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[account]:role/deployment

you already assumed into a role.

Run:

aws-vault exec deployment_user -- aws sts get-caller-identity

[skeller88]

I get the following response. The Account matches my account and the Arn
matches the "deployment_user" Arn, but the UserId does not match any of the
access key IDs for "deployment_user".

{
"UserId": "[deployment_user access id]",
"Account": "[account]",
"Arn": "arn:aws:iam::[account]:user/deployment_user"
}

On Wed, Jul 11, 2018 at 11:31 AM Scott Piper [e-mail]
wrote:

From the error message An error occurred (AccessDenied) when calling the
AssumeRole operation: User:
arn:aws:sts::[account]:assumed-role/deployment/[id] is not authorized to
perform: sts:AssumeRole on resource: arn:aws:iam::[account]:role/deployment
you already assumed into a role.

Run:

aws-vault exec deployment_user -- aws sts get-caller-identity

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/99designs/aws-vault/issues/266#issuecomment-404267722,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFQFAmKAx4w8Z7El4X0e6z6hukr_TE2eks5uFkSJgaJpZM4VIzNo
.

[0xdabbad00]

Don't worry about the UserId that is its own thing, but I think there may be something to look into with regard to the error message saying your identity was arn:aws:sts::[account]:assumed-role/deployment/[id] but get-caller-identity saying it is arn:aws:iam::[account]:user/deployment_user .

[lox]

I would guess the issue is layering of an STS session and then trying to assume a role with that.

aws-vault is designed with a primary objective of "protect your root credentials". E.g. only disclose short-lived temporary credentials to subprocesses. There are two ways to do this, either creating an STS session and providing those credentials to the subprocess, or assuming a role and providing the session credentials form that to the subprocess.

AWS has complicated rules about chaining temporary credentials together with assuming roles, or assuming roles from other roles.

If you really want, you can disable the session that aws-vault creates with aws-vault exec <profile> --no-session, but I think it waters down a lot of the security that it provides.

In terms of your specific problem, I'd suggest troubleshooting it by getting the simplest config you can working and then adding complexity to it. Start without Terraform, get aws-vault simply assuming a role in your CLI. Prove that you can run something like aws s3 ls or an equivalent simple command.

Note that your config should look something like:

[profile ljd]
region=us-east-1

[profile ljd-admin]
region=us-east-1
source_profile=ljd
mfa_serial=arn:aws:iam::xxx:mfa/lachlan.donald
role_arn=arn:aws:iam::xxx:role/assume-admin-access

The source_profile refers to the profile to look for the credentials in. In this case, credentials are against my IAM user, in myljd profile. I then use an ljd-admin profile to assume a role, via the credentials in the ljd profile.

[skeller88]

Here's what I've tried:

Deleted my ~/.aws/config and ~/.aws/credentials files.
Deleted all aws-vault credentials.
Created a new IAM user, test, with no MFA.
Gave the test user permissions for all the AWS services I need.
Successfully executed the following commands from the command line:

aws-vault exec test -- aws sns list-topics
aws-vault exec test -- aws ssm get-parameters --names database_credentials
aws-vault exec test -- aws lambda list-functions
aws-vault exec test -- aws rds describe-instances
aws-vault exec test -- aws s3 ls

Even though I added the "IAMFullAccess" policy to the "test" user permissions, the following command still fails with the same error I saw at the start of this issue:

> aws-vault exec test -- aws iam list-roles 
An error occurred (InvalidClientTokenId) when calling the ListRoles operation: The security token included in the request is invalid

I debugged all of the requests and looked at what's different between the aws iam request and the other requests. The session token is the same. I don't see any noticeable differences other than the fact that the request is denied in the iam case.

I'm able to access iam without aws-vault.

aws configure
AWS Access Key ID [None]: xxx
AWS Secret Access Key [None]: xxx
Default region name [None]: us-west-2
Default output format [None]: 
aws iam list-roles 
{
    "Roles": [
     ...
}

Here's my ~/.aws/config file

[profile deployment]
region=us-west-2
[default]
region = us-west-2

Anything else I could try?

[frezbo]

yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it's authenticated with an MFA

[skeller88]

I fixed my issue last week. Thanks everyone for the help. I solved it by authenticating the IAM role with MFA, and taking out the "role_arn" section of the aws provider in my .tf file.

[geoffreywiseman]

yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it's authenticated with an MFA

Just so I better understand -- aws-vault is always using assumed roles, even if you've just set it up with an access key / secret, the same way that you might have done with aws cli? So with the example with test above, if you go directly through aws cli, there's no assumed role, but if you use aws-vault there is, even if you use the same credentials? I'm hitting this same issue, and just trying to understand what's different between the two paths.

[FernandoMiguel]

@geoffreywiseman yss, unless you use --no-session

[geoffreywiseman]

Ok, thanks. I also found this issue, #455, which also set me on a good path (in case anyone else ends up looking at this issue). I added mfa_serial to my .aws/config without doing any work to add roles on to of my IAM user, and that has helped.