Terraform-provider-aws: Doesn't ask MFA token code when using assume_role with MFA required

[jsi-p]

When using multiple AWS accounts it's good practice to only allow access via AssumeRole from a master account. This can be done with or without requiring MFA. Terraform supports assume_role with s3 state file and aws provider configurations, but doesn't seem to ask the MFA token code when one is required. This prevents using AssumeRole for credentials when MFA is required.

AWS documentation describing MFA with cross account AssumeRole: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_configure-api-require.html#MFAProtectedAPI-cross-account-delegation

Terraform Version

$ terraform --version
Terraform v0.11.0

Affected Resource(s)

Both of these support assume_role, so they should also support asking for MFA token code:

• S3 backend configuration
• AWS provider configuration

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "terraform-state-bucket"
    key = "tf-state"
    region = "eu-west-1"
    role_arn = "arn:aws:iam::916005212345:role/OrganizationAccountAccessRole"
  }
}
provider "aws" {
  region = "eu-west-1"
  assume_role {
    role_arn = "arn:aws:iam::916005212345:role/OrganizationAccountAccessRole"
    session_name = "terraform-session"
  }
}

Actual Behavior (with DEBUG)

$ TF_LOG=debug terraform init
2017/11/23 13:36:12 [INFO] Terraform version: 0.11.0  
2017/11/23 13:36:12 [INFO] Go runtime version: go1.9.2
2017/11/23 13:36:12 [INFO] CLI args: []string{"/usr/local/Cellar/terraform/0.11.0/bin/terraform", "init"}
2017/11/23 13:36:12 [DEBUG] Attempting to open CLI config file: /Users/***/.terraformrc
2017/11/23 13:36:12 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2017/11/23 13:36:12 [INFO] CLI command args: []string{"init"}
2017/11/23 13:36:12 [DEBUG] command: loading backend config file: /Users/***

Initializing the backend...
2017/11/23 13:36:12 [WARN] command: backend config change! saved: 16375281725947963338, new: 2383462577283113429
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now reconfigure for this backend. If you didn't
intend to reconfigure your backend please undo any changes to the "backend"
section in your Terraform configuration.


2017/11/23 13:36:12 [INFO] Building AWS region structure
2017/11/23 13:36:12 [INFO] Building AWS auth structure
2017/11/23 13:36:12 [INFO] Setting AWS metadata API timeout to 100ms
2017/11/23 13:36:13 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2017/11/23 13:36:13 [INFO] Attempting to AssumeRole arn:aws:iam::***:role/OrganizationAccountAccessRole (SessionName: "", ExternalId: "", Policy: "")
2017/11/23 13:36:13 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2017/11/23 13:36:13 [DEBUG] plugin: waiting for all plugin processes to complete...
Error initializing new backend: 
Error configuring the backend "s3": The role "arn:aws:iam::***:role/OrganizationAccountAccessRole" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Please update the configuration in your Terraform files to fix this error
then run this command again.

[jsi-p]

This comment describes the same issue (while they workaround by not using MFA, which we do not want to do):

https://github.com/hashicorp/terraform/issues/11270#issuecomment-292179117

[jsi-p]

I would also like to see this work when simply using AWS_PROFILE env with the target profile configured to use a cross account role_arn and mfa_serial. In fact this would be the preferred way to use terraform for us.

$HOME/.aws/credentials:

[default]
aws_access_key_id = ***
aws_secret_access_key = ***

[subaccount]
role_arn = arn:aws:iam::***:role/OrganizationAccountAccessRole
source_profile = default
mfa_serial = arn:aws:iam::***:mfa/[email protected]

Terraform invocation:

AWS_PROFILE=subaccount terraform init

[tamsky]

FYI use of shared [profile]s via ~/.aws/config only landed a few days ago in:

https://github.com/terraform-providers/terraform-provider-aws/pull/1608

Comment from that PR:

This won't support assuming roles where an MFA device has been specified for the profile (in ~/.aws/config)

There was another implementation of #1608 -- it appears to include some (possibly limited) MFA support, but was closed in favor of 1608:

https://github.com/terraform-providers/terraform-provider-aws/pull/1590/files#diff-0460e055ea98f6ed9713f2265a1b1d49R266

Getting the go-sdk to prompt (once, and only once) for an MFA code would appear to involve a bit more work, but not much, by my reading of the comments.

[tamsky]

https://github.com/aws/aws-sdk-go/pull/1088 adds the StdinTokenProvider which we'd use.

[tamsky]

Also, caching of the temporary sts credentials in a portable cross-process manner might be an open issue: https://github.com/aws/aws-sdk-go/issues/1329

[tamsky]

awless's implementation of 15min temporary credential caching:
https://github.com/wallix/awless/issues/109#issuecomment-314381704
in
https://github.com/wallix/awless/commit/5a732239d7b8acaee4996b6f8197cf60b1a41872

[danlsgiga]

Just had the same issue... Have a profile with a role to be assumed and MFA enabled but it seems MFA is not supported at this moment.
This would be a great addition to our setup.

We don't really need it to be prompted (probably should not even be considered)... working just like Packer works would be sufficient and that is having a mfa_code option where we can pass using -var or env vars.

https://www.packer.io/docs/builders/amazon-ebs.html#mfa_code

[mikemoate]

@tamsky I tried to add the MFA functionality into #1608 (we ideally wanted to be using MFA) but you are correct that the lack of an STS credentials cache essentially makes this unusable for the end user (repeated prompts for MFA token, and as tokens can only be used once, you have to wait until a new token is generated each time).

Since the inability to assume roles defined in the profile file was hurting us more, I elected to pursue that fix for now (still hasn't quite been released, see https://github.com/hashicorp/terraform/pull/16661), though I might return to the MFA issue when I next have some spare capacity (if someone else hasn't stepped in to solve it).

(the original PR https://github.com/hashicorp/terraform/pull/11734 that I resurrected to form #1608, had some discussion on MFA which led to me experimenting and finding the limitations)

[apparentlymart]

Hi all!

In general, Terraform doesn't currently support this sort of dynamic prompting for providers, and indeed the existing prompting features in Terraform are there primarily to help new users get started and we expect users to switch pretty early to using Terraform as a non-interactive, scripted tool, ideally running in a centrally-administered environment to avoid the need to sprawl various secrets across multiple user workstations.

Some users have reported success using a wrapper/helper script that calls sts:GetSessionToken to obtain temporary credentials in return for an MFA token, and then passes those _temporary_ credentials to Terraform via environment variables, thus allowing that wrapper program to use whatever interactive means is appropriate (CLI prompt, web UI, etc) to collect that MFA token. This is, indeed, the methodology used by Terraform Enterprise for its AWS MFA support.

Managing authentication use-cases in wrapper scripts allows for other more advanced use-cases too, such as SAML authentication.

At this time we do not have plans to support interactive authentication to providers since it would require some significant changes to the provider model. We will probably look at this again in future in the context of some other breaking change to the provider model so that we can bundle multiple changes together, though as noted above we generally expect Terraform to be used non-interactively in some sort of automation once users pass the experimental phase, so any features we build in this area will need to also solve for a non-interactive workflow for that common case.

[phil-hachey]

For my purposes, I needed this feature for AWS org resource management across multiple accounts, so I can easily bootstrap new accounts. Another use case is for DevOps to manage resources across accounts originating from a "security" account (as described here). In all scenarios, MFA should be enabled for cross account access (internal requirement).

I ended up getting this to work by requesting temporary credentials using MFA and saving them to ~/.aws/credentials. With these credentials, the "MFA required" condition has been met and you're free to use the assume_role block of the aws provider in Terraform:

provider "aws" {
  alias  = "dev_account"
  region = "${var.aws_region}"

  assume_role {
    role_arn = "${var.dev_account_target_role_arn}"
  }
}

I created a simple python cli to generate the creds:
https://github.com/phil-hachey/aws-mfa-tool

[scalp42]

Has anyone tried working around the issue by using something like https://github.com/remind101/assume-role ?

[kmaris]

@scalp42 assume-role works. I also used a smaller and simpler script to do about the same: https://gitlab.com/kmaris/wtf (The name is terrible, I know).

[andreas-venturini]

I ended up getting this to work by requesting temporary credentials using MFA and saving them to ~/.aws/credentials

I suggest aws-vault for this purpose as it is much more convenient. aws-vault stores your long-term access credentials encrypted and will deal with exposing temporary AWS access credentials to your shell or application so you don't have to. As a matter of fact, you can delete your credentials file and manage AWS API access entirely w/ aws-vault and named profiles defined in ~/.aws/config.

I'll show an example that uses cross account IAM role delegation for executing terraform commands.
OrganizationAccountAccessRole is the IAM role that authorized IAM users from an AWS master account can assume. The trust policy of the IAM role enforces MFA (set aws:MultiFactorAuthPresent boolean flag as described here)

The terraform AWS provider config:

provider "aws" {
  profile = "${var.aws_vault_profile}"
  region  = "${var.aws_region}"
  version = "~> 1.30.0"

  assume_role {
    role_arn = "arn:aws:iam::${var.aws_app_account_id}:role/OrganizationAccountAccessRole"
  }
}

Here is the corresponding ~/.aws/config file. It defines a terraform profile which uses MFA (the [profile ...] block is automatically created by aws-vault when adding a new profile - you just have to fill in the arn of the MFA device).

[default]
aws_region=eu-central-1

[profile terraform]
mfa_serial=arn:aws:iam::123456789012:mfa/Andreas

Then simply use aws-vault exec to wrap the terraform command - or set up an alias like so (note to change the backend depending on your system environment - this example uses the secret service API w/ keyring which works well for me on Ubuntu):

# always use aws-vault to wrap terraform command
alias terraform='aws-vault exec terraform --backend=secret-service -- terraform'

When you run terraform you will be automatically prompted for the access token:

➜  terraform-setup (master) ✗ terraform workspace list                                                          
Enter token for arn:aws:iam::123456789012:mfa/Andreas: 

Of course you can also use multiple profiles w/ aws-vault. There are more advanced configuration examples in the aws-vault docs.

[jonscheiding]

We can work around this using aws-vault or something similar, but I'd love to see the idea of interactive MFA authentication revisited (both for AWS and for other cloud providers where applicable).

I strongly disagree with @apparentlymart's rationale from 2 years ago. While iterating on a TF configuration, I might terraform apply it against our dev account dozens of times over the course of a day. Doing that through our build/deploy pipeline is an obvious non-starter.

Of course we use that pipeline for deployment to environments after dev. So it's not a matter of using TF interactively to manage our infrastructure. But we're contractually obligated to enforce MFA even in our dev account. (And we use account federation with centralized IAM management so it's not really possible to selectively enforce it anyway.) So as is, it makes our dev workflow more complex.

I can't imagine our workflow or our requirements are particularly unusual, so it seems like a high value add for TF to support it.

[stumyp]

My 2 cents:

• My workflow requires MFA even for CLI actions.
• I want to be able to use multiple providers with assume_role to a different role/account to manage different resources. It is especially important for account/organization management where multiple accounts are unavoidable.
• While I understand the point of non-interactive flow. I'd really like to see the possibility of using environment variable for specifying MFA value, something like this:

AWS_MFA_TOTP=123456 terraform apply

so if assumed role requires MFA we can work around it by using variable and if that role is not protected with MFA, it will not be used.

Thanks

[yn-academia]

There is no need to prompt for Terraform to support asking for a MFA value. If you run AWS_PROFILE=profile_that_requires_mfa aws iam list-users from the AWS CLI tool, and enter your token into the CLI tool, you will then find your temporary credentials cached in ~/.aws/cli/cache, so Terraform could just use that, and keep around the requirement that terraform is non-interactive.

[yobert]

Re @stumyp

• While I understand the point of non-interactive flow. I'd really like to see the possibility of using environment variable for specifying MFA value, something like this:

The problem with this is that an MFA token is only allowed to be used once in AWS and so this would effectively limit terraform to only one API call every 30 seconds.

[stumyp]

The problem with this is that an MFA token is only allowed to be used once in AWS and so this would effectively limit terraform to only one API call every 30 seconds.

This is incorrect: once temporary session is obtained it can lasts much longer.
see assume role API call

[yobert]

The problem with this is that an MFA token is only allowed to be used once in AWS and so this would effectively limit terraform to only one API call every 30 seconds.

This is incorrect: once temporary session is obtained it can lasts much longer.
see assume role API call

Aha I see what your saying. I thought you meant an environment variable for the MFA code would be sufficient to replace the temporary credential storage workflow. My bad!

[sir-farfan]

I got here because I also need Terraform to ask the MFA key before assuming the role.
Guess it's still not possible...

[dac73]

I found some old script and updated it a bit for a workaround, if anyone needs. It basicaly creates temporary credentials profile with assumed role.
https://github.com/dac73/terraform-session-token

[stumyp]

The goal is not be able to use assumed role with MFA enforcement.
The goal is to be able to use it in terraform, along with other roles and/or main role used for this assumption.

Example1:

• I deploy organization master account and create child accounts under
• I use assumed role to dive into that new account and configure it as well
• I have more than one child account to manage within the same terrafrom project

Example2:

• I create resources in one AWS account
• I also need to create Route53 records to point to those resources and DNS zone is hosted in another AWS account and I use assumed role to switch to it within the same terraform project

The manual assume_role is not so hard to do:

#!/bin/bash

set -e

TMP="/tmp/$$-assumedrole"
AWS=`command -v aws`
DURATION=14400

usage() {
    cat <<EOH
Usage: $0 -r role_arn [-n session name] [-t mfa_token [-m mfa_device_index]]

Where:
  role_arn: the ARN of the role you want to assume
  session name: an optional name of the session that is opened by this script in STS. Defaults to "from_the_script"
  mfa_token: TOTP code from MFA
  mfa_device_index: If you have more than one MFA (especial hardware one) number of the device. Defaults to "0"
EOH
exit 0
}
while getopts "r:n:t:m:d:" arg ; do
    case $arg in
        r)
            ROLE=$OPTARG
        ;;
        n)
            SESSION_NAME=$OPTARG
        ;;
        t)
            TOKEN="$OPTARG"
        ;;
        m)
            MFA=$OPTARG
        ;;
        d)
            DURATION=$OPTARG
        ;;
    esac
done

[ -n "$ROLE" ] || usage 
[ -n "$SESSION_NAME" ] || SESSION_NAME="from_the_script"

if [ -n "$TOKEN" ] ; then
    [ -n "$MFA" ] || MFA="0"
    MFA_SERIAL=$($AWS iam list-mfa-devices | jq -r '.MFADevices['$MFA'].SerialNumber')
    EXTRAOPTS="--token-code $TOKEN --serial-number $MFA_SERIAL"
fi

$AWS  sts assume-role --role-arn $ROLE --role-session-name $SESSION_NAME --duration-seconds $DURATION $EXTRAOPTS >$TMP

export AWS_ACCESS_KEY_ID=`jq -r '.Credentials.AccessKeyId' <$TMP`
export AWS_SECRET_ACCESS_KEY=`jq -r '.Credentials.SecretAccessKey' <$TMP`
export AWS_SESSION_TOKEN=`jq -r '.Credentials.SessionToken' <$TMP`

echo "export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID"
echo "export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY"
echo "export AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN"
rm -r $TMP

[yn-academia]

@stumyp If you are willing to write multi-line shell scripts, then I think what you want is possible by installing aws-vault, then generating multiple sessions ahead of time, and then running terraform and from inside terraform shelling out to aws-vault as an external provider and configuring the terraform-aws provider with the credentials that aws-vault returns back to you.

[stumyp]

@yn-academia If you don't need this specifc feature, it doesn't mean other people don't need it as well.

This multiline script was made for single switch between main role and assumed role. one way, no way back. In terraform I want to work with multiple accounts easily. Which I do right now, but without MFA. This weakens our security policy. I want to make my work more secure and do not overcomplicate it at the same time.

[dac73]

@stumyp I have exactly that problem, and what my solution does is it creates tf profiles with an assumed role in ~/.aws/credentials so I can then use named profiles in terraform code. So basically I run the script against multiple assumed roles once and then I can use terraform across multiple accounts/assumed roles (until tokens expire). It's not a solution for only one account/assumed role.

[stumyp]

Thank you, @dac73, I was not looking for the workaround.

I'm just adding one more voice to implement fully STS AssumeRole API call in Terraform. Not in the way it is done right now.

Currently TF supports only three input parameters:

    role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
    session_name = "SESSION_NAME"
    external_id  = "EXTERNAL_ID"

while API call accepts more.:

• DurationSeconds
• Policy
• PolicyArns.member.N
• SerialNumber
• Tags.member.N
• TokenCode
• TransitiveTagKeys.member.N

Probably it should be a separate issue to add the rest of the parameters, but I would love to see there

• DurationSeconds
• SerialNumber
• TokenCode

Let the users deal with how they want to provide those values to TF, just add support for it.

[stumyp]

@apparentlymart should I create separate issue for what I'm asking for, since this particular one is about dynamic prompting for MFA value?

[lapkritinis]

I support requesting for MFA input. Profiles, etc are good for single provider, but thats not always the case. For example I have a need to build VPC peering between two accounts. Each of them requires MFA. Unless I setup some cross account trust in AWS - there is no way to achieve that using environment variables as all of them - will be for single account only.

[mdforbes500]

Also here to support implementing MFA input. When implementing AWS CLI accounts, MFA is a must here.

[scalp42]

@andreas-venturini the issue is that yes aws-vault will work but in our scenario we have to consume the state in different accounts.

Assuming you have 2 accounts prod and dev with respective states, S3 buckets and credentials/roles:

data "terraform_remote_state" "prod_us_west_2" {
  backend = "s3"

  config = {
    profile = "prod"
    bucket  = "terraform"
    key     = "prod/us-west-2/main.tfstate"
    region  = "us-west-2"
  }
}

data "terraform_remote_state" "dev_us_west_2" {
  backend = "s3"

  config = {
    profile = "dev"
    bucket  = "terraform"
    key     = "dev/us-west-2/main.tfstate"
    region  = "us-west-2"
  }
}

If you wrap the terraform command:

aws-vault exec prod -- terraform plan

It'll work as long as you don't have a provider that references a state from a different account if it makes sense as it'll assume whatever role in the prod account but then try to access the dev account as the assumed role if it makes sense and will fail.

Without MFA/aws-vault, it'll work as Terraform can handle multiple sessions per providers.

[olfway]

Different accounts can be used with aws-vault as a credential_process for aws sdk, ie

~/.aws/config

[profile root]
mfa_serial = arn:aws:iam::XXX:mfa/olfway
credential_process = env AWS_SDK_LOAD_CONFIG=0 aws-vault --backend=pass --pass-prefix=aws-vault --prompt=osascript exec --region eu-west-1 --duration=1h root --json

[profile prod]
source_profile = root
parent_profile = root
role_arn = arn:aws:iam::YYY:role/Prod

[profile dev]
source_profile = root
parent_profile = root
role_arn = arn:aws:iam::ZZZ:role/Dev

and then run terraform as

export AWS_SDK_LOAD_CONFIG=1
terraform plan

[scalp42]

@olfway I tried to replicate the same set up but I'm running into the following:

Error: error configuring S3 Backend: Error creating AWS session: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.

What does your provider configuration looks like?

[olfway]

provider "aws" {
  profile = "prod"
  region  = "eu-west-1"
  version = "3.4.0"
}
terraform {
  required_version = "0.13.2"
  backend "s3" {
    profile        = "prod"
    region         = "eu-west-1"
    dynamodb_table = "terraform-locks"
    bucket         = "prod-terraform-states"
  }
}

Also you could check with

export AWS_SDK_LOAD_CONFIG=1
aws --profile prod sts get-caller-identity

it should use credentials from aws-vault and ask for mfa only once

or run aws-vault directly:

env AWS_SDK_LOAD_CONFIG=0 aws-vault --some-backend-options --prompt=osascript exec --region eu-west-1 --duration=1h root --json