Aws-vault: Unable to assume role with MFA

[samuelhwilliams]

Hi. This looks like just the tool I'm after, but currently having some issues getting it working with a role that uses MFA. I'm not sure I see what I'm doing wrong.

I've just installed v4.6.0 on macOS 10.14.5. As you can see below, using the AWS CLI directly is fine, and I get prompted for the MFA token. But I don't get prompted, and therefore seem to get AccessDenied, when using aws-vault.

From what I've read in the docs, it seems like this is totally supported in theory. Am I doing anything obviously wrong?

~/.aws/config

[profile root]
region=eu-west-1

[profile role-with-mfa]
source_profile = root
role_arn=arn:aws:iam::xxx:role/infrastructure
mfa_serial=arn:aws:iam::xxx:mfa/sam

sam:~/git/eff-infrastructure/accounts [master]$ aws-vault exec root -- aws sts get-caller-identity
{
    "UserId": "xyz",
    "Account": "xxx",
    "Arn": "arn:aws:iam::xxx:user/sam"
}

sam:~/git/eff-infrastructure/accounts [master]$ aws-vault exec role-with-mfa -- aws sts get-caller-identity
aws-vault: error: Failed to get credentials for root (source profile for role-with-mfa): AccessDenied: Access denied
    status code: 403, request id: cd493634-8d53-11e9-a8f2-395e97d62828

sam:~/git/eff-infrastructure/accounts [master]$ AWS_PROFILE=root aws sts get-caller-identity
{
    "UserId": "xyz",
    "Account": "xxx",
    "Arn": "arn:aws:iam::xxx:user/sam"
}

sam:~/git/eff-infrastructure/accounts [master]$ AWS_PROFILE=role-with-mfa aws sts get-caller-identity
Enter MFA code for arn:aws:iam::xxx:mfa/sam:
{
    "UserId": "xyz:botocore-session-1560372747",
    "Account": "xxx",
    "Arn": "arn:aws:sts::xxx:assumed-role/infrastructure/botocore-session-1560372747"
}

[samuelhwilliams]

Followup: removing the requirement for MFA from the role's trust relationship leads to aws-vault exec role-with-mfa -- aws sts get-caller-identity working again. So it does feel like an issue with aws-vault's handling of MFA. Am I missing something, or is this not properly/fully supported?

[FernandoMiguel]

By enforcing MFA, your root profile needs mfa too

[samuelhwilliams]

@FernandoMiguel as in, in ~/.aws/config? I did try including it there - and it made no difference.

It feels like my setup is fine, as using the aws cli it works correctly. It only breaks fails aws-vault?

Sorry. Maybe I'm misunderstanding what you're saying.

[samuelhwilliams]

From the main readme in this repo, it suggests the following (equivalent) config is fine:

[profile read-only]
region=us-east-1

[profile admin]
source_profile = read-only
role_arn = arn:aws:iam::xxx:role/admin-access
mfa_serial = arn:aws:iam::xxx:mfa/jonsmith

[FernandoMiguel]

Do you have any credentials file in your aws folder?

[samuelhwilliams]

Yes - but even if I remove that file (or add the mfa serial to the root block), the same problem seems to persist.

[xose]

I had the same problem. For me the issue was that if the profile name is "default", aws-vault will only look for a [default] block on the config file, never [profile default], so it wasn't picking up the MFA details.

[samuelhwilliams]

@xose Hmm, interesting. I've just tried dropping the word profile from the block titles, for both [profile root] and [profile role-with-mfa], and I'm still getting 403 when MFA is enabled on the role.

[samuelhwilliams]

I've found a workaround: as long as I don't specify hierarchical profiles (i.e. I remove the source_profile = root link from the role-with-mfa profile, then it works fine - I get prompted for MFA and it assumes the role just great.

So I can get around this issue. But it doesn't feel like it's fixed, so would still be nice to work out what's going wrong, as this workaround requires duplication of e.g. region and other config.

[FernandoMiguel]

as this workaround requires duplication of e.g. region and other config.
@samuelhwilliams you always have to put region. that doesnt get pass down from the source_profile

[samuelhwilliams]

Ah, good to know - thanks! :)

[jstewmon]

@samuelhwilliams the error you posted:

aws-vault: error: Failed to get credentials for root (source profile for role-with-mfa)

Looks like you may not have added your access key and secret key to your keychain with aws-vault add? I don't think aws-vault will use statically configured credentials in ~/.aws/credentials.

[samuelhwilliams]

@jstewmon I had added the credentials for the root profile (which is proven by the logs provided where I could use aws-vault as root to make an sts call). When I tried to add the role-with-mfa as separate credentials, while still saying it's source_profile was root, aws-vault was returning an error:

sam:~/git [master]$ aws-vault add role-with-mfa
aws-vault: error: Your profile has a source_profile of root, adding credentials to role-with-mfa won't have any effect

So I think I've added all the creds I should have to aws-vault already.

[jstewmon]

Ah, right, sorry - I was in a hurry when I commented and missed that command.

By default, aws-vault will try to call sts:GetSessionToken before assuming the role. Maybe the AccessDenied error is coming from that call. You could verify that your root user can get a session token with aws --profile root sts get-session-token.

You can also try using the --no-session flag with exec: aws-vault exec --no-session role-with-mfa -- aws sts get-caller-identity?

[jstewmon]

I'm able to reproduce this issue. I'm stepping through the code now to figure out the cause.

[jstewmon]

This a regression from https://github.com/99designs/aws-vault/pull/369

When the selected profile has mfa_serial and the root does not have mfa_serial, the session we create for the source_profile won't use mfa, and we try to assume role using that session without mfa.

I can work on a fix for this today.

You can move your mfa_serial setting to your root profile, and things will work as expected for now.

[samuelhwilliams]

@jstewmon Ah, amazing. Thanks. :) I'll just move mfa to the root until a fix gets released.

[schof]

If you want to downgrade to the earlier version (which works perfectly well for me) then you can do the following:

brew cask uninstall aws-vault
brew cask install https://raw.githubusercontent.com/Homebrew/homebrew-cask/7e805ba6d2a4899b56ddf60174b54b16863fb1d3/Casks/aws-vault.rb

[gawbul]

I'm still experiencing this issue in v4.6.3.

Running without --no-session gives me the following:

aws-vault: error: Failed to get credentials for read-only (source profile for admin): AccessDenied: Access denied
    status code: 403, request id: 73998596-bffe-11e9-bdb0-694f38a9977a

The following works fine and requests the MFA token as expected, however:

aws-vault --debug exec --no-session admin -- aws sts get-caller-identity

[jstewmon]

@gawbul can you share the relevant entries from your aws config file with account numbers scrubbed?

If you recently upgraded, you might need to remove an old session: aws-vault remove -s read-only

[sylwit]

I'm also facing this issue with

$ aws-vault --version
v4.6.3

[profile work]
region=ca-central-1

[profile backup]
role_arn = arn:aws:iam::8XXXXXXXXX:role/BackupRDS
mfa_serial=arn:aws:iam::8XXXXXXXXX:mfa/[email protected]
source_profile = work

$ aws-vault --debug exec backup -- aws sts get-caller-identity                                                        
2019/08/24 13:35:29 [keyring] Considering backends: [secret-service]                                                        
2019/08/24 13:35:29 Loading config file /home/sylvain/.aws/config                                                           
2019/08/24 13:35:29 Parsing config file /home/sylvain/.aws/config                                                           
2019/08/24 13:35:29 Looking for sessions for backup
2019/08/24 13:35:29 Looking up all keys in keyring
2019/08/24 13:35:29 Session not found in keyring for backup
2019/08/24 13:35:29 Looking for sessions for work
2019/08/24 13:35:29 Looking up all keys in keyring
2019/08/24 13:35:29 Session not found in keyring for work
2019/08/24 13:35:29 Looking up keyring for work
Enter token for arn:aws:iam::8XXXXXXXXX:mfa/[email protected]: 300691                                         
2019/08/24 13:35:38 Using region "ca-central-1" from profile
2019/08/24 13:35:38 Getting new session token for profile work                                                              
2019/08/24 13:35:38 Writing session for work to keyring: "work session (1566682538)"                                        
2019/08/24 13:35:38 Using session ****************L36N, expires in 3h59m59.049832443s                                       
2019/08/24 13:35:38 Using session ****************L36N, expires in 3h59m59.049762979s                                       
2019/08/24 13:35:38 Assuming role arn:aws:iam::8XXXXXXXXX:role/BackupRDS from session token                               
aws-vault: error: Failed to get credentials for work (source profile for backup): AccessDenied: Access denied               
        status code: 403, request id: 968bc586-c695-11e9-8123-25259b551bda                         

MFA is enabled on my work account and must be present to assumeRole.

I tried with vault 4.5.1 and also tried to add mfa_serial under the work profile and nothing solve this issue for now. It looks like the session created doesn't contain the MFA token.
Also I don't really understand where this 403 comes from ? Sounds to be coming back from aws api, but the previous message about getting credentials seems to happen in the keyring so previous to any aws call no ?

[FernandoMiguel]

@sylwit if you put the mfa under the work role, doesn't it work?

[sylwit]

no

I tried with vault 4.5.1 and also tried to add mfa_serial under the work profile

[FernandoMiguel]

@sylwit does cli or Web work?
Just trying to exclude an actual misconfiguration

[sylwit]

@FernandoMiguel you were right, actually my role had only ecs-tasks.amazonaws.com as trusted entities instead of the account number.