Vault: Failure To Create IAM Credential When Access Token Contains Slash

[RichardKnop]

Hello, I have a root AWS access key which contains a slash, something like:

vault write aws/config/root \
    access_key=REDA/CTED \
    secret_key=REDACTED \
    region=ap-southeast-1

I am following the tutorial and I created a role with policy to allow EC2 operations:

vault write aws/roles/deploy [email protected]

When I then try to read a credential for the IAM role, I get an error like this:

Richards-MacBook-Pro:vault richardknop$ vault read aws/creds/deploy
Error reading aws/creds/deploy: Error making API request.

URL: GET http://127.0.0.1:8200/v1/aws/creds/deploy
Code: 400. Errors:

* Error creating IAM user: IncompleteSignature: Credential must have exactly 5 slash-delimited elements, e.g. keyid/date/region/service/term, got 'REDA/CTED/20170117/us-east-1/iam/aws4_request'
    status code: 400, request id: f1b582d2-dc69-11e6-9cb5-6fc5af467f59

One of the problems I see is that the region in the output above is us-east-1 although it should be ap-southeast-1.

However this specific error is related to my root access key having a slash I believe.

[jefferai]

Hi @RichardKnop ,

This error is coming from the AWS API call, not from Vault.

You don't say which version of Vault you're using, but if it's not the most recent, you could try upgrading, as the AWS client library is updated regularly with each release and may work around this problem.

[RichardKnop]

Hi @jefferai ,

I am using latest development version. Build from source with make dev today.

Vault v0.6.4 ('f4adc7fa960ed8e828f94bc6785bcdbae8d1b263')

I was wondering whether the error is caused by Vault not escaping forward slash properly when passing the access key value to AWS client library.

[jefferai]

Doing a quick Google search I found a similar issue with Hadoop that was only just worked around on their end after 8 years(!): https://issues.apache.org/jira/browse/HADOOP-3733

It sounds like it could be worked around on our end but really seems like the library should do the right thing; I filed https://github.com/aws/aws-sdk-go/issues/1042 to track it.

[RichardKnop]

@jefferai I think I have found out the issue and it seems to be user error (sorry). I managed to get the part of tutorial explaining generating dynamic AWS credentials working.

Here was the problem. AWS has access key and secret variables which I use to connect to their API.

I have something like this in my ~/.bash_profile:

export AWS_SECRET_ACCESS_KEY=FOOBARFOOBARFOOBAR+g/HELLOWORLD
export AWS_ACCESS_KEY_ID=FOOOFOOOFOOOID

AWS allows the AWS_SECRET_ACCESS_KEY to contain a forward slash, that is allowed character so the secret can contain slashes. The AWS_ACCESS_KEY_ID seems to only allow letters.

When running this command:

vault write aws/config/root \
    access_key=FOOBARFOOBARFOOBAR+g/HELLOWORLD \
    secret_key=FOOOFOOOFOOOID \
    region=ap-southeast-1

I switched around the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID. I was using the access key as secret and vice versa.

So when I switch it around it worked:

vault write aws/config/root \
    access_key=FOOOFOOOFOOOID \
    secret_key=FOOBARFOOBARFOOBAR+g/HELLOWORLD \
    region=ap-southeast-1

Sorry for confusion. I think there is no problem with AWS library, this was user mistake.

[jefferai]

@RichardKnop No problem, glad that the AWS team pointed you in the right direction! :-D