Machine: Use of ~/.aws/credentials for AWS driver

[egut]

Hi

TL;DR Add --amazonec2-profile and read ~/.aws/credentials

• To secure command line to not include aws keys and secret key it should be possible not have these keys in commands.
• To enable (git) check in of commands without aws keys and secret key for continues integration/continues deploy.
• To be more user friendly and less error pron by enabling the user to specify a user friendly name (the profile) and if keys are updated you only need to change them in one place.
• I have several AWS profiles for different accounts and different environments its a bit of a show-stopper if I can't select profiles.

Implementation (would do it my self if I did know Go...)

Make --amazonec2-access-key and --amazonec2-secret-key optional.

If --amazonec2-profile is set or ( --amazonec2-access-key and --amazonec2-secret-key) is not set.
Read ~/.aws/credentials (ini-fileformat)
if --amazonec2-profile is set look up that profile and read those credentials.
else use credentials from default profile.

Environment variable to also look up:
AWS_DEFAULT_PROFILE would set default value for --amazoneec2-profile

More info about AWS profiles: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

/BR

Erik

[rhunter]

Docker Machine currently uses its own code to read credentials (from the command-line or environment variables) and then passes them to Go AWS Auth.

Go AWS Auth can look for credentials itself but currently uses different logic to the official AWS tools (including aws-cli and the new official Go SDK for AWS).

I agree it would be nice for Docker Machine to follow the Amazon-standard ways to find credentials (including profile and IAM support).

That could be done in "Go AWS Auth", or even migrating to the official SDK (now that it exists). I'd rather avoid putting the specific logic of reading credentials files and reading IAM roles and all that into Docker Machine's source code directly.

[nathanleclaire]

I'd be wary of this for the simple fact that it isn't precedented (i.e. users wouldn't know to expect it as it isn't consistent with other drivers), but not 100% opposed.

[ekristen]

Perhaps just a flag to use the ~/.aws/credentials file or the instance profile to meet in the middle? +1 to the idea though in general.

[moselaymd]

+1

[metcalfc]

The best practices from AWS are outlined here: http://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs

I think its fairly similar to how its being done now actually. Only one flag would need to be added: --amazonec2-profile with a default of default

The search would look like:

1. CLI flags --amazonec2-secret-key and --amazonec2-access-key override all
2. ENV AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID
3. Look in ~/.aws/credentials specified by --amazonec2-profile

Its slightly more complex than that because you can also use a session token. But again CLI -> ENV -> File.

[ocxo]

For someone new to the EC2 driver in docker-machine, the lack of this AWS best practice caught me off guard. I assumed it would read from ~/.aws/credentials.

[DrPaulBrewer]

I would also prefer it read from ~/.aws/credentials but you could also use a bit of shell scripting to extract the credentials file into env vars

[jeanlaurent]

Closing since this is done in #2839

[davclark]

No chance then of getting that --amazonec2-profile option, then? It is common practice to have different profiles in that credentials file for different projects (or even different accounts).

[nathanleclaire]

@davclark We'd be happy to review and hopefully accept PR for that but isn't currently planned for active development.

[rhunter]

@davclark The AWS CLI apps respect the AWS_DEFAULT_PROFILE environment
variable, and I expect this to be true for any app that uses the official
AWS SDK (now including Docker Machine, I believe).

[davclark]

Not sure why, but docker isn't picking up AWS_DEFAULT_PROFILE via the following very simple script:

AWS_DEFAULT_PROFILE=some_profile
docker-machine create -d amazonec2 some-id

The relevant line is here, and does indeed seem to be calling into the official AWS credentials (the AWS call is nested two function calls in).

(FWIW, I installed docker machine via brew install --HEAD docker-machine.)

For now, this seems to be a reasonable solution (slightly annoying, but less work than learning Go and submitting a pull request):

AWS_ACCESS_KEY_ID=$(aws --profile SHOzemi configure get aws_access_key_id)
AWS_SECRET_ACCESS_KEY=$(aws --profile SHOzemi configure get aws_secret_access_key) 

docker-machine create -d amazonec2 \
    --amazonec2-access-key $AWS_ACCESS_KEY_ID \
    --amazonec2-secret-key $AWS_SECRET_ACCESS_KEY \
    --amazonec2-region ap-northeast-1 \
    sz-tokyo

But that doesn't work either - both credentials and region are ignored in favor of the default from my ~/.aws directory. Config flags are documented here under "Options".
I also use ~/.aws/config anyway, to specify regions and such. So I think I need to go manual at this point anyway, even if the credential loading issue is resolved for me.

[nathanleclaire]

@davclark Does AWS_PROFILE (instead of AWS_DEFAULT_PROFILE) work? https://github.com/aws/aws-sdk-go/blob/c59d0ced7590a7e8df83b5871eb6dfbb13940af2/aws/credentials/shared_credentials_provider.go#L37

[davclark]

Good catch @nathanleclaire! I am back to using docker-machine 0.6.0 for now until I get that working. But I'll try to remember to update here once I get a chance to check your suggestion.

[donaldjmorton]

Hello, I just wanted to respond to @nathanleclaire 's comment about AWS_PROFILE rather than AWS_DEFAULT_PROFILE. Indeed it makes all the difference in the world. I was using AWS_DEFAULT_PROFILE, and it seemed to be grabbing my security group from the right account, but it was launching my instance on the wrong account. So glad I scrolled to the end and caught this. Thank you!