Boto3: Get credentials from meta-data service

[grepory]

Hey!

In the Go and Java SDKs we have the ability to provide a credential chain. It would be extremely helpful to do so here as well. For some context:

coreos/elb-presence uses boto for ELB registration. We use elb-presence to register our services with their respective ELBs. The elb-presence processes are scheduled with CoreOS Fleet units. In order to use elb-presence, that means we have to have AWS credentials (that don't expire) in our Fleet units... which, in turn, means that the credentials are stored in clear text in Etcd2.

As you can imagine, this is less than ideal. I'd rather not have to rewrite coreos/elb-presence in Go, and I think that boto could benefit from being able to, for example, get its credentials from an IAM role, or the environment, or statically specified credentials -- at runtime.

[kyleknap]

So fortunately boto3 supports some of these format thanks to the fact that it shares the same library, botocore, as the AWS CLI. So boto3 shares some of the credential chains of the CLI. For example, you can set the access key and the secret access key via environment variables, actually the same environment variables the AWS CLI uses: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment

Does this help your use case? Or is there anything specific you would like to see added?

[grepory]

With IAM Instance Profiles, we can get temporary credentials associated with a particular role from the meta-data service that we can then use to query AWS APIs. These credentials expire after a certain amount of time, and so they need to be retrieved periodically (I think the Java and Go APIs track the expiration time and retrieve new credentials if the current set have expired).

That's the functionality I'm specifically looking for. This is a long-running service that makes a call to the ELB API at startup and at shutdown. I've given it static credentials for now, but securing those credentials is not particularly easy. I'd rather get credentials from the metadata service.

[jamesls]

Boto3 already supports this. We need to get proper documentation in place for this, but for now here's the list of credential providers we support:

• env vars
• ~/.aws/credentials
• ~/.aws/config
• ~/.boto
• IAM instance metadata

If you're seeing issues with the boto3 not handling instance metadata credentials properly, please let us know we'll by happy to look into it.

[grepory]

OH! Thank you! I tried to find it in the source, but it's my first exposure to Boto, and I haven't done any Python in years.

[rozhok]

Thanks guys!

If anyone like me will google how to get secret/access using Instance Profile Credentials Provider in AWS Python SDK, here is a snippet:

from botocore.credentials import InstanceMetadataProvider, InstanceMetadataFetcher

provider = InstanceMetadataProvider(iam_role_fetcher=InstanceMetadataFetcher(timeout=1000, num_attempts=2))
creds = provider.load()

access_key = creds.access_key
secret_key = creds.secret_key

[aventurella]

@rozhok thanks for that snippet! Perhaps this, or something like it should be noted in the docs under configuration?

[aventurella]

here: https://github.com/boto/boto3/blob/develop/docs/source/guide/configuration.rst

[aventurella]

I just did a quick test. @rozhok boto3 will automatically read those credentials without specifically fetching them, even while running in a container in an ECS cluster.

In my particular case, I failed to set the AWS_DEFAULT_REGION environment var as I was operating not in us-east-1. Once I did, all it took for me to use instance meta data creds was:

client = boto3.client("s3")
client.list_objects(Bucket="com.mybucket.things")

Just to reiterate again, to use the instance meta data (by assigning an IAM role to my instance at launch), I just needed to initialize boto3 as documented and it automatically read the creds in the meta data. In my case, I was operating outside of "us-east-1" so I also needed to be sure I set AWS_DEFAULT_REGION in some appropriate way:

• env AWS_DEFAULT_REGION=us-west-2 python my_command.py
• os.environ["AWS_DEFAULT_REGION"] = "us-west-2"
• os.environ["AWS_DEFAULT_REGION"] = <read from a configuration file>

So it would automatically work.

[Pankajc123]

Hi,

I have stored AWS credentials inside ~/.aws/credentials

I want to fetch access key and secret key in my python code, but I am not able to retrieve access key and secret key

Below is code sample

import boto3
import boto
from botocore.credentials import InstanceMetadataProvider, InstanceMetadataFetcher

s3 = boto3.resource('s3')

provider = InstanceMetadataProvider(iam_role_fetcher=InstanceMetadataFetcher(timeout=1000, num_attempts=2))
creds = provider.load()
access_key = creds.access_key
secret_key = creds.secret_key

print access_key
print secret_key

Can someone please help me to figure it out where I am going wrong?

I am new to python programming.

Thanks.

Best Regards
Pankaj

[zfanswer]

@Pankajc123

I think your code has confliction of the version of boto,

from botocore.utils import InstanceMetadataFetcher
from botocore.credentials import InstanceMetadataProvider
provider = InstanceMetadataProvider(iam_role_fetcher=InstanceMetadataFetcher(timeout=1000, num_attempts=2))
creds = provider.load()
print(creds.access_key)
print(creds.secret_key)

I've tested.
You have to run it on a EC2 instance which has attached a IAM role.