Botocore: Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time

[kylegalbraith]

We are seeing a strange issue relating to boto3 and botocore. The following error is being thrown sporadically when we try to read from S3 or utilize an SQS client.

Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time

It appears that the credentials are not correctly getting refreshed via the assumed IAM role. This a Python application running inside of a Docker container within EKS. An example piece of code is below.

def fetch_message(s3, bucket, key):
    response = s3.get_object(Bucket=bucket, Key=key)

Does anybody have any ideas why this is happening and whether or not this is a known issue with boto?

[JordonPhillips]

It looks like you're sourcing credentials from the EC2 Instance Metadata and the request to fetch them failed. By default we don't retry those requests, but you can add retries with metadata_service_num_attempts and metadata_service_timeout in the config file (docs).

[kylegalbraith]

@JordonPhillips Thank you for the response, so would this be a matter of adding those environment variables into the container this is happening in?

[TattiQ]

In my case I get the same error but according to kube2iam logs the request for creds does not fail.

[TattiQ]

@JordonPhillips what if we are sourcing creds from a k8s pod which runs on a k8s worker node (ec2 instance) , so not directly on EC2 instance. do we set AWS_METADATA_SERVICE_NUM_ATTEMPTS as an env var to a pod? is it still legit then? thanks!

[shshe]

I'm also using kube2iam to have a pod assume an IAM role and seeing this error sporadically. It sometimes happens at the start of the container, but we've also seen it happen after the containers been running for a while. Any suggestions on workarounds? We've set AWS_METADATA_SERVICE_NUM_ATTEMPTS but it seems to have no effect.

[TattiQ]

@shshe what does botocore debug log level say ? also, do you use celery?

[shshe]

Hi @TattiQ , unfortunately, we didn't have DEBUG level turned on. I think our issue lies in kube2iam introducing latency when querying the EC2 metadata URI. Here's the related issue:

https://github.com/jtblin/kube2iam/issues/31

We're currently trying the workaround with setting AWS_METADATA_SERVICE_NUM_ATTEMPTS and increasing AWS_METADATA_SERVICE_TIMEOUT. So far, the issue hasn't cropped up yet.

Edit: Yes, we do use celery and saw this in our celery app. But, we've also seen this issue crop up in a Kubernetes job that used multiprocessing + boto.

[spinus]

Hey guys, did you figure out this issue by any chance?

[swetashre]

Following up on this issue. Solution provided by @JordonPhillips here should fix the issue. Is anyone still getting the error ? if yes, please reopen a new issue. I would be happy to help.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[chrisbrownnwyc]

We're having this issue specifically with K8S as well. Did setting the Environment variables work?

[kylegalbraith]

We're having this issue specifically with K8S as well. Did setting the Environment variables work?

Yes, setting the environment variables (specifically the retry attempts) seems to have mostly resolved the issue for us. We are also in an EKS K8s environment.

[dingobar]

Hi, I was facing this issue running python in a pod in an EKS cluster, and it seems at first glance the retries/timeout solution worked. Did anyone figure out a reason why these requests fail? I've seen pods restart hundreds of times because of this and I'm curious if there is something in the EKS setup that can be used to mitigate.