Botocore: Raise NoCredentialsError when no credentials are found

[tehtechguy]

This topic has come up before, see issues #29 and #1403. I'm re-raising this, with some details below. I have also created a PR to fix this #1752.

Certain credential providers, such as the InstanceMetadataProvider have the chance to fail to load credentials on start, even when the credentials exist and valid. Typically, such failures would get handled by the RefreshableCredentials class; however, that refresh only occurs if the provider was able to find credentials (thereby being the chosen credential provider).

The above situation results in a scenario where it's possible to create a session that does not have credentials. This will result in the client raising a NoCredentialsError upon the first usage of the credentials. That error will not have the ability to auto-resolve.

Having systematic credential failures like this are critical to be found upon session creation, rather than client use, as it enables services utilizing the session to know with certainty whether the session is usable or not.

To expand on this further:

• The current implementation is different from the Java SDK. In Java an exception is thrown when no credentials are found in the chain. See this.
• This has the potential for a web service to start, thinking it is healthy, pass a health check, and yet be unable to access any AWS services. A potential workaround is to call session.get_credentials() and verify that it is not None; however, this places an unnecessary burden on the user, since that workaround would need to be implemented by all endpoints.
• Similar to the above - any background workers will also have the same issue. That problem becomes further complicated, because there could be multiple calls, by the same application, to the instance metadata service where some calls passed and others didn't.
• Imagine a containerized implementation, such as Kubernetes, where there is the potential to have mutiple containers running on the same node, where some are able to reach the instance store and others are not. This is worsened when there any many containers running on one node (think lots of lightweight services), especially upon initial deploy, where the thundering herd problem may become an issue.
• For anyone expecting a provider that will refresh credentials, this will never occur, because no instance of RefreshableCredentials will be created (the credentials will be None).

[tehtechguy]

PR updated to reflect the current state of the develop branch + all broken tests fixed.

[ChristophShyper]

It would be nice to have this fixed some day. We hit NoCredentialsError almost every single day.
Hope AWS will have a look at your PR and make something good with it.

[chris-erickson]

Can someone shed some more context on how this happens? We have sporadic NoCredentialsError with a service running on EKS that uses an IAM Role to access Dynamo. It's super annoying and impossible to predict or manage. Is there some awful workaround that people are using (manually authenticating? changing retry env var settings for the metadata service?)? This seems shockingly catastrophic so I'm hoping I just don't fully understand this issue yet and have maybe misdiagnosed or problem.

[ChristophShyper]

We are using retry and it works now. But it's shameful for AWS to have such breaking bug for so long that can be easily solved internally.

[chris-erickson]

What do you mean by using retry?

[ChristophShyper]

Usually one retry is enough, but you can set it to for example 3.
Run problematic query in a loop using try-catch for NoCredentialsError exception. Use 1 second sleep inside it to make sure AWS API set new credentials properly.
Fail only after all tries are exhausted.

Basically AWS fails so often so this is our way of running anything against it.

[chris-erickson]

Oof. That's pretty awful when you have dozens of DynamoDB calls in a single app when any one of them could fail ☹️

We've upped the retries and timeout (via env vars) when using the metadata service, so I'm hoping that also helps. Our other less bad option is to have a specific user for these instances that otherwise has the same roles as the nodes themselves would have and use the API key. Tragic regression in best practices, but not working at all is not a great option either 😅

[itajaja]

I am also getting this on EKS. I am (pretty consistently) able to reproduce it:

kubectl run --restart='Never' --rm -it btest --image=ombu/boto3:1.8 -- python -c "import time; time.sleep(0); import boto3; boto3.client('s3').download_file('foo', 'bar', 'baz')"

this basically runs an image with bot and tries to download a file. most of the times, I get botocore.exceptions.NoCredentialsError: Unable to locate credentials.

Now, if I slightly modify my script to wait a little before running the s3 command:

kubectl run --restart='Never' --rm -it btest --image=ombu/boto3:1.8 -- python -c "import time; time.sleep(10); import boto3; boto3.client('s3').download_file('foo', 'bar', 'baz')"

most of the times (but not always!) the call succeeds. so, it seems related to some sort of lazy loading of the credentials that are not present when the container starts, or something along these lines.

I am running on EKS, kubernetes version 1.14. also running calico (not sure if that makes a difference).

[itajaja]

I think I am finding some useful logs:

botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2020-01-23 03:39:24,361 botocore.utils [DEBUG] Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.

really weird that it's a timeout error. how short can that timeout be? Interestengly running this command seems to reproduce it:

k run --restart='Never' --rm -it btest --image=python -- wget http://169.254.169.254/latest/meta-data/iam/security-credentials/ -T 1

yields

failed: Connection timed out.
Retrying.

--2020-01-23 03:49:35--  (try: 2)  http://169.254.169.254/latest/meta-data/iam/security-credentials/
Connecting to 169.254.169.254:80... connected.
HTTP request sent, awaiting response... 200 OK

so, works at second try. I don't think this is an issue with boto proper, but boto could have a small retry mechanism there

[chris-erickson]

Interesting intel. This reminds me that most of our issues with Dynamo creds have gone away since introducing the timeout/retry env vars.

[bhtucker]

This is still an issue. I wonder if it's exaggerated on EKS, where the additional network hops may eat into an otherwise-reasonable timeout.

For anyone reading the above, the env vars are AWS_METADATA_SERVICE_TIMEOUT and AWS_METADATA_SERVICE_NUM_ATTEMPTS, and these settings _do_ apply to the case where IMDS returns a 200 with an empty body (ie, botocore does see that as worth retrying).

[itajaja]

that seems to help, thanks @bhtucker

[attilasimon72]

We have just worked around this bug by setting:

AWS_METADATA_SERVICE_TIMEOUT=65 (or any reasonably high value)
AWS_METADATA_SERVICE_NUM_ATTEMPTS=1

Due to #2304, this config:

AWS_METADATA_SERVICE_TIMEOUT=1
AWS_METADATA_SERVICE_NUM_ATTEMPTS=15

is practically equivalent with:

AWS_METADATA_SERVICE_TIMEOUT=1
AWS_METADATA_SERVICE_NUM_ATTEMPTS=1

(If the IMDS query runs on ReadTimeoutError then botocore ignores AWS_METADATA_SERVICE_NUM_ATTEMPTS and breaks the retry loop on the first try.)

If the IMDS query ends with timeout then botocore will cache the empty credential as if it was valid. Setting AWS_METADATA_SERVICE_TIMEOUT to a high value reduces the chance of getting/caching an empty cred.