Aws-cli: "Unable to locate credentials" Error Thrown Sporadically for Recursive, Cross-Account S3 Copying

[kjdowney]

After attending the "AWS CLI: 2017 and Beyond" session at re:Invent 2017, I was excited about utilizing the credential_source=Ec2InstanceMetadata option in the AWS-CLI configuration file along with role_arn values for performing cross-account S3 copies using roles.

Prior to this, a stub configuration file was being used that would be populated with the temporary credentials from the instance role and accessed using the source_profile option. I was eager to drop this logic in favor of using a built-in capability.

Upgrading to 1.14.x and using the credential_source version of the configuration file started sporadically logging upload failed: <file> Unable to locate credentials errors when performing a recursive copy to from the EC2 instance filesystem to S3 buckets hosted in a different account.

After some investigation and testing, the following discoveries were made:

• The issue only occurs when performing a cross-account copy.
• The issue isn't readily reproducible, leading to the suspicion that it's a race condition somewhere.
  
  
  
  • It's run without problems for a week at a time, then start generating errors on every attempt.
  
  
• The issue causes the same Unable to locate credentials errors for the legacy logic that was populating the stub configuration file, which has never been seen prior to upgrades.
• The issue occurs on any version >= 1.12.0, meaning that the CLI can't be updated beyond version 1.11.190.

Has anyone else seen anything like this?

[joguSD]

Thanks for reporting this. We have heard similar reports and are currently investigating why this would occur. We have a lead on what the race condition might be and are currently working on a PR.

[kyleknap]

Marking this as a bug. While I have not reproduced it in production using the set up that you described, we have received other reports related to the exact same thing that you are experiencing. I was however to reproduce it in a stress test and track it down to a particular race condition. I submitted the following PR: https://github.com/boto/botocore/pull/1405 to fix it. This should hopefully clear up this problem for you.

[kjdowney]

I ran a stress test for a couple of days to after upgrading to a botocore to a version with the fix and the latest AWS-CLI without encountering this issue. I've since configured workflows to use the updated versions and seen no problems thus far.

Thanks for the help!

[kyleknap]

Awesome. Glad to hear! Closing the issue.

[ciarancourtney]

I'm still seeing this sporadically with botocore 1.10.6 and boto3 1.7.6

NoCredentialsError is raised at botocore/auth.py in add_auth, line 739

https://github.com/boto/boto3/issues/1408#issuecomment-385615015