Aws-sdk-ruby: Aws::Errors::MissingCredentialsError: unable to sign request without credentials set

Created on 23 Sep 2016 · 15Comments · Source: aws/aws-sdk-ruby

I use IAM-role to assign policy to my EC2 instance, but sometimes it raise missing credential error.

Aws::Errors::MissingCredentialsError: unable to sign request without credentials set

Does anyone have same problem?

response-requested

Source

rayway30419

👍9

Most helpful comment

@rayway30419 Are you still experiencing the intermittent missing credential errors? Commonly this is caused by the aggressive defaults in attempting to load credentials from the instance metadata service. Once credentials expire, they must be refreshed, but if the metadata service fails to respond, a missing credentials error is raised. I've just push a small change that will go out with the next release where you can configure expanded retries and timeouts:

client = Aws::DynamoDB::Client.new(
  region: Aws.config[:region],
  instance_profile_credentials_timeout: 5, # defaults to 1 second
  instance_profile_credentials_retries: 5, # defaults to 0 retries
)

This change will be available in the next release.

trevorrowe on 17 Nov 2016

🎉7

All 15 comments

So you're saying that on an EC2 instance, you're seeing this periodically? How is your environment set up?

awood45 on 27 Sep 2016

Hi @awood45

It happens randomly,

What kind of enviroment setting?

Ruby 2.3.0
Rails 5.0.0.1
aws-sdk 2.6.3
Ubuntu 14.04 Trusty (3.13.0-63-generic)

rayway30419 on 29 Sep 2016

Is the Ubuntu version you listed your development environment, or an EC2 AMI? How are you configuring your clients in code?

awood45 on 29 Sep 2016

It's on the EC2 AMI.

# region in configuration file
client = Aws::DynamoDB::Client.new(
  region: Aws.config[:region]
)
# do sth.
client.put_item params

rayway30419 on 30 Sep 2016

BTW, I found a clue.....it's running on my sidekiq worker, it may happen to mult-ithreading?

rayway30419 on 30 Sep 2016

Another error message when initialize DynamoDB client
NameError: uninitialized constant Aws::CredentialProviderChain::InstanceProfileCredentials

rayway30419 on 30 Sep 2016

Okay, it looks like this may be related to how Sidekiq initializes - you might need to do eager_autoload within your Sidekiq workers, rather than only in the Rails initialization. Can you try that?

awood45 on 18 Oct 2016

For what it's worth, we're working on an overhaul of the SDK which should reduce or eliminate these autoload issues - message definitely received that they cause problems when threading.

awood45 on 18 Oct 2016

👍1

@awood45
With my understand, the sidekiq process will execute the procedures within rails initializer, too.

I am glad to hear this news.

rayway30419 on 18 Oct 2016

https://bugs.ruby-lang.org/issues/12688

glennpratt on 19 Oct 2016

Just chiming in here. The preview release of the modularized SDK is coming shortly. With modularization, we have eliminated the use of autoload.

trevorrowe on 17 Nov 2016

client = Aws::DynamoDB::Client.new(
  region: Aws.config[:region],
  instance_profile_credentials_timeout: 5, # defaults to 1 second
  instance_profile_credentials_retries: 5, # defaults to 0 retries
)

This change will be available in the next release.

trevorrowe on 17 Nov 2016

🎉7

hi @trevorrowe,
It still happens, I am looking forward to it!

rayway30419 on 17 Nov 2016

It looks like this went out with a previous release. Let me know/reopen if you see this again after looking at configured retries/timeouts. Thanks!

awood45 on 15 Dec 2016

@trevorrowe I'm still seeing this issue on (actually coming from fluentd which uses the SDK):

Ruby 2.3.4
AWS SDK 2.7.0
Linux Kernel 4.9.41

Is this expected? Could this be eventual consistency, instead of code?