Aws-sdk-ruby: [dynamodb] occasionally undefined credentials in EndpointCache

[grncdr]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [x] I've gone though Developer Guide for v3 and API reference

• [x] I've checked AWS Forums and StackOverflow for answers

• [x] I've searched for previous similar issues and didn't find any solution

Describe the bug

When querying DynamoDB we see irregular (but frequent) errors with the following stack trace:

NoMethodError: undefined method `credentials' for nil:NilClass
  from aws-sdk-core/endpoint_cache.rb:112:in `extract_key'
  from aws-sdk-core/plugins/endpoint_discovery.rb:104:in `_discover_endpoint'
  from aws-sdk-core/plugins/endpoint_discovery.rb:71:in `call'
  from seahorse/client/plugins/endpoint.rb:47:in `call'
  from aws-sdk-core/plugins/param_validator.rb:26:in `call'
  from seahorse/client/plugins/raise_response_errors.rb:16:in `call'
  from aws-sdk-dynamodb/plugins/simple_attributes.rb:119:in `call'
  from aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'
  from aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
  from aws-sdk-core/plugins/param_converter.rb:26:in `call'
  from seahorse/client/plugins/request_callback.rb:71:in `call'
  from aws-sdk-core/plugins/response_paging.rb:12:in `call'
  from seahorse/client/plugins/response_target.rb:24:in `call'
  from seahorse/client/request.rb:72:in `send_request'
  from aws-sdk-dynamodb/client.rb:4529:in `query'

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

• aws-sdk-core (3.114.0)
• aws-sdk-dynamodb (1.60.0)

Version of Ruby, OS environment
Ruby 2.6 and 2.7, Ubuntu and OS X Big Sur

To Reproduce (observed behavior)

I can reproduce it very reliably by hitting endpoints that call #query repeatedly. E.g. with code like this, I get partial results on every request and my exception tracker fills up quickly:

widgets = load_widgets(params)
foobars = widgets.map do |w|
  response = dynamo_singleton.query(...)
  response[:items]
rescue => e
  report_exception(e)
  []
end

Expected behavior
Every query succeeds

Additional context

Issue #2210 and this forum post indicate this might be due to failures to query the metadata service too frequently.

We use task roles in ECS to allow access to Dynamo and a singleton client that is constructed with no arguments. Is there anything further that can be done to prevent this error?

[mullermp]

Is "dynamo_singleton" working as expected? It seems that you are using ECS/EC2 to fetch your credentials, and if this happens enough times rapidly, IMDS will throttle requests and the client will not have any credentials set.

[grncdr]

I thought it was, added some logging code, and determined that it wasn't actually being used 🤦

Thanks for the pointer to look closer!

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.