Aws-sdk-ruby: aws-sigv4 v1.1.2 breaks clients making use of STS credentials

Created on 19 Apr 2020  路  9Comments  路  Source: aws/aws-sdk-ruby

Describe the bug
After upgrading aws-sigv4 to version 1.1.2 any API client making use of STS credentials fail

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version
aws-eventstream (1.1.0)
aws-partitions (1.298.0)
aws-sdk-core (3.94.0)
aws-sdk-kms (1.30.0)
aws-sdk-s3 (1.61.2)
aws-sigv4 (1.1.2)

Version of Ruby, OS environment
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]

To Reproduce (observed behavior)
code example

gem 'aws-sigv4', '=1.1.2'
require 'aws-sigv4'
require 'aws-sdk-core'
require 'aws-sdk-s3'

sts = Aws::STS::Client.new
role_arn = 'arn:aws:iam::012345678901:role/my-awesome-role'
external_id = 'ExternalId'
role_session_name = 'bug_test'
credentials = sts.assume_role({
  role_arn: role_arn,
  external_id: external_id,
  role_session_name: role_session_name,
  duration_seconds: 3600,
  policy: "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":"\
    "\"Stmt1\",\"Effect\":\"Allow\",\"Action\":\"s3:*\","\
    "\"Resource\":\"*\"}]}"
})

s3 = Aws::S3::Client.new(credentials: credentials)
bucket = 'my-awesome-bucket-q1rktc0c0q'
key = '0374AA001-004.txt'
if Aws::S3::Object.new(client: s3, bucket_name: bucket, key: key).exists?
  puts "The object does exist"
end

Expected behavior
The S3 API client should be able to use the STS credentials

Screenshots
script output

$ ruby ./test_bug.rb
Traceback (most recent call last):
        46: from /home/deme/src/test_bug.rb:23:in `<main>'
        45: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/object.rb:358:in `exists?'
        44: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/object.rb:377:in `wait_until_exists'
        43: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/waiters.rb:198:in `wait'
        42: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:90:in `wait'
        41: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:90:in `catch'
        40: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:91:in `block in wait'
        39: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:91:in `catch'
        38: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:92:in `block (2 levels) in wait'
        37: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:102:in `poll'
        36: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:102:in `loop'
        35: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/waiter.rb:105:in `block in poll'
        34: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/poller.rb:49:in `call'
        33: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/waiters/poller.rb:63:in `send_request'
        32: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/request.rb:70:in `send_request'
        31: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
        30: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
        29: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
        28: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
        27: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
        26: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/accelerate.rb:35:in `call'
        25: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/dualstack.rb:26:in `call'
        24: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/sse_cpk.rb:22:in `call'
        23: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
        22: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
        21: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:78:in `call'
        20: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:28:in `call'
        19: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/user_agent.rb:11:in `call'
        18: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/rest/handler.rb:8:in `call'
        17: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/bucket_arn.rb:47:in `call'
        16: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/bucket_dns.rb:33:in `call'
        15: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/expect_100_continue.rb:22:in `call'
        14: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/bucket_name_restrictions.rb:24:in `call'
        13: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/md5s.rb:32:in `call'
        12: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/accelerate.rb:50:in `call'
        11: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/dualstack.rb:34:in `call'
        10: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/retry_errors.rb:350:in `call'
         9: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/redirects.rb:18:in `call'
         8: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/s3_signer.rb:83:in `call'
         7: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
         6: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:24:in `call'
         5: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
         4: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/s3_host_id.rb:15:in `call'
         3: from /var/lib/gems/2.5.0/gems/aws-sdk-s3-1.61.2/lib/aws-sdk-s3/plugins/s3_signer.rb:53:in `call'
         2: from /var/lib/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/signature_v4.rb:123:in `apply_signature'
         1: from /var/lib/gems/2.5.0/gems/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:205:in `sign_request'
/var/lib/gems/2.5.0/gems/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:657:in `fetch_credentials': undefined method `set?' for #<Aws::STS::Types::Credentials:0x00007fffeddf6fc0> (NoMethodError)

Additional context
The code above works with aws-sigv4 v1.1.1

bug pwork-in-progress
Source
picture DemeAtRBM
👍1

Most helpful comment

We may also be seeing a manifestation of this issue. We're not using STS directly but using role based credentials (which I believe uses STS under the hood). Since updating to 1.1.2, we've started seeing sporadic Aws::Sigv4::Errors::MissingCredentialsError when communicating with Elastic Search.

In our case, we're creating the client using Aws::InstanceProfileCredentials.new a la:

instance_profile = Aws::InstanceProfileCredentials.new
: 
: 
credentials: Aws::Credentials.new(instance_profile.credentials.access_key_id, instance_profile.credentials.secret_access_key, instance_profile.credentials.session_token),

We see millions of requests but only around 30 or in a million are complaining about the missing credentials.

Thinking perhaps it's the same issue with a slightly different symptom.

picture dnorth98 on 22 Apr 2020
👍2

All 9 comments

Thanks for submitting this - a change I made recently (#2288) broke this. I'm working on a fix.

In the mean time, the recommended way to use STS assume role credentials in your clients is to use the AssumeRoleCredentials credentials provider - this wraps the call to sts::assume_role and handles refreshing for you.

alextwoods picture alextwoods on 20 Apr 2020

We may also be seeing a manifestation of this issue. We're not using STS directly but using role based credentials (which I believe uses STS under the hood). Since updating to 1.1.2, we've started seeing sporadic Aws::Sigv4::Errors::MissingCredentialsError when communicating with Elastic Search.

In our case, we're creating the client using Aws::InstanceProfileCredentials.new a la:

instance_profile = Aws::InstanceProfileCredentials.new
: 
: 
credentials: Aws::Credentials.new(instance_profile.credentials.access_key_id, instance_profile.credentials.secret_access_key, instance_profile.credentials.session_token),

We see millions of requests but only around 30 or in a million are complaining about the missing credentials.

Thinking perhaps it's the same issue with a slightly different symptom.

dnorth98 picture dnorth98 on 22 Apr 2020
👍2

@dnorth98 FYI - You should be able to pass Aws::InstanceProfileCredentials.new into credentials: directly rather than creating Aws::Credentials.new(...)

mullermp picture mullermp on 22 Apr 2020
👍1

Just adding onto @mullermp 's comment - My guess on what they are seeing is that the credentials are failing when they have expired. If you use the InstanceProfileCredentials directly, they will handle refreshing before the expiration for you.

alextwoods picture alextwoods on 22 Apr 2020
👍1

I'm running into this as well in conjunction with rds-iam authentication...

"message": "Failure: NoMethodError [undefined method `set?' for #<Aws::STS::Types::Credentials:0x00005561082cd760>]",
 "backtrace": [
        "/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:657:in `fetch_credentials'",
        "/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:370:in `presign_url'",
        "/var/task/vendor/bundle/ruby/2.5.0/gems/aws-sdk-rds-1.83.0/lib/aws-sdk-rds/customizations/auth_token_generator.rb:54:in `auth_token'",

When will the fix be released?

phene picture phene on 24 Apr 2020
👍1

Running into the same issue.

pchaganti picture pchaganti on 25 Apr 2020 
ruby 2.6.2p47 (2019-03-13 revision 67232) [x86_64-darwin19]

undefined method `set?' for #<Aws::STS::Types::Credentials:0x00007f875535a5c8>:

/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:657:in `fetch_credentials'
/aws-sigv4-1.1.2/lib/aws-sigv4/signer.rb:205:in `sign_request'
/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/signature_v4.rb:123:in `apply_signature'
pchaganti picture pchaganti on 27 Apr 2020

I pushed #2291 to fix this last week but forgot to add a changelog entry. I've added that now and an update to the aws-sigv4 gem should be released today that should fix this issue.

alextwoods picture alextwoods on 27 Apr 2020

Tried today's release. The fix worked for us.

thanks!

pchaganti picture pchaganti on 27 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

allcentury picture allcentury  路  4Comments
thegreyfellow picture thegreyfellow  路  3Comments
blondowski picture blondowski  路  3Comments
broksonic21 picture broksonic21  路  3Comments
NomNomCameron picture NomNomCameron  路  4Comments