Aws-sdk-ruby: AWS::S3::Errors::BadRequest when calling S3Object#exists?

[gjastrab]

Using 1.32.0, ruby 1.9.3-p484

Doesn't seem like calling exists? on an object should be raising an error like this?

Portion of the backtrace included:

• ".../shared/bundle/ruby/1.9.1/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:368:in return_or_raise'",
• ".../shared/bundle/ruby/1.9.1/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:469:in client_request'", "(eval):3:in `head_object'",
• ".../shared/bundle/ruby/1.9.1/gems/aws-sdk-1.32.0/lib/aws/s3/s3_object.rb:293:in head'",
• ".../shared/bundle/ruby/1.9.1/gems/aws-sdk-1.32.0/lib/aws/s3/s3_object.rb:270:in exists?'"

[trevorrowe]

Do you have any additional information to share about the failed request? Is there anything unique about the bucket name, and could you share that? I am unable to currently reproduce the issue.

[gjastrab]

Yes, sorry I meant to put more info in after I had submitted it.

Nothing unique about the bucket name aside from having 1 underscore in it. A customer is passing us a large amount of data to import into our system via S3, so we're running a process iterating over the bucket which has hundreds of thousands of objects.

The code to iterate through is essentially doing something like:

tree = bucket.as_tree(prefix: 'some/prefix/')
tree.children.each do |node|
  if bucket.objects[node.prefix + 'somefile'].exists?
    ...
  end
end

The first time I got this error the process had run for about 6 hours and the 2nd time it had run for 12 hours, so I wonder if there's some edge case that's causing it to occur only after doing many many thousands of requests.

Other relevant information is it's authenticating via an IAM role, so not passing any credentials when instantiating AWS::S3 just doing:

s3 = AWS::S3.new
bucket = s3.buckets['mybucket_name']

Sorry for the initial lack of info, hope this helps narrow it down.

[trevorrowe]

I miss-read and failed to notice that it was the #exists? method on the object, not bucket class raising the error. I should ask, is there anything unique about the object key in the failed examples? Also, do you know if it fails consistently on the same keys?

[gjastrab]

I do know on the 2 instances when it failed the object did not exist, so it should have returned false. Also apologize for not mentioning that.

The format our customer is sending us data let's them optionally include a file at a predictable location within each "folder" so I need to do this exists? check to see if I can do that optional processing or not.

[gjastrab]

Hm, although tonight it just happened on an object that does actually exist in the bucket (after running for 5 hours). Since yesterday I wrapped the exists? call in a begin/rescue:

tree = bucket.as_tree(prefix: 'some/prefix/')
tree.children.each do |node|
  begin
    if bucket.objects[node.prefix + 'somefile'].exists?
      ...
    end
  rescue AWS::S3::Errors::BadRequest
  end
end

but there was another method I forgot about which also called exists? on an S3Object so the process failed again. Odd that this time when I looked in the logs the object it failed on is one that exists, so it should have returned true unlike the earlier times it raised...

[gjastrab]

This is turning into a deal breaker for us. Now it's happened when calling S3Object#content_length (the process had run for 5.5 hours when it encountered this):

AWS::S3::Errors::BadRequest
".../aws-sdk-1.32.0/lib/aws/core/client.rb:368:in return_or_raise'"
".../aws-sdk-1.32.0/lib/aws/core/client.rb:469:in client_request'"
"(eval):3:in head_object'",
".../aws-sdk-1.32.0/lib/aws/s3/s3_object.rb:293:in head'"
".../aws-sdk-1.32.0/lib/aws/s3/s3_object.rb:317:in content_length'"

The object it was being called on does exist and I was able to download it using the AWS CLI just fine.

This is getting to be quite a pain - I can't wrap every S3 interaction in a begin...rescue to prevent the script from bombing out. Have you been able to find a way to reproduce it? Something regarding it being run/authenticated via an IAM profile perhaps?

Let me know if there's something more I can gather on my end that would help you track down the issue. Thanks.

[trevorrowe]

We do not currently have any other related issues reported about this, but I agree the failures are discouraging. It does not suprise me that a call to #content_length also fails. Both #content_length and #exists? call the exact same AWS API, which is HEAD object. If one fails, the other is sure to fail.

Some questions that might help track down the issue:

• Are you using AWS account credentials, or credentials from an IAM user or policy?
• Do the requests fail consistently on the same keys, or is this intermittent? If you added a simple retry to the rescue block, will the follow request succeed?
• Is there anything special about the object keys that fail, like multibyte characters? This should not be an issue, but its worth investigating.

It may also be helpful to see the HTTP wire trace from a failed request. You can enable http wire tracing like so:

s3 = Aws::S3.new(http_wire_trace:true)

You can pass an optional Logger object as :logger to specify where to send the wire trace. It defaults to STDOUT. Also, this logs _everything_ about your http requests to S3, so the log may be large.

[gjastrab]

Will turn back on with the wire trace logging to hopefully get more specifics on the request/response for you to debug.

• I said earlier these are servers launched under an IAM role. That role has been given access to the bucket with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1280939304000",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::MY_BUCKET_NAME"
      ]
    },
    {
      "Sid": "Stmt1132939872000",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:ListMultipartUploadParts",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::MY_BUCKET_NAME/*"
      ]
    }
  ]
}

• I haven't tried a retry on the block, but I don't think it's a specific key issue. On the most recent occurrence of this crashing, I looked in the log for the key where the exists? call had failed. I opened up a new console on the same server and called exists? and content_length on that key and got back true and the appropriate content length.
• No nothing special about the object keys

[trevorrowe]

I pushed a commit a week ago related to the AWS::Core::CredentialProviders::EC2Provider related to how it detected expired credentials and rotated them, see: https://github.com/aws/aws-sdk-ruby/commit/a1c0197977560bc72d2eb0a876daa63648b2d985.

Could you try running your code against the master branch, or perhaps apply this monkey patch:

require 'aws-sdk'
class AWS::Core::CredentialProviders::EC2Provider
  def credentials
    if @credentials_expiration && @credentials_expiration.utc <= (Time.now.utc + (15 * 60))
      refresh
    end
    super
  end
end

[gjastrab]

Sure, but I had already restarted the processes with wire tracing enabled, so the next time it fails I'll send you the log info and then restart it with this monkey patch. Thanks.

[trevorrowe]

BTW, I just pushed v1.34.0 of the Ruby SDK with the EC2Provider fix. I suspect this version will resolve your issue, especially considering the 5 hour window lines up with the standard expiration of EC2 IAM profile credentials.

[gjastrab]

Excellent, although a couple of times it was after 12 hours.

[gjastrab]

a1c0197977 in v1.34.0 seems to have solved the problem, thanks! :thumbsup:

[s2t2]

FYI on gem version 2.0.41 a call to bucket.exists? throws rescue in exists?': Aws::S3::Errors::BadRequest only if the bucket exists. The bucket in question was created using bucket = resource.bucket(BUCKET_NAME). Please advise.

[trevorrowe]

@s2t2 Its not clear to me what you mean. Using v2.0.41 I tried the following:

# bucket I own, returns 200
Aws::S3::Bucket.new('aws-sdk').exists? #=> true

# bucket I do not own, returns 403, but still exists
Aws::S3::Bucket.new('demo').exists? #=> true

# non-existent bucket, returns 404, does not exist
Aws::S3::Bucket.new('asdfasdfasdfasdfasdfa1111').exists? #=> false

Can you give me a code example of what fails?

[s2t2]

Thanks for your fast reply. I was using the following code, which is now behaving differently than it was yesterday, in that yesterday it was throwing error when checking for existence of an existing bucket, whereas now its sometimes returning true and sometimes throwing an error when checking for existence of a non-existent bucket. And resource.create_bucket was working yesterday but now is attempting to create the bucket in the wrong region.

creds = Aws::Credentials.new(KEY_ID, KEY_SECRET)
resource = Aws::S3::Resource.new(:credentials => creds, :endpoint => ENDPOINT, :region => REGION)
bucket = resource.bucket(BUCKET_NAME)
resource.create_bucket(:bucket => BUCKET_NAME) unless bucket.exists? #=> Aws::S3::Errors::BadRequest

Using the code you suggested, I'm getting the following...

S3::Bucket.new(BUCKET_NAME)
=> NameError: uninitialized constant S3
Aws::S3::Bucket.new(BUCKET_NAME)
=>  Aws::Errors::MissingRegionError: missing region; use :region option or export region name to ENV['AWS_REGION']
bucket = Aws::S3::Bucket.new(BUCKET_NAME, :region => REGION)
=> #<Aws::S3::Bucket name="testt">
bucket.exists?
=> Aws::S3::Errors::Http301Error
bucket = Aws::S3::Bucket.new(BUCKET_NAME, :credentials => creds, :endpoint => ENDPOINT, :region => REGION)
=> #<Aws::S3::Bucket name="testt">
bucket.exists?
=> Aws::S3::Errors::BadRequest

I may just be using the gem wrong, so any guidance you can give is appreciated.

EDIT: I have found more luck using the code below, but that is throwing region-related errors which are not applicable to this issue.

resource = Aws::S3::Resource.new(:region => REGION, credentials: creds)
bucket_names = resource.buckets.map{|bucket| bucket.name}
resource.create_bucket(:bucket => BUCKET_NAME, :acl => "private") unless bucket_names.include?(BUCKET_NAME)
#> Aws::S3::Errors::IllegalLocationConstraintException: The us-west-1 location constraint is incompatible for the region specific endpoint this request was sent to.

[trevorrowe]

@s2t2 Sorry, I was running my examples from pry inside the Aws module with a globally configured region.

The Http301Error indicates that you have received a permanent redirect. Would you open this as a new issue. It looks like there are some bucket dns redirect issues to resolve.

[MicTech]

@trevorrowe

I have a same problem.

version 1.65.0

policy

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::bucketname",
                "arn:aws:s3:::bucketname/*"
            ]
        }
    ]
}

ruby

s3 = AWS::S3.new(:access_key_id => 'xxx', :secret_access_key => 'xxx')
bucket = s3.buckets['bucketname']

puts bucket.objects[source_key].acl
AWS::S3::Errors::NoSuchKey: No Such Key

puts bucket.objects[source_key].exists?
AWS::S3::Errors::BadRequest: AWS::S3::Errors::BadRequest

request

HEAD /directory/filename.ext HTTP/1.1
Content-Type: 
Accept-Encoding: 
User-Agent: aws-sdk-ruby/1.65.0 ruby/2.2.3 x86_64-darwin15
Date: Thu, 21 Apr 2016 13:51:44 GMT
Authorization: AWS xxx:xxx=
Accept: */*
Host: bucketname.s3.amazonaws.com

[awood45]

I think upgrading to version 2 of the SDK is the best path forward for that. We've dealt with these issues in V2 in a more thorough way. Plus, V1 is essentially deprecated, so I don't see non-critical work happening on it going forward.

[MicTech]

@awood45 That's correct and I agree with that, but unfortunately Paperclip doesn't support v2, at least not yet.

[awood45]

I'd encourage them to upgrade, if there are any blocking issues we're happy to look in to them.

[anoldguy]

I realize this is an old issue, but I found that being explicit about the region to fix the problem for us.

We were getting the AWS::S3::Errors::BadRequest for every call to .exist?. Our bucket was in us-east-2, and was somehow working without specifying the region. Once we passed region into the S3 client, all was well.

[pedroVinko]

I solved by changing my time zone (wich wasn't the correct one)