Aws-sdk-ruby: bucket.objects[object].exists? returns AWS::S3::Errors::Forbidden when object doesn't exist

[cwholt]

should return false.

[trevorrowe]

Can you give an example of when S3 returns the Forbidden response? I am able to reproduce this only when I am calling #exists? against an object in a bucket that I do not own, otherwise, it always returns no such key.

[cwholt]

@trevorrowe -

initialize bucket

irb(main):019:0> bucket = AWS::S3.new(:access_key_id => "key", :secret_access_key => "secret").buckets["bucket"]
=> #<AWS::S3::Bucket:bucket>

exists?

irb(main):020:0> bucket.objects["test.jsa"].exists?
AWS::S3::Errors::Forbidden: AWS::S3::Errors::Forbidden
    from /<redacted>/bundle/ruby/1.9.1/gems/aws-sdk-1.8.4/lib/aws/core/client.rb:339:in `return_or_raise'
    from /<redacted>/bundle/ruby/1.9.1/gems/aws-sdk-1.8.4/lib/aws/core/client.rb:440:in `client_request'
    from (eval):3:in `head_object'
    from /<redacted>/bundle/ruby/1.9.1/gems/aws-sdk-1.8.4/lib/aws/s3/s3_object.rb:294:in `head'
    from /<redacted>/bundle/ruby/1.9.1/gems/aws-sdk-1.8.4/lib/aws/s3/s3_object.rb:271:in `exists?'
    from (irb):20
    from /<redacted>/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:47:in `start'
    from /<redacted>/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/console.rb:8:in `start'
    from /<redacted>/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:41:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'
irb(main):021:0> bucket.objects["test.js"].exists?
=> true

This is on aws-sdk gem release 1.8.4, and it's likely worth noting I am using IAM, but cannot determine if there is a specific Action that I need to add to get this to work correctly. I have tried updating the IAM policy to allow for "s3:*" Actions, and even gone so far as using a "superuser" IAM policy that has access to all actions and resources, but the resulting behavior is still the same.

[trevorrowe]

If AWS::S3::Bucket#exists? returns true, that does not mean you have access to head objects in the bucket. Bucket#exists? works by calling GET bucket versioning against the bucket. The SDK will get one of the following three responses:

• 404 NoSuchBucket (bucket does not exist)
• 403 AccessDenied (bucket exists, but user does not have privileges to get the versioning state, the user may not own the bucket)
• 200 OK (bucket exists)

Calling AWS::S3::S3Object#exists? works by calling HEAD object. This can generate the following responses:

• 404 NoSuchKey (object does not exist)
• 200 OK (object exists)
• 403 Forbidden (maybe ????)

In the case of 403 forbidden it is no longer possible to tell if the object exists or not, only that you do not have privileges to call HEAD. I we catch the 403 Forbidden and return false. In this case S3 doesn't want to fold its hand and indicate if there is something there or not, only that you do not have privileges to know.

Have you tried the same request using your account credentials instead of an IAM user? Also, does your S3 bucket have a special policy or ACL?

[cwholt]

The IAM policy does seem to have permissions:

irb(main):004:0> bucket
=> #<AWS::S3::Bucket:bucket>
irb(main):005:0> bucket.exists?
=> true
irb(main):006:0> bucket.objects["test.js"].exists?
=> true
irb(main):007:0> bucket.objects["test.jsa"].exists?
AWS::S3::Errors::Forbidden: AWS::S3::Errors::Forbidden

I have tested with the account credentials and everything works as expected (returns false when object does not exist).

What I have determined after a lot more testing with IAM is that the API is returning 403 Forbidden on my request when the Resource is explicitly set to that of the bucket.

This works:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::*"
    }
  ]
}

This does not:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::bucket",
        "arn:aws:s3:::bucket/*"
      ]
    }
  ]
}

So it's likely this is not an issue with the Ruby AWS SDK, but a problem w/ IAM. It's also possible I just need another resource statement that I don't know about nor can find documented anywhere.

For now I will just monkeypatch this in my own codebase, but would like to know if you have any insight into the IAM issue.

Note: in all cases, bucket is replaced with my actual bucket name.

[trevorrowe]

That is interesting. I would post a question on the IAM forums (https://forums.aws.amazon.com/forum.jspa?forumID=76) and also the S3 forums (https://forums.aws.amazon.com/forum.jspa?forumID=24). I would expect HEAD object to behave the same with both of those policies.

Back to the original issue. While we can modify #exists? to return false in the case of a 403 response, I am afraid it could mask other issues (like poorly crafted policies, invalid bucket name, etc).

[cwholt]

That's fine, I think if the IAM issue got solved, this would also be fixed.

As I said, I'll monkeypatch until the cause of the 403 is determined.

[shipstar]

@cwholt I just came across this issue. As far as I can tell, specifying both resources in the IAM policy works now. The following works as expected.

{
"Version": "2012-10-17",
"Id": "MyID",
"Statement": [
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "AWS": "MyIAMUser"
        },
        "Action": "s3:*",
        "Resource": [
            "arn:aws:s3:::bucket/*",
            "arn:aws:s3:::bucket"
        ]
    }
]
}

Just FYI!

[jpwynn]

the fyi from @shipstr worked very well for us as well - thanks!

[dmagliola]

Another "vote" for @shipstar's comment, for anyone reading this, that's definitely the problem.

[frankwallis]

works for me too, thanks @shipstar!

[vickeryj]

I had this problem when I specified an s3 path in my IAM policy (arn:aws:s3:::bucket/path/*). Changing the policy per @shipstar's suggestion to grant access to the entire bucket fixed this. What's a good way to bring this up with the IAM people?

[kgf]

Does anyone know what permissions are actually needed ? Would prefer to not just wildcard...
What I currently have, but not enough yet.

            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-us-east-1"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-us-east-1/*"
            ]
        },

[kgf]

Found this link, testing now
https://groups.google.com/forum/#!topic/fluentd/Vvok7FTVuq4

[coreystinson2]

tl;dr Adding s3:ListBucket instead of s3:* worked for me.

So this had been an ongoing issue with me for quite some time.

I have a library responsible for uploading new objects to s3, you can see the code below. I generate bucket keys at the time of upload however first do a quick existence check to prevent any accidental overrides if we somehow generate the same hex

key = "#{SecureRandom.uuid}#{extension}"
existing = Aws::S3::Object.new(bucket, key)
if existing.exists?
  raise ExistsError, 'Please try uploading the file again.'
end

Now developing locally I use the following IAM permissions, I like keeping these down to the bare minimum for both my development and production policies.

"Action": [
    "s3:GetObject",
    "s3:PutObject"
],
"Resource": [
    "arn:aws:s3:::my-bucket/*",
    "arn:aws:s3:::my-bucket"
]

The above policy works great, the upload library executes without problems and returns both true and false based on whether or not the file actually exists.

Now we push this code to production on an instance registered to ECS and we start experiencing some weird behaviour. This exact same policy now starts throwing the following error
/usr/local/bundle/gems/aws-sdk-resources-2.5.3/lib/aws-sdk-resources/resource.rb:134:in rescue in exists?'

While the aforementioned fixes such as changing the policy actions to below, do in fact work - I'm super anal about giving my applications excessive permissions and think it's a great practise to ensure they only have what they need to have.

"Action": [
    "s3:*"
],
"Resource": [
    "arn:aws:s3:::my-bucket/*",
    "arn:aws:s3:::my-bucket"
]

Anyway I spent ages going one by one through all the policies until I found the combination that worked. s3:ListBucket.

"Action": [
    "s3:GetObject",
    "s3:ListBucket",
    "s3:PutObject"
],
"Resource": [
    "arn:aws:s3:::my-bucket/*",
    "arn:aws:s3:::my-bucket"
]

Still havn't figured out the difference between development and production in regards to exactly why production requires this new action. However I'm much happier using this as oppose to just straight up s3:*

[AlexMorreale]

Just ran into this myself. It's pretty crazy .exists? requires GetObject and not just ListBucket

[jacksonrayhamilton]

I ran into this too, but it was not even a real permission problem, my server’s clock was off. Using ntp fixed it for me: https://stackoverflow.com/a/49260424/1468130 Leaving this here in case it helps anyone else.

[pkoch]

Supposedly, just s3:ListObjects and s3:GetObject is sufficient. Source: https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/