Aws-sdk-java: S3 doesObjectExist throws 403 Forbidden

[jelbatnigi]

I am calling doesObjectExist(bucketName, object) to check if object exists or not in the bucket.
I get
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden
only when the object does not exist. it works when the object exist.

I am calling this from a lambda function. I do have role permission to add, put and delete objects. So I don't know if this is a role or permission issue.
Also, This works when I run it from my local where I have to specify my access secret keys when initializing my S3Client.

Any thoughts?

looking at a similar issue
https://github.com/aws/aws-sdk-java/issues/707

It was mentioned that account does not have permission to access object's metadata. How to give access to view object meta data?

Thanks

[spfink]

Which version of the SDK are you using? Can you provide a code sample?

Do you have Get* permissions for S3? Can you provide a (redacted) copy of your IAM policy?

[jelbatnigi]

I am using SDK 1.11.65

`public AmazonS3Client s3Client()
{

   ClientConfiguration config = new ClientConfiguration();
   config.setProxyHost(environment.getProperty("proxy.host"));
   config.setProxyPort(Integer.valueOf(environment.getProperty("proxy.port")));
   if (environment.getProperty("env").equals("dev")) {
       String accessKeyId = environment.getProperty("accessKeyId");
       String secretKey = environment.getProperty("secretKey");
       BasicAWSCredentials credentials = new BasicAWSCredentials(accessKeyId, secretKey);

       return new AmazonS3Client(credentials,config);
   }

   return new AmazonS3Client(config);

}
objectExists = s3Client.doesObjectExist(bucketName, objectName);

• As you can see I am only passing credentials to S3Client when I am using my local dev env.

I only get this error when

1. I call it from within AWS (e.g Lambda) where I am not passing my credential to my S3Client. but getObject, outOject and DeleteObject work fine.
2. also, When the file or key doesn't exist. if the file exist, doesObjectExist returns true and everything works fine.

So here are my scenarios:

1. I call other S3 actions such as geObject, putObject and deleteObject from lambda without passing credentials to S3Client and everything works with no errors.

2. I call doesObjectExist from lambda without passing credentials to S3Client and object doesn't exist - I get 403 error

3. I call doesObjectExist from Lambda with passing my credentials to S3Client and object doesn't exist - doesObjectExist returns false with No errors!

4. I call doesObjectExist from Lambda without passing my credentials and object exist - doesObjectExist returns true with no errors.

So only scenario #2 fails.

Here is my IAM policy
"PolicyName":"AppS3AccessPolicy", "PolicyDocument":{ "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Action":[ "s3:*" ], "Resource":[ { "Fn::Join":[ "/", [ { "Fn::Join":[ ":", [ "arn:aws:s3::", {"Fn::GetAtt":["VPCInfo","VPCname"]} ] ] }, { "Ref":"WmcDeployPkgPathKey" }, "*" ] ] } , { "Fn::Join":[ "/", [ { "Fn::Join":[ ":", [ "arn:aws:s3::", {"Fn::GetAtt":["VPCInfo","VPCname"]} ] ] }, { "Ref":"AWS::StackName" }, "*" ] ] }] }, { "Effect":"Allow", "Action":[ "s3:GetObject" ], "Resource":[ { "Fn::Join":[ "/", [ { "Fn::Join":[ ":", [ "arn:aws:s3::", {"Fn::GetAtt":["VPCInfo","VPCname"]} ] ] }, "EGUS/vault", "*" ] ] } ] }, { "Effect":"Allow", "Action":[ "s3:GetObject*", "s3:PutObject*", "s3:DeleteObject*"], "Resource":[ { "Fn::Join":[ "/", [ { "Fn::Join":[ ":", [ "arn:aws:s3::", {"Fn::GetAtt":["VPCInfo","VPCname"]} ] ] }, "EGUS", "*" ] ] } ]

[spfink]

Is the policy that you pasted the policy for the user/role you are using for your local dev environment of the policy attached to the Lambda function?

[jelbatnigi]

no this is for the one I am using for the test env not local.
The way I am specifying the resource in the role is:
"Resource":[ { "Fn::Join":[ "/", [ { "Fn::Join":[ ":", [ "arn:aws:s3::", {"Fn::GetAtt":["VPCInfo","VPCname"]} ] ] }, "EGUS", "*" ]
my bucket is my vpc name so technically it is
{"Effect":"Allow", "Action":[ "s3:GetObject*", "s3:PutObject*", "s3:DeleteObject*"], "Resource":"arn:aws:s3::vpcname/folder/*" }]

I got it to work only when I specify the resource as
{"Effect":"Allow", "Action":"s3:*", "Resource":"arn:aws:s3:::*" }]

If I specify the bucket and the folder it doesn't work
for example this doesn't work
{"Effect":"Allow", "Action":"s3:*", "Resource":"arn:aws:s3::bucket" }]
or
{"Effect":"Allow", "Action":"s3:*", "Resource":"arn:aws:s3::bucket/folder/*" }]
that causes a forbidden error.

[vitorreis]

Guys this is actually a feature, right?!

As you discovered, Amazon S3 will return an AccessDenied error when a nonexistent key is requested and the requester is not allowed to list the contents of the bucket. By the Amazon S3 definition, not being allowed to list the contents of a bucket means not being allowed to discover whether a particular key exists. Returning NoSuchKey would leak information about the nonexistence of the requested key. Instead, Amazon S3 returns AccessDenied. AccessDenied does not say anything about the existence or nonexistence of the requested key, so no information is leaked.

Amazon S3 will return a NoSuchKey error when a nonexistent key is requested and the requester is allowed to list the contents of the bucket.

[spfink]

Thanks @vitorreis for the link.

This is indeed a "feature" of S3 to prevent the leakage of whether or not certain keys exist.

@jelbatnigi, please trying adding s3:List* permissions to your IAM policy to see if that fixes the issues.

[spfink]

Closing this based on the previous comment. @jelbatnigi let me know if the above guidance doesn't actually work for you.

[rudradixit]

@jelbatnigi I encountered a similar issue. Originally, I had the following permission setup for my lambda:

{
      "Action": ["s3:*"],
      "Effect": "Allow",
      "Resource": ["${aws_s3_bucket.my_custom_bucket.arn}/*"]
}

However, when I added the bucket itself to the Resource in addition to anything inside it, it worked without issues.

{
      "Action": ["s3:*"],
      "Effect": "Allow",
      "Resource": ["${aws_s3_bucket.my_custom_bucket.arn}", "${aws_s3_bucket.my_custom_bucket.arn}/*"]
}

Note: I'm using Terraform to deploy, hence the syntax above.

[kousikpaul4u]

@rudradixit Following policy is working for me as well. Thanks 👍

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "arn:aws:s3:::aws-ttze-de-alp-abc-service", "arn:aws:s3:::aws-ttze-de-alp-abc-service/*" ] } ] }

[Aleksandr-Filichkin]

We need "Action": ["s3:List"] and "Resource": [ "${bucket}", "${bucket}/*" ]

[captcha17]

Hi @Aleksandr-Filichkin

You should specify action with asterisk like : ["s3:List*"] to include all 'List' actions
or in this case just set "s3:ListBucket" action to resolve this problem

Thanks.

[samthebest]

Is it possible to restrict list to a prefix? So I don't have to give list to an entire bucket?

[dmitry-weirdo]

Could someone please provide a policy with explanation of what to set in each field (Principal, Resource etc).
The same program with the same credentials runs for me on Windows but fails with 403 on doesObjectExist on Linux.