Aws-sdk-java: AWS S3: listBuckets and listObjects

[gowthamsankar312]

Hi,

We are collecting aws s3 data via aws java sdk v1.11.171. On using the following API's (listBuckets and listObjects), there is a misleading in calling those API calls.

'listBuckets' API allows me to get the data using both the default CLIENT as well as the client with an endpoint having any specific available regions of AWS. Doesn't depend upon the exact region on which the S3 bucket resides.

But 'listObjects' API throws an AmazonS3Exception saying "The authorization header is malformed" as it is expecting us to create a CLIENT with an endpoint having the exact region in which that S3 bucket is located. Why there is a difference in the functionality between these two?

Without knowing the region of a particular S3 Bucket in the first hand, how we could able to call 'listObjects' API for the first time.

Currently we are managing by trying with each endpoint with all possible regions.

Even to call the 'getBucketLocation' API, aws expect us to call using a CLIENT with an endpoint having exact region where the S3 bucket is located.

Please clarify on this.

Thank you,
Garry.

[dagnir]

But 'listObjects' API throws an AmazonS3Exception saying "The authorization header is malformed" as it is expecting us to create a CLIENT with an endpoint having the exact region in which that S3 bucket is located. Why there is a difference in the functionality between these two?

This limitation exists because the request to the S3 Bucket needs to be signed with the target region.

If you need to be able to list objects for buckets in multiple regions from a single client, you can enabled Global Bucket Access in the client: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#enableForceGlobalBucketAccess--

Even to call the 'getBucketLocation' API, aws expect us to call using a CLIENT with an endpoint having exact region where the S3 bucket is located.

Hmm this seems to work for me...

        AmazonS3 s3 = AmazonS3ClientBuilder.standard()
                .withRegion("eu-central-1")
                .build();

        System.out.println(s3.getBucketLocation("my-bucket-in-us-west-2"));

[fernomac]

GetBucketLocation works irrespective of bucket location as long as the client is not configured for us-east-1 (s3.amazonaws.com). If it is the request gets signed for us-east-1 but ends up wherever the bucket is actually hosted. :(

[dagnir]

@fernomac Ah right that makes sense. Thanks for the clarification!

[gowthamsankar312]

@fernomac Yes, GetBucketLocation works irrespective of the bucket location but how to find the bucket location if my bucket resides in us-east-1 region.

I have my bucket in "us-east-1" region for which GetBucketLocation gives me something which is not usual, as "US".

Is there any specific reason to return as "US" ?

OR will the returned string "US" may change in future? @dagnir

[dagnir]

Originally, S3's US East (Northern Virginia) region was called "US Standard", which is why the client returns "US" for buckets in us-east-1: https://aws.amazon.com/s3/faqs/#regions

S3's REST API returns an empty string for the bucket location when it's located in us-east-1, and on the SDK's end, it replaces the empty string with "US":

• https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETlocation.html
• https://github.com/aws/aws-sdk-java/blob/7b4854b5e03729e831d1e43d5e20c5098fdb8d90/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/transform/Unmarshallers.java#L135-L149

For 1.11.x, we will not change this behavior as it could break existing customers.

[gowthamsankar312]

Thank you @dagnir @fernomac for your information.

Hence closing this issue.