Aws-sdk-go: GetBucketLocation fails if region specified does not match the region the bucket is in.
There seems to be a regression of this issue: #380.
This is my code:-
awsSession := session.New()
s3Service := s3.New(awsSession, &aws.Config{
LogLevel: aws.LogLevel(aws.LogDebugWithHTTPBody),
})
resp, err := s3Service.GetBucketLocation(&s3.GetBucketLocationInput{
Bucket: aws.String(c.String("bucket")),
})
if err != nil {
return cli.NewExitError(fmt.Sprintf("%s", err), 1)
}
When I set the AWS_REGION environment variable be different (us-east-1) to that of the bucket (eu-west-1), I get 403 errors:-
2016/06/11 15:06:31 DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3.amazonaws.com/s3up-test?location= HTTP/1.1
Host: s3.amazonaws.com
User-Agent: aws-sdk-go/1.1.34 (go1.6; darwin; amd64)
[… AWS auth headers …]
Accept-Encoding: gzip
-----------------------------------------------------
2016/06/11 15:06:32 DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 11 Jun 2016 14:06:31 GMT
Server: AmazonS3
X-Amz-Id-2: EGxuZn97P1HxAYJQLVZBqsyFM3zxgdJd2G7oIR1gGK9fZ1MH2r7D0fMx3wZo4IGdPR1TS83oAyM=
X-Amz-Request-Id: 54E6B7601DD1DC3B
f3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>54E6B7601DD1DC3B</RequestId><HostId>EGxuZn97P1HxAYJQLVZBqsyFM3zxgdJd2G7oIR1gGK9fZ1MH2r7D0fMx3wZo4IGdPR1TS83oAyM=</HostId></Error>
0
-----------------------------------------------------
AccessDenied: Access Denied
status code: 403, request id: 54E6B7601DD1DC3B
When the region is correct, the output is this:-
2016/06/11 15:07:28 DEBUG: Request s3/GetBucketLocation Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-eu-west-1.amazonaws.com/s3up-test?location= HTTP/1.1
Host: s3-eu-west-1.amazonaws.com
User-Agent: aws-sdk-go/1.1.34 (go1.6; darwin; amd64)
[… AWS auth headers …]
Accept-Encoding: gzip
-----------------------------------------------------
2016/06/11 15:07:29 DEBUG: Response s3/GetBucketLocation Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sat, 11 Jun 2016 14:07:29 GMT
Server: AmazonS3
X-Amz-Id-2: yz1z8vs+rFrZA+aGKSwSfN/moPNPrDLeYPU4Zh/amT8HoIqhYq9c3MkwL0ixs3aqWFdP/Vgk5KQ=
X-Amz-Request-Id: D949FAF64D0BD627
89
<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">eu-west-1</LocationConstraint>
0
-----------------------------------------------------
{
LocationConstraint: "eu-west-1"
}
I should be able to use the API/SDK to determine the location of an S3 bucket without knowing the location of the S3 bucket, shouldn't I?
matthew-andrews
All 19 comments
Hello @matthew-andrews, thank you for bringing this to our attention. I cannot reproduce this. I've tried creating a bucket in eu-west-1 and still it succeeds. What Go and SDK versions are you using?
xibz
on 13 Jun 2016
@matthew-andrews I've been trying to reproduce this issue. I'm only able to encounter the AccessDesigned error when I don't have access to the bucket. For example I try to head your s3up-test bucket I get the error regardless of which region I make the request from. When I do a GetBucketLocation on a bucket I have permission to access I can call it the API successfully from any region. Do you have any policies setup on this bucket restricting the the region a request can be made from?
jasdel
on 13 Jun 2016
Hi, I've just tried again and can definitely reproduce the issue.
I've created a reduced test case here:-
https://github.com/matthew-andrews/s3-reduced
Which I have also reproduced on CircleCI (click make test to see the output):-
WITH EU REGION
export AWS_REGION=eu-west-1; echo $AWS_REGION && go run main.go;
eu-west-1
RESP:
{
LocationConstraint: "eu-west-1"
}
ERR:
<nil>
----
WITH NO REGION
unset AWS_REGION; echo $AWS_REGION && go run main.go;
RESP:
{
}
ERR:
MissingRegion: could not find region configuration
The user is a plain empty AWS IAM user with no groups, no policies, etc. It is granted access to a bucket by a bucket policy attached to the bucket itself and that policy is this:-
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::862433486593:user/DeployUserFor_s3up"
},
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::s3up-test/*",
"arn:aws:s3:::s3up-test"
]
}
]
}
I believe the go version on both my local system and in CircleCI is 1.6 and I think Circle probably pulls in this sdk from master. I just tested my local machine on v1.1.34.
Hope this helps!
matthew-andrews
on 13 Jun 2016
Hm, actually that's a different error isn't it…
matthew-andrews
on 13 Jun 2016
OK, try here — https://circleci.com/gh/matthew-andrews/s3-reduced/4
Click make test
EU REGION
export AWS_REGION=eu-west-1; echo $AWS_REGION && go run main.go;
eu-west-1
RESP:
{
LocationConstraint: "eu-west-1"
}
ERR:
<nil>
----
WITH NO REGION
export AWS_REGION=us-east-1; echo $AWS_REGION && go run main.go;
us-east-1
RESP:
{
}
ERR:
AccessDenied: Access Denied
status code: 403, request id: 740F596935D64CDE
Hello @matthew-andrews, I've reached out to s3 to further investigate this issue. Please let us know if there is anything else we can do for you.
xibz
on 16 Jun 2016
Hi @matthew-andrews are you still encountering this issue? It may be helpful to ask on the S3 AWS forums. Other users may of experienced the issue you are encountering, and will help get more visibility with your issue with S3.
jasdel
on 28 Jun 2016
I think it's a bug…?
matthew-andrews
on 29 Jun 2016
Hi @matthew-andrews I think this is a S3 service API issue, not SDK. This is why I was suggesting the S3 forums.
GetBucketLocation requires the requester to be the owner of the bucket.
The Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket blog post goes through the process needed to grant permission to a bucket, including permission to get a bucket's location.
jasdel
on 29 Jun 2016
Ah you think the IAM permissions are wrong? I'll check that and get back to you. Apologies in advance if that's the case.
matthew-andrews
on 29 Jun 2016
I think that its a possibility, especially since you're able to get the bucket in one region but not another. I'm thinking that it is possible that there is a policy on the bucket that is limiting permissions to a specific region.
jasdel
on 29 Jun 2016
Hi @matthew-andrews any luck identifying if this issue was caused by a policy rule?
jasdel
on 7 Jul 2016
Hi @matthew-andrews We's escalated this issue to the S3 Service team, and they are currently investigating. We'll update once we have more information.
jasdel
on 11 Jul 2016
Hi @matthew-andrews The S3 team got back with me and suggested the best API to use is HEAD bucket. GetBucketLocation uses a more complex permissions model where HeadBucket can be called by anyone. The bucket's region will be returned as a X-Amz-Bucket-Region header.
The HEAD bucket request can be made regardless of request signing, so anonymous (unsigned) requests can be made requesting a bucket's region.
I'm going to close this issue with this info. Please let us know if you have additional feedback, or issues.
jasdel
on 31 Aug 2016
I'm not sure this issue should be closed. I can reproduce this still. If I have to set region to match the bucket location, what is the point of this API call at all? You can access the region from Config, and it must be set: https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config. At this point the only useful purpose to the API call would be to check if the bucket is in the current region and iterate all regions creating a new Session for each region until you found the right one. Hopefully I'm missing something obvious, but for now I'm heading towards the HEAD workaround.
moofish32
on 21 Jan 2020
I'm not sure this issue should be closed. I can reproduce this still. If I have to set region to match the bucket location, what is the point of this API call at all? You can access the region from Config, and it must be set: https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config. At this point the only useful purpose to the API call would be to check if the bucket is in the current region and iterate all regions creating a new Session for each region until you found the right one. Hopefully I'm missing something obvious, but for now I'm heading towards the HEAD workaround.
I agree, this should not have been closed. I have reproduced the problem just today when using this dependency:
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.747</version>
</dependency>
Note: The suggested workaround above to use HeadBucket did not work for me using the Java API. It returned a 400 Bad Request when making the call on a bucket in a region different than the region used to create the AmazonS3 client.
A workaround which does work is to connect to each region in turn to determine which connection allows getBucketLocation to return without throwing an exception. The region which does not throw the exception is the region of the bucket. Pretty poor workaround but it does the job.
owbmoore
on 23 Mar 2020
Going to echo the previous sentiments and add some more. @jasdel can this be reopened?
I just did some more testing with both the HeadBucket path and GetBucketLocation path.
First, here is my code:
package main
import (
"log"
"net/http"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
bucket := "168-dlx-one-staging-us-west-2-assets"
region := "us-west-1"
log.Printf("Testing %s with client in %s", bucket, region)
headBucketRegion(bucket, region)
getBucketLocation(bucket, region)
region = "us-west-2"
log.Printf("Testing %s with client in %s", bucket, region)
headBucketRegion(bucket, region)
getBucketLocation(bucket, region)
region = "us-east-1"
log.Printf("Testing %s with client in %s", bucket, region)
headBucketRegion(bucket, region)
getBucketLocation(bucket, region)
region = "us-east-2"
log.Printf("Testing %s with client in %s", bucket, region)
headBucketRegion(bucket, region)
getBucketLocation(bucket, region)
}
func getBucketLocation(bucket, region string) {
c := getCli(region)
input := &s3.GetBucketLocationInput{
Bucket: aws.String(bucket),
}
result, err := c.GetBucketLocation(input)
if err != nil {
log.Printf("GetBucketLocation failed: %s", err.Error())
return
}
log.Printf("GetBucketLocation %s location: %s",
bucket,
*result.LocationConstraint,
)
}
func headBucketRegion(bucket, region string) {
c := getCli(region)
input := &s3.HeadBucketInput{
Bucket: aws.String(bucket),
}
req, _ := c.HeadBucketRequest(input)
err := req.Send()
if err != nil {
log.Printf("HeadBucket failed: %s", err.Error())
return
}
log.Printf("HeadBucket %s region: %s",
bucket,
req.HTTPResponse.Header.Get("X-Amz-Bucket-Region"),
)
}
func getCli(region string) *s3.S3 {
cl := &http.Client{
Timeout: 60 * 30 * time.Second,
}
creds := credentials.NewEnvCredentials()
_, err := creds.Get()
if err != nil {
log.Printf("bad s3 credentials: %s", err)
return nil
}
cfg := aws.NewConfig().
WithRegion(region).
WithCredentials(creds).
WithMaxRetries(10).
WithHTTPClient(cl)
return s3.New(session.New(), cfg)
}
And here is the output:
2020/03/26 14:38:24 Testing 168-dlx-one-staging-us-west-2-assets with client in us-west-1
2020/03/26 14:38:25 HeadBucket failed: BucketRegionError: incorrect region, the bucket is not in 'us-west-1' region at endpoint ''
status code: 301, request id: , host id:
2020/03/26 14:38:26 GetBucketLocation failed: AccessDenied: Access Denied
status code: 403, request id: EB2818BE9D1502C8, host id: +KeyBs1bF9Q8jCZfAmpLeq4izmCdz/E94rYpHA4N+8KKEkKZtxrVtEAiiojwdWBuW6ZzGHQalrs=
2020/03/26 14:38:26 Testing 168-dlx-one-staging-us-west-2-assets with client in us-west-2
2020/03/26 14:38:26 HeadBucket 168-dlx-one-staging-us-west-2-assets region: us-west-2
2020/03/26 14:38:27 GetBucketLocation 168-dlx-one-staging-us-west-2-assets location: us-west-2
2020/03/26 14:38:27 Testing 168-dlx-one-staging-us-west-2-assets with client in us-east-1
2020/03/26 14:38:28 HeadBucket failed: BadRequest: Bad Request
status code: 400, request id: 88F82FAA425B9252, host id: w82ARt4dbQ+UzKeCKyMh1q5Keg3sumXB8N+2Cm61y1RE+4/ESvCldlpINPLx/NiDkqVGbatdEMU=
2020/03/26 14:38:28 GetBucketLocation failed: AccessDenied: Access Denied
status code: 403, request id: 3EAD2DC6D028FD58, host id: khsJrRtiYFm0g7CNKojSRRZiOatGrBwvTFoVIycVpsD4g3Ppg+ue6QYRAq6xO8w5D/7d734gl/s=
2020/03/26 14:38:28 Testing 168-dlx-one-staging-us-west-2-assets with client in us-east-2
2020/03/26 14:38:29 HeadBucket failed: BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint ''
status code: 301, request id: , host id:
2020/03/26 14:38:30 GetBucketLocation failed: AccessDenied: Access Denied
status code: 403, request id: 4D5B7B87060EFB8D, host id: eL4+X9uv2ZNALL0ZjnKLk26umRysZ4jhDrwpEEi9N2ZgYcIrEzcvZsv97f0PALkLjhQSWc24+7g=
You can see here using _either_ method you must have your client region set to the region of the bucket for them to work. I feel like this is counterintuitive to a major (if not the primary) use case for GetBucketLocation: I have a bucket, but I do not know the region it exists in.
metroidprototype
on 26 Mar 2020
After some discussion with AWS support I found out that the issue I was having with this behavior is the previously recommended solutions do NOT work when you are making cross account requests to HeadBucket and GetBucketLocation. A simple http HEAD does work and requires no credentials or cross account access in all cases for me. Here is my sample go code that I am now using:
func getBucketRegion(bucket string) (string, error) {
url := fmt.Sprintf("https://%s.s3.amazonaws.com", bucket)
res, err := http.Head(url)
if err != nil {
return "", err
}
return res.Header.Get("X-Amz-Bucket-Region"), nil
}
@metroidprototype, this approach will throw an error in case your bucket "mutates" the endpoint URL.
For example, bucket www.website.com will cause an SSL error: Head "https://www.website.com.s3.amazonaws.com": x509: certificate is valid for *.s3.amazonaws.com, s3.amazonaws.com, not www.website.com.s3.amazonaws.com
The only working solution I found is to create a map of clients for each region, check where the bucket resides and use relevant client to perform operations on that bucket 🤯 🔫. Something like:
package main
import (
"fmt"
"os"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/endpoints"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
func main() {
var cli map[string]*s3.S3 = make(map[string]*s3.S3)
for _, partition := range endpoints.DefaultPartitions() {
for region := range partition.Regions() {
cli[region] = s3.New(session.Must(session.NewSession(&aws.Config{
Region: aws.String(region),
})))
}
}
r, err := cli["us-east-1"].ListBuckets(&s3.ListBucketsInput{})
if err != nil {
fmt.Println(err)
os.Exit(1)
}
for _, bucket := range r.Buckets {
l, err := cli["us-east-1"].GetBucketLocation(&s3.GetBucketLocationInput{
Bucket: bucket.Name,
})
if err != nil {
fmt.Println(err)
os.Exit(1)
}
var region string
if l.LocationConstraint == nil {
region = "us-east-1"
} else {
region = *l.LocationConstraint
}
o, err := cli[region].GetBucketTagging(&s3.GetBucketTaggingInput{
Bucket: bucket.Name,
})
if err != nil && !strings.Contains(err.Error(), "The TagSet does not exist") {
fmt.Println(err)
os.Exit(1)
} else if err == nil {
fmt.Println(o.TagSet)
}
}
}
I believe the solution should be cross region calls (blog):
New pseudo region: aws-global to make cross region requests
But I was not able to operate against aws-global region 😢
anton-yurchenko
on 15 Aug 2020
Related issues
weaselsnak
·
3Comments
delitescere
·
5Comments
novalagung
·
4Comments
bflad
·
4Comments
svett
·
4Comments
Most helpful comment
Hi @matthew-andrews The S3 team got back with me and suggested the best API to use is HEAD bucket. GetBucketLocation uses a more complex permissions model where HeadBucket can be called by anyone. The bucket's region will be returned as a
X-Amz-Bucket-Regionheader.The HEAD bucket request can be made regardless of request signing, so anonymous (unsigned) requests can be made requesting a bucket's region.
I'm going to close this issue with this info. Please let us know if you have additional feedback, or issues.