Aws-sdk-go: Can't use presigned URL with specifying Content-Length header

[cdelguercio]

Version of AWS SDK for Go?

v1.10.14 (2017-07-20)

Version of Go (go version)?

go version go1.8.3 linux/amd64

What issue did you see?

So I'm trying to generate presigned URLs that DON'T specify a Content-Length. My code is based of the presigned URL example code here: https://github.com/aws/aws-sdk-go/tree/master/example/service/s3/presignURL. I can run the example fine, but if I try and change it to not include Content-Length I get the same issues that my code has.

In the example's server.go file there is a comment that says that you can set ContentLength to 0 in the PutObjectRequest and then the Content-Length header won't be included in the signature:

sdkReq, _ := svc.PutObjectRequest(&s3.PutObjectInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(key),

    // If ContentLength is 0 the header will not be included in the signature.
    ContentLength: aws.Int64(contentLen),
})

In my code I changed that line to "ContentLength: aws.Int64(0)," (I've also tried just commenting the line out)

I get the presigned URL fine:

https://my-s3-bucket.s3-us-west-2.amazonaws.com/test/e615fcf8-a832-48da-9c1a-6c0e43f7fe00?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXX%2F20170722%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20170722T041317Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256&X-Amz-Signature=XXXXXXXXXXXXXX

But when I try and use it I get:
<Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

Here's my test client code:

data, err := os.Open(localFilePath)
if err != nil {
    return err
}
defer data.Close()
req, err := http.NewRequest("PUT", url, data)
if err != nil {
    return err
}

//req.Header.Set("Content-Length", "0")
//req.Header.Set("X-Amz-Content-Sha256", "UNSIGNED-PAYLOAD")

client := &http.Client{}
res, err := client.Do(req)

As you can see, I've tried including a "0" Content-Length header, which does nothing. I've also tried including an X-Amz-Content-Sha256 header with all sorts of capitalizations, but it always gives me this:

<Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message>

[jasdel]

Thanks for reaching out to us @cdelguercio. I'm taking a look at this test case. Do you experience the issue if the PutObjectInput's ContentLength value is not set at all? I would expect this to be the same as setting it to 0.

[cdelguercio]

The goal is to have the client not include a ContentLength header at all in both their request for the presigned URL and the actual upload to S3. Yes, I've tried commenting out the "ContentLength: aws.Int64(contentLen)," line completely to no effect.

[jasdel]

Thanks for the update I've been able to reproduce the issue and trying to figure out if this is just an issue in how the presigned URL is being used or a bug in the SDK's signing.

[jasdel]

After a lot of experimentation I realized the error I was "duplicating" was because I was using POST instead of PUT in the HTTP method of my new request.

The following code example is what i did to create and send a pre-signed request.

    svc := s3.New(sess)

    sdkReq, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })

    u, header, err := sdkReq.PresignRequest(15 * time.Minute)
    if err != nil {
        panic(err)
    }

    req, _ := http.NewRequest("PUT", u, nil)
    for k, vs := range header {
        for _, v := range vs {
            req.Header.Add(k, v)
        }
    }

    b, _ := httputil.DumpRequest(req, true)
    fmt.Println("Request")
    fmt.Println(string(b))

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }

    b, _ = httputil.DumpResponse(resp, true)
    fmt.Println("Response")
    fmt.Println(string(b))

[cdelguercio]

It works fine when I put nil data in http.NewRequest("PUT", u, nil). But when I try and upload a file, it gives me:

<Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message>

Here's my file reading code:

data, err := os.Open(localFilePath)
if err != nil {
    return err
}
defer data.Close()
req, err := http.NewRequest("PUT", url, data)

When all I do is change data to nil, it works fine.

[jasdel]

Thanks for clarifying that @cdelguercio. It looks like S3 require's the ContentLength value to be set if a body is used. If the body is empty the ContentLength does not need to be set.

Not setting the content length when creating the pre-signed URL just prevents it from being included in the signature. Not being included allows the user of the pre-signed url to provide a PUT with any length content.

[jasdel]

Oh also, make sure to set Request.ContentLength not the Content-Length header. I noticed that setting Content-Length header is ignored for these kind of requests.

[jasdel]

Ah ok I think i figured out the issue. In Go 1.8 if a content length is not set the request will be sent with Transfer-Encoding: chunked. S3 does not support chunked transfer encoding. To prevent this you must set the http.Request.ContentLength value to the length of the content you want to put to S3.

[cdelguercio]

Haha, yup, just figured that out. As a reference for others, here's some of my solution Go code:

localFile, err := os.Open(localFilePath)
if err != nil {
    return err
}
defer localFile.Close()
req, err := http.NewRequest("PUT", url, localFile)
if err != nil {
    return err
}

for k, vs := range header {
    for _, v := range vs {
        fmt.Printf("Header name: %v, Header value: %v\n", k, v)
        req.Header.Add(k, v)
    }
}

info, _ := localFile.Stat()
req.ContentLength = info.Size()

client := &http.Client{}
res, err := client.Do(req)
if err != nil {
    return err
}

This corresponds with asking for the presigned URL WITHOUT the content length like this:

sdkReq, _ := svc.PutObjectRequest(&s3.PutObjectInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(key),
})
url, header, err := sdkReq.PresignRequest(s.presignedURLExpirationTime)

Thanks so much for all your help!

[jasdel]

Great!, let us know if you have additional questions or feedback about the SDK.

[NeelamAggarwal]

@jasdel I am doing the same with one difference, my bucket is encrypted and
I added ServerSideEncryption: aws.String("aws:kms"),

I get error
&{403 Forbidden 403 HTTP/1.1 1 1 map[X-Amz-Request-Id:[F7A7303D7D2545D6] X-Amz-Id-2:[5LZEcPAO7qIYwyWGTPU5vTgJhisDpdGn4bIwR8CIziOndVtPqup2M6BB0nWxkXbrjcyXFyUppAs=] Content-Type:[application/xml] Date:[Fri, 24 Aug 2018 19:37:24 GMT] Server:[AmazonS3]] 0xc420378040 -1 [chunked] true false map[] 0xc4201d0200 0xc4202c2210} <nil>