Aws-sdk-go: S3 Presigned URL Upload Example Works With Large Empty Slice but Not File (A header you provided implies functionality that is not implemented)

[IoTPanic]

Version of AWS SDK for Go?

v1.27.0

Version of Go (go version)?

1.13.3

What issue did you see?

The following HTTP response from the PUT request after creating a presigned URL-

https://idus-media.s3.amazonaws.com/test.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXXXXXXXXX%2F20200104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200104T022842Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=7463e0e927c7a4b900ac4a6d462f42459459931e776dcacfafb854011eff8e72
Status 501
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>Transfer-Encoding</Header><RequestId>5D052D3A2428E66D</RequestId><HostId>/F80WQNlI2ON/fe4Ah6l82g9SUk3g2Ox0BX47GrYKFlj1l771bN1QLW4zGola5s+nNVgTd8pUGU=</HostId></Error>

Steps to reproduce

I have this example I found from @jasdel after many hours of trying to get an image upload to work and I can see it successfully put a 1MB empty file into whatever S3 bucket on my account I specify with the file name I specify (Feel free to try yourself go run main.go {bucketname} {key})-

package main
import (
    "bytes"
    "crypto/md5"
    "encoding/base64"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)
func main() {
    buf := bytes.NewReader(make([]byte, 10*1024*1024))
    h := md5.New()
    io.Copy(h, buf)
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-east-1")},
    )
    svc := s3.New(sess)
    //var cl int64 = 12
    sum := base64.StdEncoding.EncodeToString(h.Sum(nil))
    r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String(os.Args[1]),
        Key:    aws.String(os.Args[2]),
        //ContentMD5: aws.String(sum),
        //ContentLength: aws.Int64(12),
    })
    fmt.Println(sum)
    r.HTTPRequest.Header.Set("Content-MD5", sum)
    url, err := r.Presign(15 * time.Minute)
    if err != nil {
        fmt.Println("error presigning request", err)
        return
    }
    fmt.Println("URL", url)
    buf.Seek(0, 0)
    //
    // Use the presigned URL to put a object to S3
    req, err := http.NewRequest("PUT", url, buf)
    if err != nil {
        fmt.Println("error creating request", url)
        return
    }
    req.Header.Set("Content-MD5", sum)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println("failed making request")
        return
    }
    defer resp.Body.Close()
    // Print out the response.
    fmt.Println("Status", resp.StatusCode)
    o := &bytes.Buffer{}
    io.Copy(o, resp.Body)
    fmt.Println(o.String())
}

Now, the os.File type in golang implements the io.Reader interface, so I than replace buf := bytes.NewReader(make([]byte, 10*1024*1024)) with a buf, err := os.Open("pic.png") (Handling the error of course). I than get returned the error shown above, and the only way to change the outcome I have found is to add a content length, and even setting the content length to the legitamit file content length I instead get returned a signature invalid (Adding content length to both the PutObjectInput and req). Why would this create a different outcome? I have tried evertyhing I can think of to upload this dang photo, even using other examples or writing my own code.

[IoTPanic]

Figured out that if you copy the contents of the file into another buffer, than it will work, but I'm still struggling to understand why this and a few other attempts didn't work.

[jasdel]

Thanks for reaching out @IoTPanic. I think the issue that you're running into is that the Content-Length header is not being set. This is the behavior of the Go HTTP client when a request with a non-nil Body, the client is not able to determine the length of the body, and the Content-Length header is not set. In that case the HTTP client will send the request using Chunked Transfer encoding, which is not supported by S3. To resolve this regardless of the type of bytes buffer/file input used for the http.Request.Body value set the request's Content-Length header.

When calling PutObjectRequest you could provide the Content-Length header by setting the PutObjectInput.ContentLength member. This will include the Content-Length value in the presigned URL as well, though as a header.

When using the ContentLength value in the Presigned URL its important to call PresignRequest instead of Presign. The Presign method will only return the URL, not the headers that were also included in the signature. Several of the API parameter values for PutObject must be sent as headers not in the signed URL's query string.

[IoTPanic]

Hey Jasdel,

Thanks for the reply, I understand that the content length needs to be added (Or is supposed to be), but if thats the case, than what I have would be unexpected behavior. My example code is code you wrote, and works without specifying the content length, or a body, it creates an empty 1MB slice in a buffer that implements io.reader. Replacing the empty io.Reader with a file object which also implements the reader interface fails bringing my error. The guy working with me found that creating a buffer, copying the file contents into the buffer, than using that buffer instead works just fine without the need for PresignRequest or adding a content length. When I was using the file buffer, and I added a content length to both the presign request and the PUT request, I was getting an invalid signature back, even when using PresignRequest and adding the returned headers through an iterator.

Just trying to figure out why I'm experiencing this behavior, it is no longer an issue for me.

Sam

[jasdel]

Thanks for the update. The reason you'll see different behavior between os.File and bytes.Buffer is that Go's http.Request's constructor http.NewRequest will attempt to type assert the body parameter into a bytes.Buffer, bytes.Reader, or strings.Reader. If the input body is one of those types it will extract the length of the body to use as the Content-Length header. This doesn't happen for os.File, or any other type. For those types Content-Length header must be set explicitly. This behavior is documented on http.NewRequestWithContext not NewRequest though.

Have you tried not setting the PutObjectInput.ContentLength member, leaving it blank, and only set the Content-Length header on the new Go http.Request created? I would not expect a signature validation error in that case.

Also, if don't mind could you share the link to that example. I'm happy to update it for clarity.

[IoTPanic]

Aha! Okay, that makes sense! Putting content length on the HTTP request but not the presign was the only thing I didn't try 😂.

The code example is in the steps to reproduce in my original post. Thank you very much!