Aws-sdk-go: S3 PutObject Presigned URL

[rayrutjes]

Hi there,

I would like a user to be able to upload a whole directory directly to S3 in a specific folder of a bucket.
The destination folder is dictated by my app, and the content sent by a cli on the client side.
I don't want the files to actually reach my server so I thing I want to pre-sign a PutObjectRequest for each file.

Is that actually possible with the sdk?

My cli is also written in go, how would I write the actual upload to the presigned url?

Otherwise, would a temporary IAM user (Temporary Security Credentials) per folder be more suited for this? It would avoid the roundtrip per file to obtain the presigned url.

Please point me to the correct resources if this is possible.

[jasdel]

Hi @rayrutjes thanks for contacting us. Using presigned urls with the SDK is straightforward and only requires a few extra steps compared to using the API operations directly. A good strategy would be for your CLI client to send your service the file key, and MD5 checksum of each file to be uploaded to S3. Your service would then generate and return presigned URLs for each file.

At a minimum the Bucket and Key must be provided to put an object to S3. You may also want to consider adding a MD5 checksum for each file to ensure the presigned URL is used with the correct file. This also ensures that if the presigned URL is leaked it cannot be used to upload other unexpected content to your S3 bucket. Checkout this example from the SDK wiki on how to use presigned URLs.

In your service the presigned URL will be generated for the bucket, key, and MD5 of the file to be uploaded. Your CLI should provide these fields to your service. The service would then return a presigned URL.

    svc := s3.New(session.New())
    r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String("myBucket"),
        Key:    aws.String("myKey"),
    })
    r.HTTPRequest.Header.Set("Content-MD5", checksum)
    url, err := r.Presign(15 * time.Minute)
    if err != nil {
        fmt.Println("error presigning request", err)
        return
    }

    fmt.Println("URL", url)

The CLI client would receive the URL and make a PUT request with the file content to upload.

    req, err := http.NewRequest("PUT", url, fileReader)
    if err != nil {
        fmt.Println("error creating request", url)
        return
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println("failed making request")
        return
    }

[rayrutjes]

@jasdel thank you so much, this is actually just what I needed.

[jasdel]

Great, glad to help. Let us know if you have any other questions, feedback, or ideas how we can improve the SDK.

[rayrutjes]

@jasdel I have implemented the suggested solution and also the md5 control but I keep getting the following error message back from s3:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<Error>
    <Code>NotImplemented</Code>
    <Message>A header you provided implies functionality that is not implemented</Message>
    <Header>Transfer-Encoding</Header>
    <RequestId>0A6E12B1B237E050</RequestId>
    <HostId>gnpz4TUIqaWp67TsNZUpJu80Ll+XK+G55P+/qkLU/kiMH1yK/h2gUbsb9YIAtYqr12a9jSwGE98=</HostId>
</Error>

Any clue?

[jasdel]

Thanks for getting back in touch with us @rayrutjes. Is the Transfer-Encoding header being added by your app by chance? The following is the example I tried running to reproduce the error you reported, But wasn't able to. I also tried setting Transfer-Encoding header explicitly and wasn't able to reproduce it that way either.

Any additional script or example code you have would be very helpful in investigating this issue.

package main

import (
    "bytes"
    "crypto/md5"
    "encoding/base64"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

func main() {
    buf := bytes.NewReader(make([]byte, 10*1024*1024))
    h := md5.New()
    io.Copy(h, buf)

    //
    // Generate presigned PutObject URL
    svc := s3.New(session.New())
    r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String(os.Args[1]),
        Key:    aws.String(os.Args[2]),
    })
    r.HTTPRequest.Header.Set("Content-MD5", base64.StdEncoding.EncodeToString(h.Sum(nil)))
    url, err := r.Presign(15 * time.Minute)
    if err != nil {
        fmt.Println("error presigning request", err)
        return
    }
    fmt.Println("URL", url)
    buf.Seek(0, 0)

    //
    // Use the presigned URL to put a object to S3
    req, err := http.NewRequest("PUT", url, buf)
    if err != nil {
        fmt.Println("error creating request", url)
        return
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Println("failed making request")
        return
    }
    defer resp.Body.Close()

    //
    // Print out the response.
    fmt.Println("Status", resp.StatusCode, resp.StatusCode)
    o := &bytes.Buffer{}
    io.Copy(o, resp.Body)
    fmt.Println(o.String())
}

[rayrutjes]

@jasdel thank you again for your time, you put me in the right direction. I finally got it working.
The Transfer-Encoding was indeed added dynamically by "net/http/transfer.go" because I was not using a fixed sized type as body.
The solution was simply to manually set the req.ContentLength, and it now works like a charm.
I am very grateful for your help.

Cheers.

[rayrutjes]

@jasdel one last note for anyone trying to figure out the same kind of implementation. If you try to add ACL to the presigned url, you will also have to pass that exact same header to the request processing the actual upload. Is this actually a bug or a feature?

[jasdel]

The SDK intentionally will not hoist any x-amz header (which is what the ACL is send as) to the presigned URL. This is a blanket operation because those fields are not valid in the query string.

One way we've considered improving this experience is to add an additional way to generate a presigning. But instead of just generating a URL a net/http#Request would be returned. This would include all headers used in the signing so sharing the information would be a little bit easier.

[rayrutjes]

@jasdel I have to agree with you, a little helper function like the one available for adding the checksum to a request service/s3/content_md5.go would be nice to have. By the way, the official doc should also mention an example in go: http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html
Tell me if I can be of any use on those matters, would love to help you in return!

[brydavis]

Hotdog, @rayrutjes, thanks for the tip on setting req.ContentLength... Had given up on this a few weeks ago and then came across this! Works perfectly 👍