Google-cloud-go: storage: custom headers are overwritten by either Go SDK or Storage service with SignedURL

[muratsplat]

Client

Go sdk -> cloud.google.com/go v0.41.0
and I use Storage service.

Describe Your Environment

Debian on GCP K8s Engine

Expected Behavior

I want to add follow Header to be download able by any web browser for regarding object.

    signedURL, err := storage.SignedURL(file.BucketTempBlobVideo, tmpFileName, &storage.SignedURLOptions{
        // TODO: move it to the env
        GoogleAccessID: GCPAccount,
        PrivateKey:     readCredential(),
        Method:         "GET",
        Expires:        expiresAt,
        Headers: []string{
            "Content-Disposition: inline",
            "Content-Disposition: attachment",
            fmt.Sprintf("Content-Disposition: attachment; filename=\"%s\"", tmpFileName),
        },
    })

Actual Behavior

But my headers fields are overwrite by GCP Storage like that
this is the result of curl -v https://storage.googleapis.com/bla/bala/foo.mp4

* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< x-guploader-uploadid: AEnB2UphcEqdN92EHyvLdNlGUQPc6MKxg2obPdlCS-SK25tpwEJ007_d697VN-uAususISbP65l5lU4TGD9cLK_AuSTxoBs1uw
< expires: Thu, 18 Jul 2019 08:24:03 GMT
< date: Thu, 18 Jul 2019 08:24:03 GMT
< cache-control: private, max-age=0
< last-modified: Thu, 18 Jul 2019 08:20:52 GMT
< etag: "54ae4dcd870a1c2653309f41fb61842b"
< x-goog-generation: 1563438052792850
< x-goog-metageneration: 1
< x-goog-stored-content-encoding: identity
< x-goog-stored-content-length: 1407113503
< content-type: video/mp4
< x-goog-hash: crc32c=sVmYIA==
< x-goog-hash: md5=VK5NzYcKHCZTMJ9B+2GEKw==
< x-goog-storage-class: REGIONAL
< accept-ranges: bytes
< content-length: 1407113503
< server: UploadServer
< alt-svc: quic=":443"; ma=2592000; v="46,43,39"

where is the my headers in the response of GCP signURL ?

[jadekler]

cc @frankyn @odeke-em I think this is intentional, but y'all would know better.

[odeke-em]

Hello @muratsplat, thank you for this question and welcome to the Google-Cloud-Go project!

So to answer your question: There are 2 things to tackle in your question:

1. I believe that you can only modify that metadata when creating a resumable upload with a POST and when doing a PUT. In your code you are doing a GET(which doesn't mutate the object's metadata) and perhaps you are passing along those headers but that's only to match the signature (verification) as per https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.Headers
   which documents what those headers are
   

2. If you were patching to update "Content-Disposition", you are setting all the possible values for "Content-Disposition" so the first one is going to match as "inline" and the file will always be displayed in the browser. Please see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition. If you want to force it to be downloadable from the browser with a specific filename, please use only Content-Disposition: attachment; filename="<DOWNLOADABLE_FILENAME>"

End to end example

Upload and set the custom headers

You can set the settable metadata as well as the custom metadata headers by

package main

import (
    "context"
    "encoding/json"
    "io/ioutil"
    "log"
    "net/http"
    "net/http/httputil"
    "strings"
    "time"

    "golang.org/x/oauth2/google"

    "cloud.google.com/go/storage"
)

func main() {
    log.SetFlags(0)

    ctx := context.Background()
    creds, err := google.FindDefaultCredentials(ctx)
    if err != nil {
        log.Fatalf("Failed to find default credentials: %v", err)
    }

    type sa struct {
        ClientEmail string `json:"client_email"`
        PrivateKey  string `json:"private_key"`
    }

    sa1 := new(sa)
    if err := json.Unmarshal(creds.JSON, sa1); err != nil {
        log.Fatalf("Failed to unmarshal JSON to get private key: %v", err)
    }

    hc := http.DefaultClient
    filename := "1511.txt"
    signedURL, err := storage.SignedURL("cloud-go-1511", filename, &storage.SignedURLOptions{
        Method:         "POST",
        ContentType:    "text/plain",
        GoogleAccessID: sa1.ClientEmail,
        Expires:        time.Now().Add(time.Hour),
        PrivateKey:     []byte(sa1.PrivateKey),
        Headers: []string{
            "x-goog-resumable: start",
            `Content-Disposition: attachment; filename="dl.txt"`,
            "x-goog-meta-issue: 1511",
            "x-goog-meta-repo: go-cloud",
        },
    })
    if err != nil {
        log.Fatalf("SignedURL error: %v", err)
    }

    // 1. Generate the upload URL with a resumable upload.
    rreq, err := http.NewRequest("POST", signedURL, nil)
    if err != nil {
        log.Fatalf("Failed to create initial resumable upload: %v", err)
    }
    // Ensure that the headers match exactly as in the signedURL options.
    rreq.Header.Set("Content-Type", "text/plain")
    rreq.Header.Set("x-goog-resumable", "start")
    rreq.Header.Add("Content-Disposition", `attachment; filename="dl.txt"`)
    rreq.Header.Add("x-goog-meta-issue", "1511")
    rreq.Header.Add("x-goog-meta-repo", "go-cloud")

    rreqOut, _ := httputil.DumpRequestOut(rreq, true)
    log.Printf("Resumable request out:\n\n%s\n\n", rreqOut)
    rres, err := hc.Do(rreq)
    if err != nil {
        log.Fatalf("Resumable upload response failure: %v", err)
    }
    rresBlob, _ := httputil.DumpResponse(rres, true)
    log.Printf("Response:\n\n%s", rresBlob)

    if rres.StatusCode != http.StatusCreated {
        log.Fatalf("Failed to start resumable upload got statusCode: %d want %d\n", rres.StatusCode, http.StatusCreated)
    }

    uploadURL := rres.Header.Get("Location")
    if uploadURL == "" {
        log.Fatal("Failed to get uploadURL after creating resumable URL")
    }

    // 2. Now that we have the upload URL, actually upload it.
    payload := "This is the content at: " + time.Now().Format(time.RFC3339Nano)
    reqBody := strings.NewReader(payload)
    req, err := http.NewRequest("PUT", uploadURL, reqBody)
    if err != nil {
        log.Fatalf("Failed to compose http.Request: %v", err)
    }
    reqOut, _ := httputil.DumpRequestOut(req, true)
    log.Printf("Request out:\n\n%s\n\n", reqOut)
    reqBody.Reset(payload)
    req.Body = ioutil.NopCloser(reqBody)

    res, err := hc.Do(req)
    if err != nil {
        log.Fatalf("Failed to make request to remote GCS\n%v\n\n", err)
    }
    resBlob, _ := httputil.DumpResponse(res, true)
    log.Printf("Response:\n\n%s", resBlob)
}

prints out

go run main.go 
Resumable request out:

POST /cloud-go-1511/1511.txt?<REDACTED> HTTP/1.1
Host: storage.googleapis.com
User-Agent: Go-http-client/1.1
Content-Length: 0
Content-Disposition: attachment; filename="dl.txt"
Content-Type: text/plain
X-Goog-Meta-Issue: 1511
X-Goog-Meta-Repo: go-cloud
X-Goog-Resumable: start
Accept-Encoding: gzip



Response:

HTTP/2.0 201 Created
Content-Length: 0
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Content-Type: text/html; charset=UTF-8
Date: Fri, 19 Jul 2019 23:44:39 GMT
Location: https://storage.googleapis.com/cloud-go-1511/1511.txt?<REDACTED>
Server: UploadServer
X-Guploader-Uploadid: <REDACTED>

Request out:

PUT /cloud-go-1511/1511.txt?<REDACTED> HTTP/1.1
Host: storage.googleapis.com
User-Agent: Go-http-client/1.1
Content-Length: 56
Accept-Encoding: gzip

This is the content at: 2019-07-19T16:44:40.066595-07:00

Response:

HTTP/2.0 200 OK
Content-Length: 0
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Content-Type: text/html; charset=UTF-8
Date: Fri, 19 Jul 2019 23:44:40 GMT
Etag: "d89db7289d5f29f2c200c47869a34671"
Server: UploadServer
Vary: Origin
X-Goog-Generation: 1563579880461930
X-Goog-Hash: crc32c=3cgUNg==
X-Goog-Hash: md5=2J23KJ1fKfLCAMR4aaNGcQ==
X-Goog-Metageneration: 1
X-Goog-Stored-Content-Encoding: identity
X-Goog-Stored-Content-Length: 56
X-Guploader-Uploadid: <REDACTED>

and when viewed in the Google Cloud Storage UI

Download with SignedURL

package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "net/http/httputil"
    "time"

    "golang.org/x/oauth2/google"

    "cloud.google.com/go/storage"
)

func main() {
    log.SetFlags(0)

    ctx := context.Background()
    creds, err := google.FindDefaultCredentials(ctx)
    if err != nil {
        log.Fatalf("Failed to find default credentials: %v", err)
    }

    type sa struct {
        ClientEmail string `json:"client_email"`
        PrivateKey  string `json:"private_key"`
    }

    sa1 := new(sa)
    if err := json.Unmarshal(creds.JSON, sa1); err != nil {
        log.Fatalf("Failed to unmarshal JSON to get private key: %v", err)
    }

    hc := http.DefaultClient
    filename := "1511.txt"

    getSignedURL, err := storage.SignedURL("cloud-go-1511", filename, &storage.SignedURLOptions{
        Method:         "GET",
        GoogleAccessID: sa1.ClientEmail,
        Expires:        time.Now().Add(time.Hour),
        PrivateKey:     []byte(sa1.PrivateKey),
    })

    rreq, err := http.NewRequest("GET", getSignedURL, nil)
    if err != nil {
        log.Fatalf("Failed to create initial resumable upload: %v", err)
    }

    rreqOut, _ := httputil.DumpRequestOut(rreq, true)
    log.Printf("Resumable request out:\n\n%s\n\n", rreqOut)
    rres, err := hc.Do(rreq)
    if err != nil {
        log.Fatalf("Resumable upload response failure: %v", err)
    }
    rresBlob, _ := httputil.DumpResponse(rres, true)
    log.Printf("Response:\n\n%s", rresBlob)
}

and when made public and gotten with cURL

curl -i https://storage.googleapis.com/cloud-go-1511/1511.txt

HTTP/2 200 
x-guploader-uploadid: <REDACTED>
expires: Sat, 20 Jul 2019 00:40:07 GMT
date: Fri, 19 Jul 2019 23:40:07 GMT
last-modified: Fri, 19 Jul 2019 23:34:24 GMT
etag: "d89db7289d5f29f2c200c47869a34671"
x-goog-generation: 1563579880461930
x-goog-metageneration: 1
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 56
x-goog-meta-issue: 1511
x-goog-meta-repo: go-cloud
content-type: text/plain
content-disposition: attachment; filename="dl.txt"
x-goog-hash: crc32c=3cgUNg==
x-goog-hash: md5=2J23KJ1fKfLCAMR4aaNGcQ==
x-goog-storage-class: MULTI_REGIONAL
accept-ranges: bytes
content-length: 56
server: UploadServer
cache-control: public, max-age=3600
age: 5
alt-svc: quic=":443"; ma=2592000; v="46,43,39"

This is the content at: 2019-07-19T16:44:40.066595-07:00

when downloaded by my browser

Hope this helps clear any confusion! I shall close this but please don't hesitate to open issues and ask questions in case of anything and thank you for using this package!

[muratsplat]

@odeke-em thank you feedback.

my fault is that try to make be downloadable object on getting object. I have should update or put the metadata while the object was creating or updating.

I clearly understand..