Nodejs-storage: File size limits for getSignedUrl

[jorilallo]

We're using getSignedUrl extensively for client uploads but unfortunately need more controls over what size uploads the user can make. I know S3 has pretty flexible policies for defining file size ranges but haven't found any documentation if this is possible with getSignedUrl function outside content type and expiration. Or is the solution to calculate contentMd5 and use it to limits uploads?

[jkwlui]

Hi @jorilallo, you can use a signed POST policy to limit the size of uploads:

const options = {
  expires: new Date() + 60 * 1000,
  conditions: [
    ["content-length-range", 0, 1024] // Content-Length between 0 to 1024 bytes.
  ]
}
const [response] = file.generateSignedPostPolicyV4(options);

// Then, make a POST request using the client:
// response.url The URL to make request.
// response.fields HTTP Form fields to include in the request, includes the signature and policy limiting the Content-Length

Here's an example of the entire code:
https://github.com/googleapis/nodejs-storage/blob/master/samples/generateV4SignedPolicy.js

[jorilallo]

@jkwlui Thanks for this, now it all makes sense if getSignedUrl is meant to be used for generating read URLs, not upload ones. By the looks of it, I was using older client version which didn't even specify generateSignedPostPolicyV4 so maybe it was added later.

I re-implemented using the other API but I'm unable to get the upload working with fetch and FormData in the client side JS (I can't use a pure form):

const formData = new FormData();
fields.map(field => {
  formData.append(field.key, field.value);
});
formData.append("file", file);
await fetch(url, { method: "POST", body: formData })

The request fails in a 400 and thus a CORS error. I think I seen this in the past and thus my older implementation was using XMLHttpRequest (I never had issues with S3).

My cors.json:

[
  {
    "origin": ["http://localhost:8080"],
    "responseHeader": ["Content-Type", "cache-control", "max-age", "Access-Control-Allow-Origin"],
    "method": ["GET", "HEAD", "PUT", "POST"],
    "maxAgeSeconds": 3600
  }
]

I think it would be great if you could add more about client-side XHR uploads to the documentation as it's pretty common use case. I did quite a bit of research when implementing this a year ago for the first time and mainly had to rely on external blog posts why I probably also stuck with getSignedUrl.

[jorilallo]

I was able to get past the CORS error by using PUT but the request fails on error about missing headers:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>

For additional info, here's what I used to create the signed upload policy:

const [response] = await file.generateSignedPostPolicyV4({
  expires: new Date().getTime() + 10 * 1000,
  conditions: [ { "content-length": size }, { "content-type": contentType }],
});

[flunderpero]

Perhaps this helps - this is how we do it (we enforce the file-size). The key is the x-goog-content-length-range header:

async function get_upload_url({
    file,
    content_type,
    content_md5_base64,
    size,
}: {
    file: Storage.File
    content_type: string
    content_md5_base64: string
    size: number
}) {
    return ((
        await file.getSignedUrl({
            action: "write",
            expires: Date.now() + 5 * 60 * 1000,
            extensionHeaders: {
                "x-goog-content-length-range": `${size},${size}`,
            },
            contentType: content_type,
            contentMd5: content_md5_base64,
        })
    )[0] as any) as string // ... there is a bug in the d.ts
}

Then on upload, we do this:

// "data" is some kind of object we want to upload - a string or a byte array ...
const content_type = "text/html"
const content_md5_base64 = md5(data)
const size = data.length
const upload_url = get_upload_url({
    file: my_file, 
    content_type, 
    content_md5_base64, 
    size
})
axios.put(upload_url, data, {
    headers: {
        "Content-Type": content_type,
        "Content-MD5": content_md5_base64,
        "x-goog-content-length-range": `${size},${size}`,
    },
})

And this is our CORS configuration:

[
  {
    "origin": ["https://...."],
    "responseHeader": [
      "Content-Type",
      "Content-MD5",
      "Content-Disposition",
      "x-goog-content-length-range"
    ],
    "method": ["GET", "PUT"],
    "maxAgeSeconds": 3600
  }
]

[jorilallo]

@flunderpero Thanks so much, x-goog-content-length-range works perfectly for me. I'll close the issue, hopefully this thread will come in use for others.

[lairdm]

Thank you @flunderpero, it took me quite a bit of digging to find this feature request issue. It would be nice if Google documented more clearly somewhere that x-goog-content-length-range is compatible with signurl PUT uploads. Our use case is needing chunked uploads (very large files saturating the user's bandwidth as best we can) but also being able to enforce file size sanity limits (we don't want someone uploading a 4TB file...).

We'll give this a try and hopefully we can have the best of both worlds with that header and chunked PUT uploads. My own wishlist would be for policy documents to be supported in combination with signedurls, it just feels like that final missing linkage. Thanks again.