Google-cloud-ruby: Bucket#signed_url for resumable uploads

[quartzmo]

@pugsiman Thanks for opening this issue, and sorry for the delay in responding. We'll look into this today for you.

[quartzmo]

Sorry for the delay. I found some additional background info:

• Using signed URLs with resumable uploads
• JSON API - Performing a Resumable Upload
  Resumable Uploads with the XML API
• createResumableUpload in google-cloud-node/storage
• stephenplusplus/gcs-resumable-upload

I'm wondering if you can use Bucket#signed_url with method: "PUT" (and other options as needed) to initiate the sequence explained in JSON API - Performing a Resumable Upload Resumable Uploads with the XML API, then handle the rest of the sequence yourself? That's what you want to do, right? Have you tried it? Sorry in advance if I'm misunderstanding.

Update: Below, @frankyn identified Resumable Uploads with the XML API as the best instructions for resumable uploads with Google Cloud Storage. Be sure to reference these instructions rather than the JSON API instructions linked above. (@frankyn noted also that: "Signed URLs only work for the XML API.")

[quartzmo]

@stephenplusplus I'm wondering if you might be able to take a quick look at what we have now, and perhaps comment on what we might consider adding? We have Bucket#signed_url (which delegates to File::Signer#signed_url) and Bucket#create_file.

[stephenplusplus]

This sounds like something we recently implemented in Node.js, and were caught off guard by that it even existed (as such, advanced apologies that I'm not as good of a resource as @frankyn might be).

Here's the issue we got: whole thing or just the important part, and the PR: https://github.com/googleapis/nodejs-storage/pull/129.

Hopefully we're in the right neighborhood of what @pugsiman is looking for.

[quartzmo]

@stephenplusplus Thank you!

@frankyn Do you have any suggestions regarding this request?

[frankyn]

Hi all,
Clarifying, storage.googleapis.com is the XML API endpoint for GCS. I'd recommend following documentation here. Signed URLs only work for the XML API.

IIUC @pugsiman you want to generate a signedURL with POST method. Then forward that to the UI then use a Javascript AJAX request to perform the PUT.

Unless I have misunderstood, the Ruby client doesn't need to be changed to generate a signed url for resumable uploads: (_However, it would be nice to have a simpler way of generating a resumable upload session URL as it's a pain._)

require "google/cloud/storage"
require "net/http"

storage = Google::Cloud::Storage.new
bucket = storage.bucket "bucket-name"
file = bucket.file "test-file", skip_lookup: true # file may not exist.
signed_url = file.signed_url(method: "POST",
                             content_type: "image/jpeg",
                             headers: {'x-goog-resumable':'start'})

# Ruby example of getting the resumable session.
# Create the HTTP object
uri = URI.parse(signed_url)
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
# https.set_debug_output $stderr # debugging purposes only

# Prepare POST request to get resumable session
headers = {'x-goog-resumable': 'start'}
request = Net::HTTP::Post.new(uri.request_uri, headers)
request.content_type = "image/jpeg"

# Send POST request to get resumable session URL.
response = https.request(request)

# Process response
resumable_session_url =  response["location"] # You can continue onto step 3 with follow-up PUT requests.

# Continue with PUTs

[frankyn]

Full example with no interruption recovery. Example reads from STDIN and can be ran by using:

cat file.ext | ruby stream_upload.rb "bucket-name" "file-name" "mime-type"

stream_upload.rb

Created a Gist based on feedback from @quartzmo.

[quartzmo]

@frankyn Thank you for these solutions, and perhaps most critically, for identifying Resumable Uploads with the XML API as the best instructions for resumable uploads with Google Cloud Storage. (@frankyn noted also that: "Signed URLs only work for the XML API.") I will update my earlier comment on background info.

[quartzmo]

@pugsiman Can you try @frankyn's solution and let us know if it works for you?

[pugsiman]

@quartzmo @frankyn @stephenplusplus Thank you everyone for helping! I was sick in the last week so I didn't get a chance to try frankyn's solution, I'll update here my results as soon as I have the chance.

[pugsiman]

@frankyn Mmm weird, it seems to fail for me after I PUT into the received session url.

"<?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>PUT

text/csv
1523750779
/csv_import_files/resumable_test.csv</StringToSign></Error>"

As far as I'm concerned my signature is correct. Is there anything else besides content-type? As a test I didn't bother sending chunks, but a whole small csv instead (888 bytes), could the size be an issue?

For reference, here's how my request looks like as cURL:

curl <session url> -X PUT -H 'origin: http://localhost:8000' -H 'content-type: text/csv' -H 'authority: storage.googleapis.com' -H 'x-client-data: xxx' -H 'content-length: 888' --compressed

Also, just want to point out that I had to change my CORS config in the bucket to allow the x-goog-resumable header in requests, possibly worth mentioning in your snippet.

[frankyn]

Thanks for getting back to us!

Are you using the script or did you try implementing your own? Could you
share what you have?

If you are using the script it may be a mismatch between headers and query
parameters with the signed url. The request needs to match the signed url
form.

On Sat, Apr 14, 2018, 4:23 PM Aviv Levinsky [e-mail]
wrote:

Mmm weird, it seems to fail for me after I PUT into the received session
url.

"SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.PUT

text/csv
1523750779
/csv_import_files/resumable_test.csv"

As far as I'm concerned my signature is correct. Is there anything else
besides content-type? As a test I didn't bother sending chunks, but a small
csv instead (888 bytes), could the size be an issue?

Also, just to point out that I had to change my CORS config in the bucket
to allow the x-goog-resumable header in requests.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-ruby/issues/2024#issuecomment-381367737,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB0bwCdtJfmVkRnF9KJaccuqOLurl33yks5tooT1gaJpZM4S8jQS
.

[quartzmo]

@pugsiman Please let us know what we still need to do to close this issue, thanks!

[pugsiman]

Well I ran it again and it worked, I have no idea what was changed so forgive me. My JS code looks something like this:

    var xhr = new XMLHttpRequest();
    document.querySelector('input[type="submit"]').addEventListener('click', function() {
        let signedURL = <some result from my server>
        xhr.open('POST', signedURL, true)
        xhr.setRequestHeader('Content-Type', 'csv/text')
        xhr.setRequestHeader('x-goog-resumable', 'start')

        xhr.onload = function() {
            if (this.status >= 200 && this.status < 400) {
                uploadCsvToGcloud(this.getResponseHeader('location'))
            }
        }

        xhr.send();
    })

    function uploadCsvToGcloud(responseURL) {
        let csv = document.querySelector('input[type="file"]').files[0]

        xhr.open('PUT', responseURL, true)
        xhr.onload = function() {
            console.log('uplodaed to bucket')
            doSomething()
        }

        xhr.send(csv)
    }

Thank you every much for the help! @quartzmo @frankyn @stephenplusplus