Nodejs-storage: getSignedUrl for multiple files

[rakyesh22]

getSignedUrl works only for single file.
need to work with multiple files, so there is no option to pass and retrieve multiple files signed url.
please suggest the fastest option to create multiple file signedUrl.

[stephenplusplus]

@rakyesh22 sorry for the trouble, but would you be able to provide more context around what you're trying to do?

[rakyesh22]

there is folders in firestore lets say images, excel, json. so now i want to get some files ( a_1.png, a_14.png....a_55.png ) from images folder in single api call. ..
so how can i get all those specified files in single call.
Note: I am using node js as backend language.

[stephenplusplus]

Here's the understanding I'm working from, so let me know if I messed it up: you would like to download multiple files from Firestore into memory, using the @google-cloud/storage module.

so how can i get all those specified files in single call.

I don't believe this is an option that the upstream API offers. You can use bucket.getFiles() to retrieve a reference to multiple File objects, but you would then have to download their contents:

const [files] = await bucket.getFiles({ prefix: 'a_' })
const fileContents = await Promise.all(files.map(file => file.download()))

[rakyesh22]

Thanks for answer...
But my concern is with Dont want getting all files with prefix... only want to get specific files URL's .

 const [files] = await bucket.getFiles({ prefix: 'a_' })  

above code will give me All file links
eg. a_1.png, a_2.png, a_3.png, a_4.png.......a_90.png and so on.. ( there might be a 1000 files )

I just want to get Specific Files link.
eg. a_5.png, a_11.png, a_39.png, a_50.png ( so here i want only 4 files Links in single api )

[jkwlui]

I hope I'm understanding your requirements correctly, but it appears you'd like for the client library to generate pre-signed URLs for specific Cloud Storage files with known file names, not directly downloading the file.

If that's the case, you would use file.getSignedUrl() which involves no remote API call:

const fileNames = ['a_5.png', 'a_11.png', ...];
const bucket = storage.bucket('bucket_name');

for (const fileName of fileNames) {
  const [signedUrl] = await bucket.file(fileName).getSignedUrl({
    version: 'v4',
    expires: new Date('2020-12-31T11:59:59'),
    action: 'read'
  });
  console.log(`The signed URL for ${fileName} is ${signedUrl}`);
}

[rakyesh22]

@jkwlui Thanks buddy .
Thats what i am doing currently, using foreach loop to get all files links.
But getting signed url one by one slow downs the executions,

so i am looking for firebase APi/function, where i can pass whole array of file-names and which returns list of all signedUrl .

[yogeshs31]

Yes I have the same issue for getting multiple files url in one api hit. Please suggest better solution . Because by using foreach loop to get all files links. with signed url one by one slow downs the application,

[sboudouk]

To summarise what people want here I think:

Imagines having images/ path in your Google Cloud Storage bucket. This images path contains like 200 pictures. The bucket is not set to public. And we need our frontend to display all the 200 pictures in an HTML page.

The actual process imply that you have to generate a signedUrl for each of the files present in images/ in your cloud storage. This will slowdown your project a lot, despite where you process this (frontend is the worst, but even backend will slow everything in consequences.)

People are requesting something like bucket.file({prefix : 'images/'} ).getSignedUrl() , that will return an array with signedUrl for every single files in images/.

[stephenplusplus]

@frankyn do you have any thoughts about if and how we would want to support this, or recommend users handle the situation described by @sboudouk? Thanks!

[sboudouk]

I managed to kind of solve my issue, so I'll explain what I did here maybe it will help peoples there.

Since getSignedUrl for multiple files is slow and restraint you to use loops to do it, I only run one getSignedUrl.

I use the Google Cloud Function availaible in the Google Cloud solutions (https://cloud.google.com/functions?hl=fr), and use the hook when a file is stored to my Cloud Storage. When a file is stored, i use getSignedUrl with the v2 protocole to generate a SignedURL that last for X years (I put like 10 years.), then make a request to my backend to store the SignedURL.

My Google Cloud function looks like this:

const {Storage} = require("@google-cloud/storage");
const axios = require("axios");

exports.saveSignedUrlToMongo = async (event, context) => {
  const gcsEvent = event;                                     // GCS TRIGGERED EVENT.
  console.log(`Processing file: ${gcsEvent.name}`);
  const storage = new Storage();
  const [signedUrl] = await  storage.bucket($YOURBUCKETNAME)       // Generate Signed URL for uploaded file
      .file(`${gcsEvent.name}`).getSignedUrl({
        version: 'v2',                                                        // v2 Allow to set long expire timestamps
        action: 'read',                                                       // Read Only
        expires: Date.now() + 3000000 * 60 * 1000,                            // 6 Years from now
  });

  axios.post($YOURBACKENDURL, {      // Post in document our DB Links
    fileName: gcsEvent.name,
    signedMediaUrl: signedUrl,
    contentType: gcsEvent.contentType,
  }).then(response => {
    console.log(response);
    return 'Ok';
  }).catch(error => {
    console.error('Error in axios request:' + error);
    return error;
  })
};

By doing so, you're only generating SignedURL once, and it is fast. I know it's not the best approach since it's not completly resilient, but that's how I solved my problem now, waiting for a better approach.

[frankyn]

Hi folks, apologies for the delay.

The Cloud Storage API doesn't support a prefix based signed URL to access a group of objects.

At the moment the signing process is going to be slow because each signed URL needs to be signed individually either through IAM API or a local service account private key.

The main solution that I'm aware of is what @jkwlui provided using a foreach. I also do like your solution @sboudouk, but definitely only see two options.

I just noticed that there was a performance improvement released last week with the updated IAM signBlob endpoint https://github.com/googleapis/google-auth-library-nodejs/releases/tag/v6.0.0 in PR: https://github.com/googleapis/google-auth-library-nodejs/pull/908. In https://github.com/googleapis/nodejs-common/releases/tag/v3.1.0. This is being updated when this PR is released: https://github.com/googleapis/nodejs-storage/pull/1157.

The performance issue is for the case that you're signing from a location far away from the West Cost data centers. The new IAM SignBlob API does not have this performance hit and may help improve performance when signing a group of blobs, but _it will still take time_.

@jkwlui could you help cut the release?

[stephenplusplus]

@frankyn we have released 5.0.0 👍

[frankyn]

Thanks @stephenplusplus! For folks who were seeing a slow turn around time, this may help improve performance as it did for our Java library but don't expect it to be a solution for all.

We may want to close out this issue. I'll forward this feedback to the Cloud Storage team for now, but don't have a turn around time at this time.

Leaving it open in case folks have feedback.