Nodejs-storage: rsync using nodejs client?

[VikramTiwari]

Is there a way to use rsync with nodejs client? I couldn't find anything in API other than file.exists.

https://cloud.google.com/storage/docs/gsutil/commands/rsync

[stephenplusplus]

We do not have that, but it could be cool. Any interest in sending a PR?

[VikramTiwari]

I would love to do that but I am not sure about the starting point for this. I did some research on my side and here's how I see it being solved:

• Identify all the assets in a directory by walking the path
• generate an array of [{localPath: 'file_path_on_local', hash: 'hash_of_file'}]
• For each file, check if the file exists on GS and file hashes match, if not generate a signed URL while making sure that directory structure is preserved
• Upload all files which get a signedURL

I haven't looked into how gcloud does their rsync but treating it like just another folder (which gcloud seemingly does) would be a much better approach. Thoughts @stephenplusplus?

[stephenplusplus]

That sounds pretty correct to me. Some added flavor to support options:

• Walk the local directory, generate array of paths and hashes
  
  
  
  • (if recurse option was specified, dig recursively)
  
  
• Walk the remote directory, generate array of paths and hashes
  
  
  
  • (if recurse option was specified, dig recursively)
  
  
• Compare results to extract the differences (there's probably a lodash method for this)
• If delete option was specified, delete the remote files that don't exist locally
• For any files that exist locally, but not remotely, upload!
• For any files that exist locally, but the hash didn't match the remote file, upload!

Thanks for your willingness to help!

[jakemmarsh]

@VikramTiwari did you ever get around to implementing this? Looking at doing the same

[VikramTiwari]

Hey @jakemmarsh Sadly no. Since this was a server use case for us, we started packaging gsutil in our container and execute the bash command using nodejs.

Though I would still love to see a solution native to this package (or without it even).

[irontitan76]

I'm posting this here since I think it may help those looking to upload entire folders until an official solution is implemented. The below code walks the specified directory and executes a passed-in function on each file (in this case an upload function), then returns an array with the file paths. The upload function in this case will simulate and "mirror" the local directory in Cloud Storage.

const fs = require('fs').promises; // Experimental API
const path = require('path');
const { Storage } = require('@google-cloud/storage');

const storage = new Storage();
const bucketName = 'BUCKET-NAME';
const build = 'build/'; // Specify local folder
const app = 'app.yaml'; // Specify local file

async function walk(dir, fn) {
  let files = await fs.readdir(dir);
  files = await Promise.all(files.map(async (file) => {
      const filePath = path.join(dir, file);
      const stats = await fs.stat(filePath);

      if (stats.isDirectory()) {
        return walk(filePath, fn);
      } else if(stats.isFile()) {
        if (fn) fn(filePath);
        return filePath;
      }
  }));

  return files.reduce((all, folderContents) => all.concat(folderContents), []);
}

async function upload(file) {
  await storage.bucket(bucketName).upload(file, { destination: file, gzip: true });
};

walk(build, upload);
upload(app);

[AVaksman]

There is an uploadDirectory sample available in the docs (sourcecode)

[VikramTiwari]

Hey @AVaksman - The example is to upload a directory which doesn't solve the request of rsync. rsync would not upload a file in a directory if the file on source machine hasn't been modified since last upload on destination.

[AVaksman]

Perhaps gsutil functionality could be utilized.
gsutil#rsync

[VikramTiwari]

Yes. I am already doing that. But you can't utilize it in scenarios where you can't package a binary (cloud functions).