Nodejs-storage: Rename storage objects returns 404 Not Found

[ds185359]

Hi. This feature over here for renaming GCS Objects is not working for me. Is this still active? I keep getting 404 Not found whereas I verified all my resources are present and I'm using correct paths and credentials.

Also, found mistake in documentation - you are using a different variable name in comments guide here and different in actual code over here.

Environment details

• Node.js version: v14
• npm version: v6
• @google-cloud/storage version: ^5.8.5

Steps to reproduce

1. I used this template - here. I'm trying to rename an existing file in same bucket.

Thanks!

[cojenco]

Hi @ds185359, thanks for filing the issue and catching the typo in the code sample. We will fix the typo with a PR shortly. The renaming feature as in the sample is still active. Could you share in the comments the code you are trying to run? Are you running into errors with other HTTP requests or only with move() in attempt to rename an object?

[ds185359]

@cojenco Sure. Adding details below. I'm trying to use the move() to rename a file. But I also tried using copy() and I get same error 404 Not Found.

Problem statement and code snippet:

I'm trying to rename files in Cloud storage with a script that calls google api move(). This is sample template I wrote for initial test:

async function rename(){
    //GCS Data Source
    const bucketName = 'bucket-name';

// Import the Google Cloud client libraries
    const {Storage} = require('@google-cloud/storage');

// Instantiate BigQuery and Storage clients
    let projectId = require("../../credentials-file.json").project_id;
    const storage = new Storage({
        keyFilename: "../../credentials-file.json",
        projectId: projectId
    });

    let destFileName = `test.txt`;  //also tested with filename as just `test`
    await storage.bucket(bucketName).file('some-file').move(destFileName);
}

rename().catch(e => console.log(e));

Thank you for quick response.

[stephenplusplus]

The code worked for my test. I used a file named "testfile" that previously existed in the bucket. After running the code, the file was successfully renamed "testfile2":

'use strict'

async function rename() {
  const {Storage} = require('@google-cloud/storage');
  const storage = new Storage();
  await storage.bucket('my-bucket-name').file('testfile').move('testfile2');
}

rename().catch(e => console.log(e));

[ds185359]

Do you think this can be an issue with permissions? I'm using service account with Storage Admin permissions to create, update, delete storage objects, buckets etc.

[ds185359]

Either ways I shouldn't be receiving a 404 in that case. I see this URL callout in error response
href: 'https://storage.googleapis.com/storage/v1/b/parent-bucket/child-bucket/o/some-file/rewriteTo/b/parent-bucket/child-bucket/o/test.txt?' do you find anything suspicious here?

[stephenplusplus]

That could be the case. Is the bucket owned by the same project that the service account key is generated from?

[ds185359]

Yes its generated from same project I'm trying to access the bucket from.

[stephenplusplus]

parent-bucket/child-bucket is suspicious. GCS doesn't have bucket hierarchies like that, as far as I know. It should just be /b/source-bucket/o/original-file-name/rewriteTo/b/source-bucket/o/new-file-name

[ds185359]

my bad.. it should be parent-bucket/some-folder/some-file

[stephenplusplus]

Could you paste the code you're using, including more realistic strings that match your actual paths provided to storage.bucket() and file() and move()? Feel free to jumble up the actual resource names, but keep in place the folder/file separators.

[stephenplusplus]

I tried with a directory structure, and this way worked:

await storage.bucket('bucket-name').file('dir/testfile').move('dir/testfile2');

[ds185359]

Sure. Following is the code I'm using:

//Import all files
async function rename(){
    //GCS Data Source
    const bucketName = 'bucket-name/folder-name';   //folder-name is in format `TEST.FOLDER`

// Import the Google Cloud client libraries
    const {Storage} = require('@google-cloud/storage');

// Instantiate BigQuery and Storage clients
    let projectId = require("../../credential-file.json").project_id;
    const storage = new Storage({
        keyFilename: "../../credential-file.json",
        projectId: projectId
    });

    let destFileName = `test-2`;
    await storage.bucket(bucketName).file('test-env-data-000000000000').move(destFileName);
}

rename().catch(e => console.log(e));

[ds185359]

@stephenplusplus Yes what you suggested works for me. Using filename as folder/file but that renames and stores my file some-bucket instead of renaming and storing it some-bucket/some-folder.

[stephenplusplus]

Thank you for taking the time to do that. The issue is how bucketName is defined. To get the folder name, pass that into the file() and move() methods. Could you try it with these changes?

//Import all files
async function rename(){
    //GCS Data Source
-   const bucketName = 'bucket-name/folder-name';   //folder-name is in format `TEST.FOLDER`
+   const bucketName = 'bucket-name';   //folder-name is in format `TEST.FOLDER`

// Import the Google Cloud client libraries
    const {Storage} = require('@google-cloud/storage');

// Instantiate BigQuery and Storage clients
    let projectId = require("../../credential-file.json").project_id;
    const storage = new Storage({
        keyFilename: "../../credential-file.json",
        projectId: projectId
    });

-   let destFileName = `test-2`;
+   let destFileName = `folder-name/test-2`;
-   await storage.bucket(bucketName).file('test-env-data-000000000000').move(destFileName);
+   await storage.bucket(bucketName).file('folder-name/test-env-data-000000000000').move(destFileName);
}

rename().catch(e => console.log(e));

[ds185359]

@stephenplusplus Solved it. I had to add folder name in both source and destination file names. Now it's working. Thanks a lot for the help!

[stephenplusplus]

Woo! Glad we got it figured out. Sorry for the confusion!