Nodejs-storage: bucket.upload(filePath, options) producing an 404 error with detail-less description

[Idomo]

Well, the title couldn't be clearer, this is one of the detail-less error's description I've ever seen, all I get is "Not Found" (after upgrading to 3.1.0 it also added the headers object, that also doesn't means anything to me), I don't even know what haven't been found (but I can be sure it's not about the file in filePath, 'cause this produced another [detailed] error).

Hope that someone will be able to help me, 'cause I don't even know where to start looking to fix this.

Environment details

• OS: macOS
• Node.js version: 10.12.0
• npm version: 6.10.2
• @google-cloud/storage version: 3.1.0

What I've already tried to do

1. In https://console.cloud.google.com I've added the rule Storage Admin in addition to Owner to the mail in the google-key.json file (that passed to Storage() constructor).
2. Comment out the options, so it will be only bucket.upload(filePath) and may upload the file to the top level directory.
3. Chang storage rules to:

service firebase.storage {
  match /b/{bucket}/o {
      allow read: if true;
      allow delete: if true;
      allow write: if true;
  }
}

and

service firebase.storage {
  match /b/{bucket}/o {
      allow read: if request.auth != null;
      allow delete: if request.auth != null;
      allow write: if request.auth != null;
  }
}

Steps to reproduce

Call

bucket.upload(filePath, {
        destination: destPath
      }).then(() => {
        res.end('Successfully uploaded!');
      }).catch((err) => {
        res.end('Failed to upload: ' + JSON.stringify(err));
      });

and get this error in the catch() calback:

Failed to upload: 
{  
  "code":404,
  "errors":[  
    {  
      "domain":"global",
      "reason":"notFound",
      "message":"Not Found"
    }
  ],
  "response":{  
    "headers":{  
      "alt-svc":"quic=\":443\"; ma=2592000; v=\"46,43,39\"",
      "content-length":"165",
      "content-type":"application/json; charset=UTF-8",
      "date":"Thu, 15 Aug 2019 13:58:50 GMT",
      "server":"UploadServer",
      "vary":"Origin, X-Origin",
      "x-guploader-uploadid":"AEnB2Ur6Hhd5uVLDCSH...A8pw7rD1-g"
    }
  },
  "message":"Not Found"
}

Also, I'm getting some unclear message in the console (if thats related), don't understand from what should I be "careful" and right now I'm using the emulator on my mac, so it's not production:

⚠  Google API requested!
   - URL: "https://www.googleapis.com/oauth2/v4/token"
   - Be careful, this may be a production service.
⚠  Google API requested!
   - URL: "https://www.googleapis.com/upload/storage/v1/b/my-projectf/o?uploadType=multipart&name=path%2to%2Ffile.jpg"
   - Be careful, this may be a production service.

Thanks,
Ido.

[stephenplusplus]

Does the bucket exist that you're trying to upload to? (the my-projectf name from the URL).

Regarding those logs, what emulator are you using?

[Idomo]

Yes, it does exist, I've console.logged bucket before than.
I'm using firebase emulators:start --only functions for the emulator.

[stephenplusplus]

I believe those logs (which aren't ours) are just letting you know that you're actually making changes to real resources, and not just playing in a sandbox.

Can you try running this code outside of the emulator/functions instance to eliminate those variables?

[Idomo]

Getting the same error also with live version (after deploying it to firebase).

[stephenplusplus]

Just confirming, filePath is a path to a local file? And what is the value of destPath?

[Idomo]

filePath is the local file and destPath is the path to upload the file into Firebase Storage (e.g. images/image.jpg).
See also What I've already tried to do.2 in the first comment.

[stephenplusplus]

I have to ask just once more to make sure the Bucket resource exists remotely, as we've seen these vague "Not Found" errors appear under the same circumstance. You mentioned you console.logged bucket, but make sure the remote Bucket resource exists (bucket.exists().then(...), as it cannot be created automatically when you upload a file.

[Idomo]

bucket.exists().then(() => console.log('bucket exists'))
      .catch((err) => console.log('bucket error: ' + err));

Printed bucket exists.

[stephenplusplus]

Could you create this simple script to run locally, not after being deployed . to Firebase?:

const storage = new Storage() // provide necessary credentials
const bucket = storage.bucket('your-bucket')

bucket.upload('local-path.jpg')
  .then(() => console.log('Successfully uploaded!'))
  .catch((err) => console.log('Failed to upload: ' + JSON.stringify(err)))

Also, is the bucket being created at runtime? Is there a chance there's a race condition between the bucket coming to exist and the upload attempt?

[Idomo]

Same error.
The bucket is live and created long time ago from Firebase Console, what I've posted here is the only code that using @google-cloud/storage services.

[stephenplusplus]

I just went through the Firebase console to make a bucket, using the default rules as well as the ones you had, and it worked successfully. I was only able to reproduce the Not Found error when I intentionally made a typo in the bucket name.

What version of @google-cloud/storage are you using?

[Idomo]

Noted this in the first comment, @google-cloud/storage version: 3.1.0
With the test I've ran and printed bucket exists typo is not an option here.

[stephenplusplus]

Could you do what I have done in my last comment? Create a new Firebase bucket and attempt to upload to it?:

'use strict'

const {Storage} = require('@google-cloud/storage')
const storage = new Storage()
const bucket = storage.bucket('new-firebase-bucket-name')

bucket.upload('package.json')
  .then(() => console.log('Upload successful'))
  .catch((err) => console.log(err))

[Idomo]

Getting this (basically the same error) after creating the file with the code you've posted and ran in console node file.js:

{ Error: Not Found
    at Util.parseHttpRespBody (/.../functions/node_modules/@google-cloud/common/build/src/util.js:194:38)
    at Util.handleResp (/.../functions/node_modules/@google-cloud/common/build/src/util.js:135:117)
    at retryRequest (/.../functions/node_modules/@google-cloud/common/build/src/util.js:420:22)
    at onResponse (/.../functions/node_modules/retry-request/index.js:206:7)
    at /.../functions/node_modules/teeny-request/build/src/index.js:193:17
    at process._tickCallback (internal/process/next_tick.js:68:7)
  code: 404,
  errors:
   [ { domain: 'global', reason: 'notFound', message: 'Not Found' } ],
  response:
   PassThrough {
     _readableState:
      ReadableState {
        objectMode: false,
        highWaterMark: 16384,
        buffer: BufferList { head: null, tail: null, length: 0 },
        length: 0,
        pipes: null,
        pipesCount: 0,
        flowing: true,
        ended: true,
        endEmitted: true,
        reading: false,
        sync: false,
        needReadable: false,
        emittedReadable: false,
        readableListening: false,
        resumeScheduled: false,
        emitClose: true,
        destroyed: false,
        defaultEncoding: 'utf8',
        awaitDrain: 0,
        readingMore: false,
        decoder: null,
        encoding: null },
     readable: false,
     _events:
      { prefinish: [Function: prefinish],
        error: [Array],
        data: [Function],
        end: [Function] },
     _eventsCount: 4,
     _maxListeners: undefined,
     _writableState:
      WritableState {
        objectMode: false,
        highWaterMark: 16384,
        finalCalled: false,
        needDrain: false,
        ending: true,
        ended: true,
        finished: true,
        destroyed: false,
        decodeStrings: true,
        defaultEncoding: 'utf8',
        length: 0,
        writing: false,
        corked: 0,
        sync: false,
        bufferProcessing: false,
        onwrite: [Function: bound onwrite],
        writecb: null,
        writelen: 0,
        bufferedRequest: null,
        lastBufferedRequest: null,
        pendingcb: 0,
        prefinished: true,
        errorEmitted: false,
        emitClose: true,
        bufferedRequestCount: 0,
        corkedRequestsFree: [Object] },
     writable: false,
     allowHalfOpen: true,
     _transformState:
      { afterTransform: [Function: bound afterTransform],
        needTransform: false,
        transforming: false,
        writecb: null,
        writechunk: null,
        writeencoding: 'buffer' },
     statusCode: 404,
     statusMessage: 'Not Found',
     request:
      { agent: [Agent],
        headers: [Object],
        href:
         'https://www.googleapis.com/upload/storage/v1/b/my-projectf/o?uploadType=multipart&name=path%2to%2Ffile.jpg' },
     body:
      '{\n "error": {\n  "errors": [\n   {\n    "domain": "global",\n    "reason": "notFound",\n    "message": "Not Found"\n   }\n  ],\n  "code": 404,\n  "message": "Not Found"\n }\n}\n',
     headers:
      { 'alt-svc': 'quic=":443"; ma=2592000; v="46,43,39"',
        'content-length': '165',
        'content-type': 'application/json; charset=UTF-8',
        date: 'Thu, 15 Aug 2019 16:40:36 GMT',
        server: 'UploadServer',
        vary: 'Origin, X-Origin',
        'x-guploader-uploadid':
         'AEnB2UoLqJPPERriH-zc-...-wkmGAsM3Y6g' },
     toJSON: [Function: toJSON] },
  message: 'Not Found' }

[stephenplusplus]

You're probably scrubbing out your bucket name, but I do have a mild concern over what "my-projectf" is that keeps appearing in the URL? Is that the actual bucket name?

[Idomo]

My bad, I thought that bucketName is like the projectId, but I had to append to it .appspot.com.
So so sorry for wasting your time for that, really appreciate that you’ve didn't gave up and insisted that the issue is with the bucketName.
Thank you very much!

[stephenplusplus]

No problem, I'm glad we were both around at the same time to quickly talk it out! Let me know if anything else pops up 🥂

[Christilut]

I just had the same problem. It was because I was under the assumption that the bucket is the top level folder I can see in Firebase. But the bucket is really the project name and everything is inside that one bucket when using Firebase.

So the SDK was looking for my-bucket inside the default root bucket called my-project because I did this:
storageBucket: 'my-project.appspot.com'

and

FirebaseAdmin.storage().bucket('my-bucket')

But in the case of using Firebase, you have to add my-bucket in the filename and leave bucket() empty.

Anyway, maybe the error message can be more clear? Seems like this happens more often, especially to users new to Firebase.

[boknoypatotoy]

I do also have a problem in uploading a local file to bucket ...

bucket .upload("C:UsersbillgatesAppDataLocalTempbill-gates.appspot.comelg0jwdH3YXqqOnjilP1mb1P70D2export.zip", {
destination: users/${uid}/exportData/export.zip,
metadata: { contentType: "application/zip" },
})

that is my code for uploading a zip file from temp folder created by os.tempdir() ... there is a newly created file with 0 byte in the bucket created by the destination option ... but the upload is not working ... no errors ... runs smoothly ... any suggestions?

[prakhar-goel]

My bad, I thought that bucketName is like the projectId, but I had to append to it .appspot.com.
So so sorry for wasting your time for that, really appreciate that you’ve didn't gave up and insisted that the issue is with the bucketName.
Thank you very much!

I had the exact same error and same problem in django. The bucket name should contain ".appspot.com"