Google-cloud-node: Could writestream overwrite an exist file?

[lbeeon]

As title, is there any option provide to do so. BTW, the error msg response forbidden, while I upload an exist file. It really confuse me, it takes me a little of time to find what is really cause.

[stephenplusplus]

A writeStream should overwrite an existing fine. Can you provide reproducible code so I can see the issues you're describing?

[lbeeon]

Hi stephenplusplus, here is:

node version: 7.4.0
"@google-cloud/storage": "^0.6.1"

var storage = require('@google-cloud/storage')
var fs = require('fs')

var dst_gcs = storage({keyFilename: 'google_application_credentials', projectId: 'google_project_id'});

var dst_bucket = dst_gcs.bucket("test")

function main(){
  if (fs.existsSync('./test')){
    let file = fs.createReadStream('./test')
    let upload_file = dst_bucket.file('test')
    file.pipe(upload_file.createWriteStream({predefinedAcl:'publicRead'}))
    .on('error', function(err){
      console.log(err);
    }).on('finish', function(){
      console.log('finish');
    })
  }
}

exec twice will get err msg

{ errors: [ { domain: 'global', reason: 'forbidden', message: 'Forbidden' } ],
  code: 403,
  message: 'Forbidden' }

[stephenplusplus]

Thanks for the code! I was able to run this successfully after adjusting the keyFilename argument. Is your keyFilename a full path to a JSON file?

[lbeeon]

Yes, the keyFilename is a full path to a JSON file.

[lbeeon]

Any progress? I mean could you reproduce the error?

[stephenplusplus]

Sorry, I haven't been able to. I don't believe this problem is specific to the write stream method, however, but more as a matter of the type of authentication. What is the contents of the keyfile (blacking out any sensitive information, of course)?

[lbeeon]

keyfile content format, which download from the gcloud IAM.

{
  "type": "",
  "project_id": "",
  "private_key_id": "",
  "private_key": "",
  "client_email": "",
  "client_id": "",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": ""
}

[stephenplusplus]

Sorry for the delay on this. I still haven't been able to reproduce. After the file is uploaded, I can call main() again, and the upload will repeat successfully.

Can you try again with the latest @google-cloud/storage release?

[lbeeon]

Thank you, I already know the reason, It's all about the credentials permission. Also seems the error message have been updated to "Caller does not have storage.objects.delete access to bucket img.meepcloud.com.".

[stephenplusplus]

Oh, that probably helped! Glad things worked out 👍

[jameshhx]

@stephenplusplus FYI, I came across this same error recently while experimenting with GCS and node. The solution I found was to specify resumable: false when creating the write stream. I believe the scenario was as follows.

1. Start upload of assets to GCS (resumable enabled by default)
2. Upload stopped mid-upload
3. Folders intended to contain the partially uploaded files deleted
4. Attempt to upload files again
5. Received 403 error - attempting to delete non-existent partial files / folder?
6. Disable resumable and re upload
7. Successfully uploaded files

I've not tried to recreate this yet, but I thought you might find this information useful.

[mhirsch]

I see something like this when trying to write to a location where a previous file existed, even once the previous file has been deleted from the storage console. It seems specific to this one file path. I haven't been able to replicate it with other files, but I still have this one file location in my google cloud storage that no account can write to.

[lbeeon]

@mhirsch , you are right. To my experience, GCS seems to cache the file that you delete. If you want to rewrite/re-upload an existing file, you need to grant your credentials with delete permission.

[mhirsch]

That makes sense, but I've tried creating multiple service accounts, with the full "Editor" role or with the "Storage Object Admin" role, both of which contain the storage.objects.delete permission, and I still get

{ errors: 
   [ { domain: 'global',
       reason: 'forbidden',
       message: 'Caller does not have storage.objects.delete access to bucket <my bucket>.' } ],
  code: 403,
  message: 'Caller does not have storage.objects.delete access to bucket <my bucket>' }

[lbeeon]

I think you could try grant you permission to "Storage Admin". If it works well, then the errors definitely related to this.

[mhirsch]

The permissions in Storage Admin are a subset of Editor, so I don't see how that would give any more information.

edit: Sorry -- I didn't mean to sound rude there, but it takes a bit to run that test, and you can see in the console here that the Editor role should have all storage-related permissions:
https://console.cloud.google.com/iam-admin/roles/details/roles+editor

[lbeeon]

I know what you means, but I think the IAM sometimes might not work as what we expected. I have meet several times that the credentials already have high level permission like you said "Storage Admin are a subset of Editor", but it just doesn't work. However, after I check the permissions independently then it went back to normal.