Google-cloud-node: Project ID required even if running script from Compute Engine

[HighNoonSolutions]

var datastore = require('@google-cloud/datastore')();

The line above used on compute engine (not app engine) will still throw the error below -

[Error: Sorry, we cannot connect to Google Cloud Services without a project ID. You may specify one with an environment variable named "GCL
OUD_PROJECT". See https://googlecloudplatform.github.io/google-cloud-node/#//docs/guides/authentication for a detailed guide on creating an
 authenticated connection.]

Yet in the documentation it says -

If you are running your app on Google Compute Engine, you won't need to worry about supplying connection configuration options to @google-cloud/datastore— we figure that out for you.

That is not the case.

To reproduce create a google Compute engine instance and run a node.js script without a supplied project ID it will fail.

[stephenplusplus]

Thanks for reporting. Which version of @google-cloud/datastore are you running?

[Splaktar]

I'm seeing this today on App Engine Flex.

Error: Sorry, we cannot connect to Google Cloud Services without a project ID. You may specify one with an environment variable named "GCLOUD_PROJECT". See https://googlecloudplatform.github.io/google-cloud-node/#//docs/guides/authentication for a detailed guide on creating an authenticated connection.

But I have an environment variable set for GCLOUD_PROJECT=myproject. This was working fine on January 3rd when I last deployed. But it is blocking me from deploying today.

Note that I'm still on "google-cloud": "0.41.2". Will upgrading to the latest solve this or is it a bug in App Engine Flex?

I opened #1918 to fix the broken link in the error message.

[Splaktar]

I was on gcloud 138.0.0, but I found this:

139.0.0 (2017/01/11)
  Breaking Changes
- (Google App Engine) Flexible deployments now skip files matching `node_modules` and `.git` by default, but no longer skip other hidden files by default. Standard deployments are not changing.

We're using https://github.com/motdotla/dotenv for our ENV variables and we use .env and .env-prod files that were apparently being stripped out of our container by the builder service in 138.0.0.

I tried again with gcloud 139.0.0 and we're able to deploy our app again. Though this doesn't seem like it will resolve the OP's issue unfortunately :(

[michaelfreund]

This also happens on non app or compute engine environments with >= 0.45.1. Downgrading to 0.45.0 resolves the issue.

const Gcloud = require('google-cloud')({
  keyFilename: Config.GCLOUD.KEYFILE
});

@stephenplusplus any ideas?

[stephenplusplus]

Looking into now, sorry for the delay!

[stephenplusplus]

@HighNoonSolutions I haven't been able to reproduce on GCE. I set up a micro box, installed Node, and tried against both [email protected] and @google-cloud/[email protected].

@michaelfreund Using [email protected] locally (non-GAE or GCE) didn't throw for me.

I must be failing to mimic both of your environments in some way, so please share anything else I can do to reproduce. In general, be sure that you're starting from a clean node_modules directory and un- and then re-install google-cloud or any of the sub-modules you might be using, so you're getting the latest copies.

[michaelfreund]

@stephenplusplus completely removed and reinstalled node modules. I still get the same error. If I output the error message from https://github.com/GoogleCloudPlatform/google-cloud-node/blob/b1c6b210106bb064b0662173e2dbcd41ad211f89/packages/common/src/util.js#L355 I can see a Maximum call stack size exceeded running on node v6.9.2.

Taking a look at https://github.com/GoogleCloudPlatform/google-cloud-node/blob/b1c6b210106bb064b0662173e2dbcd41ad211f89/packages/common/src/util.js#L519 I figured out that iteration is running 47933 times.

[arbesfeld]

Hey @michaelfreund, did you find a fix fox this? We're experiencing the same problem

[stephenplusplus]

This may have been resolved with a recent change. Please try upgrading to [email protected].

[arbesfeld]

Unfortunately this didn't fix it. We are using projectID like this:

import makeGcloud from 'google-cloud';

const gcloud = makeGcloud({
  projectId: 'my-project',
});
const bigtable = gcloud.bigtable()

While we are also running in App Engine. Would this be expected to work?

[bjwatson]

@stephenplusplus Is this a release blocking issue? It looks like several users have encountered it.

[stephenplusplus]

This is one where I have to think in environments it's not working, it's because of some stale dependencies. We have released 0.48.0-- could anyone still experiencing this please try rm -rf node_modules where applicable and re-installing this new version?

@bjwatson If there are still problems, I will need someone to help test in GAE (@jmdobry maybe?)

[bjwatson]

@stephenplusplus Will this be a problem with future updates? Or is this a one-time cost?

[stephenplusplus]

What do you mean?

[bjwatson]

Once users run rm -rf node_modules and re-install, should they be done with this issue forever and ever?

[stephenplusplus]

I hope so! I was never able to pin down exactly what was causing this issue, so I have to hope that it was corrected by a related change that shipped in 0.47.0. If it does correct itself by a fresh upgrade, then no further work by the user is necessary.

[bjwatson]

Ok, thanks. I dropped this priority to P1. Let's close it in another week if we don't hear any other complaints about this.

[dan-petty]

Getting this with 0.48.0 on App Engine nodejs runtime.
var gcloud = require('google-cloud')({ projectId: 'my-project-id-here', keyFilename: './keyfile.json' });

Log error when trying to access gcs bucket:

"Error: Sorry, we cannot connect to Cloud Services without a project ID. You may specify one with an environment variable named "GCLOUD_PROJECT". See https://googlecloudplatform.github.io/google-cloud-node/#/docs/guides/authentication for a detailed guide on creating an authenticated connection."

EDIT: Reverting to:

`"gcloud": "^0.37.2"`

resolves issue with no changes to code.

EDIT 2: Also resolved by:

var storage = require('@google-cloud/storage');
var gcs = storage({
  projectId: 'my-project-id-here',
  keyFilename: './keyfile.json'
});

[stephenplusplus]

Thanks @mixmatch. I'm about to kick off a publish-fest. At the end of it, we'll have [email protected], and hopefully you can test for us again.

[stephenplusplus]

The publish fest is over-- I believe everyone should be covered now if you un- and re-install the latest dependencies that you need. Let us know if there are still any issues.

[xyk2]

I am still having the same problems on 0.49.0 after doing a clean uninstall and reinstall. Downgrading back to 0.45.0 resolves this issue.

var google_cloud = require('google-cloud')({
    projectId: 'PROJECT',
    keyFilename: 'KEYFILE.json'
});

var gcs = google_cloud.storage();
var bucket = gcs.bucket('SOURCE-BUCKET');
var dest_bucket = gcs.bucket('DESTINATION-BUCKET');

Strangely the error only occurs on the upload stage to dest_bucket.

[stephenplusplus]

What environment are you running in @xyk2? Also, could you show the code you're using to make the API request that fails?

[xyk2]

Apologies should have included that earlier - Node 6.10.0 on Mac OSX 10.11.6 using restify.

var restify = require('restify');
var server = restify.createServer({
    name: 'ffmpeg-runner'
});

var google_cloud = require('google-cloud')({
    projectId: 'PROJECT-ID',
    keyFilename: 'keys/KEYFILE.json'
});

var gcs = google_cloud.storage();
var dest_bucket = gcs.bucket('BUCKET_NAME');

server.get('/download', download);
server.get('/upload', upload);

function download(req, res, next) {
    dest_bucket.file('sandbox/index.m3u8').download({destination: 'index.m3u8'}, function(err) {
        if(err) return console.log(err);

        res.send("Download successful");
        console.log("Download successful");
    });
}

function upload(req, res, next) {
    dest_bucket.upload('index.m3u8', {destination: 'sandbox/index.m3u8'}, function(err, gFileObj) {
        if(err) return console.log(err);

        res.send('Upload successful');
        console.log('Upload successful');
    });
}

server.listen(8080, function() {
    console.log('%s listening at %s', server.name, server.url);
});

The code where the error occurs is in the restify upload handler. When there is a GET request /download, the file downloads fine from GCS. The upload handler throws Error: Sorry, we cannot connect to Cloud Services without a project ID. The same upload code works when placed outside the Restify handler though, so I may be missing something very stupid...

[stephenplusplus]

Actually, I think that was exactly what I needed to see the problem. We have been iterating over a larger object than I believe we should be. I'm putting a fix through and I think we'll all be happier people after it thanks to you 👍

[stephenplusplus]

PR sent in #2100.