Google-cloud-node: "Could not load the default credentials" when using PUBSUB_EMULATOR_HOST

[ctavan]

Environment details

• OS: Debian Jessie
• Node.js version: 8.4.0
• npm version: yarn 0.25.7
• google-cloud-node/pubsub version: 0.14.1

Steps to reproduce

I am trying to use gcloud beta emulators pubsub locally for development. This used to work fine with google-cloud/[email protected] and before, but since 0.14 I get authentication errors, see below.

Try any operation that sends requests to the api:

const pubsub = require('@google-cloud/pubsub')();

console.log(`Emulator host: ${process.env.PUBSUB_EMULATOR_HOST}`);

pubsub.getTopics(function(err, topics) {
  console.log(err, topics);
});

Output:

$ node test.js

Emulator host: pubsub:8283
Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9) undefined

With @google-cloud/[email protected] it was easily possible to run against a local pubsub emulator without any credentials or whatsoever.

Is this intended behavior? What's the new recommended way of testing against local emulators? To me it feels unexpected to require credentials for running against the local emulator. Also what credentials would that be?!

I would rather expect that running locally against the emulator behaves the same as running on GCE, where all authentication happens automagically.

[ctavan]

Quick update: If I "short-circuit" the _checkIsGce() from google-auth-library to always return true (i.e. skipping the actual test and just pretending that I'm on GCE) everything works as expected with the local emulator.

https://github.com/google/google-auth-library-nodejs/blob/0a810e45f668e57ae99af478fdef6d8853e1b5b6/ts/lib/auth/googleauth.ts#L237

[callmehiphop]

@ctavan sorry for the inconvenience, we made some core changes to PubSub between 0.13 and 0.14 that seem to have introduced this bug. I've opened a PR (#2617) that I believe will fix the issue you are seeing. Would you be able to help us out and test to see if it resolves the issue on your end?

[ctavan]

@callmehiphop thanks for looking into this. I have just tested with your PR: It does indeed help for the toy-example I gave in the issue, but I still get the same error (Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.) with my actual application.

I will dig deeper into this issue and provide more information tomorrow.

[ctavan]

@callmehiphop the remaining issue I had with my local code was that I was still specifying a projectId which is of course nonsense when using the emulator. Everything regarding PubSub now works fine with your patch!

I had to make sure that in my development environment neither a GCLOUD_PROJECT environment variable was set, nor that I passed a projectId into the PubSub() constructor.

However it now seems that the Datastore client doesn't play nice yet when no GCLOUD_PROJECT or projectId parameter is specified. I now get:

Error: "{{projectId}}" is an invalid project id.
    at /node_modules/grpc/src/node/src/client.js:434:17

When trying to access datastore. I believe a similar patch that you've added to the PubSub client also needs to be added to the Datastore client.

[ctavan]

@callmehiphop seems like I was a bit too fast. While topic creation/publishing works fine now, subscriptions still don't work with your PR.

Here's another minimal example that now includes subscriptions. Please note that the createTopic call works fine now, but the subscription still emits error events:

const pubsub = require('@google-cloud/pubsub')();

console.log(`Emulator host: ${process.env.PUBSUB_EMULATOR_HOST}`);

pubsub.createTopic(`test-topic-${Math.floor(Math.random() * 1000)}`, function(err, topic) {
  console.log('createTopic', err, topic.name);

  const subscription = topic.subscription('test-subscription');
  console.log('topic.subscription', subscription.name);
  subscription.on('message', message => console.log('onMessage', message));
  subscription.on('error', err => console.log('onError', err));
});

Output when run with the PubSub client from #2617:

Emulator host: pubsub:8283
createTopic null projects/{{projectId}}/topics/test-topic-807
topic.subscription projects/{{projectId}}/subscriptions/test-subscription
onError Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
onError Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
onError Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
onError Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9)
onError Error: Could not load the default credentials. Browse to https://developers.google.com/accounts/docs/application-default-credentials for more information.
    at /node_modules/google-auth-library/lib/auth/googleauth.js:316:21
    at /node_modules/google-auth-library/lib/auth/googleauth.js:346:7
    at Request._callback (/node_modules/google-auth-library/lib/transporters.js:70:30)
    at self.callback (/node_modules/request/request.js:188:22)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at Request.onRequestError (/node_modules/request/request.js:884:8)
    at emitOne (events.js:115:13)
    at ClientRequest.emit (events.js:210:7)
    at Socket.socketErrorListener (_http_client.js:401:9)

[callmehiphop]

Thanks for all the feedback @ctavan, I think I should have this all wrapped up by this afternoon!

[callmehiphop]

I've released [email protected] which should resolve this issue. Thanks again for all the help!

[ctavan]

@callmehiphop thanks for the release. It does indeed solve the problems reported in this issue!

Unfortunately 0.14.2 is still not usable for me since message acknowledgement doesn't seem to work as expected. I've reported all details in #2621.

[alizaidi606]

I am having this issue in latest version 0.27
Here are the logs from pubsub emulator start command

Executing: cmd /c C:\Users\<user-name>\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\pubsub-emulator\bin\cloud-pubsub-emulator.bat --host=localhost --port=8085
[pubsub] This is the Google Pub/Sub fake.
[pubsub] Implementation may be incomplete or differ from the real system.
[pubsub] Apr 10, 2019 11:57:47 AM com.google.cloud.pubsub.testing.v1.Main main
[pubsub] INFO: IAM integration is disabled. IAM policy methods and ACL checks are not supported
[pubsub] Apr 10, 2019 11:57:49 AM io.gapi.emulators.netty.NettyUtil applyJava7LongHostnameWorkaround
[pubsub] INFO: Unable to apply Java 7 long hostname workaround.
[pubsub] Apr 10, 2019 11:57:50 AM com.google.cloud.pubsub.testing.v1.Main main
[pubsub] INFO: Server started, listening on 8085

and this is the error I get when publishing a message to a specific topic

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.

[JustinBeckwith]

@alizaidi606 if you're still having issues, would you mind creating a new issue?

[oreporan]

Any updates on this? Still happening on docker node:12 with pubsub client version "@google-cloud/pubsub": "^0.28.1",

[JustinBeckwith]

@oreporan if you're still having issues, would you mind creating a new issue?