Nodejs-pubsub: Pubsub emulator with v1 doesn't work

[kbariotis]

Environment details

• OS: MacOS
• Node.js version: 12
• npm version: Latest yarn 1.x
• @google-cloud/pubsub version: Latest

Steps to reproduce

Using the synchronous pull example, the PusSub emulator doesn't seem to be working. It's as the env variable PUBSUB_EMULATOR_HOST is being ignored.

Also,

with the default emulator local host (localhost:8085), if I do:

const subClient = new v1.SubscriberClient({
  apiEndpoint: 'localhost:8085',
});

I am getting the error:

Error: Failed to parse target localhost:8085:443
        at Function.getDefaultAuthority (/node_modules/@grpc/grpc-js/src/resolver-dns.ts:313:11)
        at Object.getDefaultAuthority (/node_modules/@grpc/grpc-js/src/resolver.ts:134:28)
        at new ChannelImplementation (/node_modules/@grpc/grpc-js/src/channel.ts:199:31)
        at new Client (/node_modules/@grpc/grpc-js/src/client.ts:82:30)
        at new ServiceClientImpl (/node_modules/@grpc/grpc-js/src/make-client.ts:104:3)
        at GrpcClient.createStub (/node_modules/google-gax/src/grpc.ts:306:18)
        at processTicksAndRejections (internal/process/task_queues.js:93:5)

[feywind]

@kbariotis Thanks for the bug report! I just recently worked on these samples, and it's possible there's something missing in there that needs to be re-piped-through. I'll take a look.

[feywind]

Yeah, I'm able to see this too. There are errors on both synchronous pull samples with the emulator. There is some higher priority stuff in the queue ahead of this, but I'll be looking to fix it.

[feywind]

Adding this to my look-at-shortly queue.

[remi-bruguier]

Can confirm the same exact behaviour in a similar environment.

[oande]

Any update on this?

[feywind]

@oande Sorry for the delay, I've still got this on my list.

[antoinefBam]

Any update on this? We got the same problem in a similar environment.

This fix used to work but the issue (TypeError: Channel credentials must be a ChannelCredentials object) is happening again since a few days...

[rlucioni]

@feywind this is also preventing my team from working with the Pub/Sub emulator locally, which makes development difficult. The workarounds proposed in https://github.com/googleapis/nodejs-pubsub/issues/346 don't work.

[feywind]

@rlucioni I thought I had a suggestion for you, but digging a bit deeper suggests it didn't fix the issue.

The slightly longer answer to the original question is that the environment variable is being interpreted by the veneer client, so if you use v1 directly, it won't see it. That also seems suboptimal. There's been some talk about normalizing the environment variables in the GCP client libraries on this point, but that's still in progress.

It seems to me that the generated gapic clients should interpret that environment variable, but I think the reason they don't is that it hasn't been standardized yet. 😬

There might also be a workaround here by just pulling the info from a PubSub object. I'm going to test that and get back to you.

[feywind]

Here are some ways to work around this, based on my testing:

The simplest, but probably also the most prone to breakage over time (these aren't the same types, but they do technically contain the same things):

const pubsubClient = new PubSub();
const subClient = new v1.SubscriberClient(pubsubClient.options);

That will actually pick up the emulator environment variable in the normal way, since it's just using the PubSub class' init.

The specific members could be copied like so, to be a bit less breakable:

const pubsubClient = new PubSub();
const subClient = new v1.SubscriberClient({
  apiEndpoint: pubsubClient.options.apiEndpoint,
  port: pubsubClient.options.port,
  sslCreds: pubsubClient.options.sslCreds
});

You could also manually init the options, though I understand that wasn't working for some:

const grpc = require('@grpc/grpc-js');

const subClient = new v1.SubscriberClient({
  apiEndpoint: 'localhost',
  port: '8085',
  sslCreds: grpc.credentials.createInsecure()
});

(And of course requires app logic, which may not be great.)

I also feel obligated to note that digging into PubSub.options isn't really a public interface, so that could change whenever. I think probably the good long term solution for this is still the unified environment variables plus gapic client support for it. If it looks like the timeline is long on that, then I suppose another option would be to expose a "give me a client" method on PubSub.

Let me know if anything in there seems like it works, or if you have any ideas about making an interface that works better for you.

[rlucioni]

Passing options from an instance of the PubSub client worked for us! Thank you!

[sdcaulley]

I am not getting any errors using the example of passing options from a PubSub instance. But it just spins until a timeout is reached.

Any idea or suggestions?

[bluedot74]

I am not getting any errors using the example of passing options from a PubSub instance. But it just spins until a timeout is reached.

Any idea or suggestions?

@sdcaulley

I had this exact problem and finally got it working. The issue for me was because the option attribute is servicePath on the pubSub client, not apiEndpoint.

So this worked for me:

const pubSub = new PubSub({
    projectId
});
const options = {
  apiEndpoint: pubSub.options.servicePath,
  port: pubSub.options.port,
  sslCreds: pubSub.options.sslCreds
};
const subClient = new v1.SubscriberClient(options);

Thanks for the solution, @feywind !

[feywind]

apiEndpoint should theoretically work on the PubSub object as well as v1 clients, so I'm not completely sure why it wasn't working.

Yeah, I can't say the workaround of pulling options out of the PubSub object is a good long term practice, but I'm going to make a new issue that's targeted at just figuring out what our official way forward is for v1 and alternate endpoints (and close this one). For now I'd say to consider it a temporary workaround until there's something more structured in place.