Google-cloud-node: BigTable - local instance

[arbesfeld]

Is there a way to specify the host/port of the BigTable instance? We'd like to use https://github.com/spotify/docker-bigtable for local testing.

[stephenplusplus]

@callmehiphop we can probably do this the same way we do with Datastore, with an 'apiEndpoint' option. Want to tack it onto your v2 PR?

[callmehiphop]

Sure!

[callmehiphop]

On second thought I think we should do this post-merge - this feature would have to wait for v2 completion anyways and it would ship around the same time regardless of whether or not we lump them together.

[stephenplusplus]

@lesv is it possible to connect to a Bigtable emulator host through gRPC?

I tried to make a request, and the emulator prints:

[bigtable] 2016/09/02 12:28:41 transport: http2Server.HandleStreams received bogus greeting from client: "\x16\x03\x01\x00\xa6\x01\x00\x00\xa2\x03\x03\x9bp|Mb)z\xf7\xe8\x17\xba3\xbe"
[bigtable] 2016/09/02 12:28:46 transport: http2Server.HandleStreams failed to receive the preface from client: EOF

And output from the script that made the request:

E0902 12:28:41.194034000 123145321807872 ssl_transport_security.c:945] Handshake failed with fatal error SSL_ERROR_SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number.
E0902 12:28:41.194212000 123145321807872 handshake.c:241] Handshake failed with error TSI_PROTOCOL_FAILURE
E0902 12:28:41.194218000 123145321807872 secure_channel_create.c:99] Secure handshake failed with error 1.
E0902 12:28:46.549982000 123145338609664 tcp_client_posix.c:191] failed to connect to 'ipv4:127.0.0.1:8051': timeout occurred

[garye]

@stephenplusplus The emulator only supports plaintext negotiation, try to disable TLS. Maybe this is grpc.ChannelCredentials.createInsecure()?

[stephenplusplus]

Thank you, I forgot about that! I've run into the next error which is returned from gRPC:

Error: unknown service google.bigtable.admin.v2.BigtableInstanceAdmin
      at Function.GrpcService.decorateError_ (node_modules/@google-cloud/common/src/grpc-service.js:494:34)
      at Object.callback (node_modules/@google-cloud/common/src/grpc-service.js:253:35)
      at node_modules/grpc/src/node/src/client.js:420:14

This is trying to create an instance-- are there any methods/operations/services that aren't accessible through the emulator?

[garye]

It doesn't support any instance-related services. So just use any project/instance and start by creating a table.

[arbesfeld]

Hi, I finally got around to giving this a shot and seems to be hanging on this RPC:

const client = bigtable({
  projectId: config.get('gcloud.projectId'),
  zone: config.get('bigtable.zone'),
  maxRetries: 6,
  credentials,
}).instance(config.get('bigtable.instance'));

...

  async init() {
    this.table = await new Promise((resolve, reject) => {
      client.table('events').get({
        autoCreate: true,
        families: [{
          name: 'event',
          rule: {
            versions: 1,
          },
        }],
      }, (err, table) => {
        if (err) return void reject(err);
        resolve(table);
      });
    });

I'm not seeing any logging in the bigtable process:

➜  ~ gcloud beta emulators bigtable start --host-port localhost:8721 --quiet
Executing: /Users/mata/Downloads/google-cloud-sdk/platform/bigtable-emulator/cbtemulator --host=localhost --port=8721
[bigtable] Cloud Bigtable emulator running on 127.0.0.1:8721

[callmehiphop]

Looks like there's a discrepancy between the emulator and the upstream API. When autoCreate is set to true we first attempt to get the object in question and in the event of a 404 we'll attempt to create said object. The emulator appears to be returning a 500 error instead of a 404.

@garye do we know who's in charge of the emulator?

@arbesfeld In the meantime you could try something like

async init() {
  const table = client.table('events');

  this.table = await new Promise((resolve, reject) => {
    table.get(err => {
      if (!err) return resolve(table);

      table.create({ families: [...] }, err => {
        if (err) return void reject(err);
        resolve(table);
      });
    });
  });
}

[garye]

@callmehiphop I am :) The emulator is actually just this:
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/bigtable/bttest/inmem.go

Let me look into changing the error code. So the autoCreate option calls GetTable?

[garye]

and grpc NotFound gets treated as a 404, right?

[callmehiphop]

So the autoCreate option calls GetTable?

Correct! First we call GetTable and if that returns a 404 (grpc 5) then we call CreateTable

and grpc NotFound gets treated as a 404, right?

Yep! Sorry I didn't mention that up front, we map grpc codes to HTTP codes to make it easier for the user.

[garye]

There's a fix in for this: https://github.com/GoogleCloudPlatform/google-cloud-go/commit/1d0367cf8d57ee2decadbf5788971bbfad421964

Will take at least a week to make its way into the gcloud emulator. Let me know if you're up for trying against the go app itself.

[garye]

@callmehiphop @arbesfeld Any issues left connecting to the emulator or are things looking good here?

[arbesfeld]

It works! Thank you