Google-cloud-go: bigtable: rowkey support only string and it is not compatible with Java HBase driver

[b-2-83]

Hello,

I made a modification in bigtable.go so it can support any rowkey format, not only String.
The representation is hexadecimal.

Check @ https://github.com/reagere/gcloud-golang

Unfortunately it is not compatible with the current way.

Cheers,

[dsymonds]

Your diff doesn't show anything except an added blank line.

But the library doesn't need to support every possible row key format. It takes a string, which can come from anywhere.

[b-2-83]

Hey right,
Thank you for closing this so quickly.

If some of you are still interesting in this mod, take a look here:
https://github.com/reagere/gcloud-golang/commit/bf24f944679f3103bcdc083bb4f3949f32ad5177

We have formatted the keys in byte[] in Java, the Go client is not able to retrieve the keys where some 0 are in the middle, because it is an end of string.

[heldtogether]

I think this is a valid issue.

I've added a bunch of data to BigTable using the Java HBase client (https://github.com/GoogleCloudPlatform/cloud-bigtable-client) but I'm now trying to read the same table using this Go library and no results are being returned.

I believe Java and Go must be encoding row keys differently so no data is returned when I try to do

rr := bigtable.PrefixRange(prefix)
tbl.ReadRows(ctx, rr, func(r bigtable.Row) bool {
    print(r)
    return true
})

If I do a straight scan, I get bytes back.

[garye]

@heldtogether Can you give an example row key?

[heldtogether]

Sure. In Java:

String rowKey = "GB##en##+iPad +Case";

Table table = connection.getTable(TableName.valueOf("table"));

Put put = new Put(Bytes.toBytes(rowKey));
put.addColumn(Bytes.toBytes("i"), Bytes.toBytes("region"), Bytes.toBytes("GB"));
table.put(put);

Then in Go:

prefix := "GB##en##+iPad";

tbl := client.Open("table")

rr := bigtable.PrefixRange(prefix)
err := tbl.ReadRows(ctx, rr, func(r bigtable.Row) bool {
    print(r)
    return true
}, bigtable.RowFilter(bigtable.ColumnFilter("i")))

It seems tbl.ReadRows returns an empty set.

[garye]

Thanks for the example, I'll take a look.

[garye]

@heldtogether I can't seem to reproduce an empty set from tbl.ReadRows given your example row key. It appears to be a normal ascii row key so that's the behavior I would expect. If you do a straight scan and dump the bytes of the problematic row key, do you get something that makes sense?

In the case of non-ascii prefix that you want to scan, can you just create a Go string from a byte array that represents the prefix bytes and scan with that? From my testing it seems to work with row keys consisting of arbitrary byte arrays being populated by HBase/Java code.

There definitely might be a problem lurking here I but I haven't been able to find one yet.

[heldtogether]

@garye sorry you're right, the scan does work as expected. My issue must be somewhere else!

Thanks for checking this out.

[garye]

@heldtogether No problem. Good luck tracking it down. I'll close this for now, open it back up or open an new issue if you need to.

[charith-qubit]

Hi,

It is easy enough to wrap byte slices as strings and use them as row keys for simple read/write operations. However, it appears that filters expect their inputs to be UTF-8 strings.

Eg:

Bad rowkey_regex_filter pattern "J\x1c\xae\xec\xa5\xd0N֏k9H\xc3\x04Vm\x1f\xabq\xc7P\xc5,\x85": error parsing regexp: invalid UTF-8: `®ì¥ÐN֏k9HÃVm«qÇPÅ,…`

Have I missed something obvious? We have a lot of data imported from HBase (byte row keys) residing in Bigtable and it looks like the Go client cannot be used to access those.

Thanks.

[garye]

That error message is from the inmem/fake server (which is also the gcloud emulator). You're right that this server does not support non-UTF-8 regexps because of a limitation in the Go regexp library. This is frustrating for integration test scenarios.

However, this shouldn't impact operations against the real Bigtable service. Can you confirm that that's the case?

[charith-qubit]

Thanks for clarifying. Filters with binary keys do indeed work with the real service. It's unfortunate that the emulator cannot support them as that would simplify development quite a bit.

[garye]

I agree :( I've run into this before so I'd love to find a solution.

FYI, this and other limitations of the emulator are documented here: https://cloud.google.com/bigtable/docs/emulator

[charith-qubit]

I can't view that link because I am not a Googler :-).
I see that the public docs for the emulator do mention the regexp limitation but it didn't occur to me to look there first because I was a bit distracted by the fact that the Go library only supported string row keys.

Sorry for reopening this issue. It can be closed now. Thanks for your help.

[garye]

Heh, sorry updated the link to point to the public docs.