Google-cloud-go: Bigtable emulator: Intermittently throwing "Multiple responses found for Get"

[gabyreilly]

Hi all,

We are intermittently getting this exception from the Bigtable emulator against our Java tests:

    17/08/08 16:18:33 ERROR hbase.BigtableTable: Encountered exception when executing get.
java.lang.IllegalStateException: Multiple responses found for Get
    at com.google.cloud.bigtable.hbase.BigtableTable.get(BigtableTable.java:237)

This can be reproduced with the following JUnit tests: https://github.com/urbanairship/bigtable-test-multiple-get/blob/master/src/test/java/report/BigtablePaginatorMinimalTest.java

The tests repeatedly exercise these functions against the emulator and the emulator intermittently throws the "Multiple responses found for Get" exception.

    private long compareAndIncrementPage(final String collection, final long currentPage) throws IOException {
        try (Table table = connection.getTable(BigtableSchema.TABLE_APP_TAG)) {
            byte[] rowId = Bytes.toBytes(currentPageKey(collection));
            Get get = new Get(rowId);
            get.addColumn(CF_PAGE, COL_PAGE);
            Result result = table.get(get);
            if (result.size() == 0) {
                return table.incrementColumnValue(rowId, CF_PAGE, COL_PAGE, 1l);
            } else {
                long pageOnDisk = Bytes.toLong(result.list().get(0).getValue());
                if (pageOnDisk <= currentPage) {
                    return table.incrementColumnValue(rowId, CF_PAGE, COL_PAGE, 1l);
                }
                return pageOnDisk;
            }
        }
    }

    public long getCurrentPage(final String collection) throws IOException {
        try (Table table = connection.getTable(BigtableSchema.TABLE_APP_TAG)) {
            Get get = new Get(Bytes.toBytes(currentPageKey(collection)));
            get.addColumn(CF_PAGE, COL_PAGE);
            Result result = table.get(get);
            if (result.size() > 0) {
                return Bytes.toLong(result.list().get(0).getValue());
            }
            return -1L;
        }
    }

    public String currentPageKey(String collection) {
        return collection + ".current_page";
    }

Also note: I have not been able to reproduce this issue while running only one single test function -- I only see the issue when running the test class with both functions (testAddItem() and testAddItem1()) in the same run (See https://github.com/urbanairship/bigtable-test-multiple-get/blob/89aeb47d11516a361b805921b5dfa0cd18d97e56/src/test/java/report/BigtablePaginatorMinimalTest.java#L34-L59)

Thanks for your help!

[garye]

Thanks for test case, will take a look!

[mbrukman]

BTW, superb repro instructions, @gabyreilly! Excellent detective work.

[garye]

@gabyreilly I can't reproduce this on my box with your test case. What platform are you using and what version of the emulator (assuming you run it via gcloud)?

[gabyreilly]

@garye -

When I asked my coworkers to run the version from this test repo, it failed to start due to bad copy/paste from the old private repo, oops :) I pushed changes to https://github.com/urbanairship/bigtable-test-multiple-get

We only see the failures intermittently even with the looping in the test class -- every few runs.

I am running on Mac El Capitan with the following gcloud version:

Google Cloud SDK 166.0.0
beta 2017.03.24
bigtable 
bq 2.0.25
core 2017.08.07
gcloud 
gsutil 4.27
kubectl 

My coworkers on Mac Sierra also see the failure intermittently.

Another coworker on Linux can reproduce with the following specs:

Linux 4.10.0-22-generic #24-Ubuntu SMP Mon May 22 17:43:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Google Cloud SDK 166.0.0
alpha 2017.03.24
beta 2017.03.24
bq 2.0.25
cbt 
core 2017.08.07
gcloud 
gsutil 4.27
kubectl 

Another coworker on linux could not get the test to fail in 25 runs with this version:

gcloud --version
Google Cloud SDK 153.0.0
beta 2017.03.24
bq 2.0.24
core 2017.04.24
gcloud 
gsutil 4.25
kubectl

They then updated the emulator version to Google Cloud SDK 166.0.0 and the tests started failing for them.

Thank you!

[garye]

That's very helpful, thanks a lot!

[garye]

Hi @gabyreilly, I think I have a fix for this. Would you be able to try one of these builds to see if it actually fixes your problem?

https://storage.googleapis.com/cloud-bigtable-emulator/bigtable_darwin_386.tar.gz
https://storage.googleapis.com/cloud-bigtable-emulator/bigtable_linux_386.tar.gz

Thanks again for the test case, it made this process much easier!

[gabyreilly]

Hi @garye I ran my test class (20x) against the new emulator build on Mac & it fixed my issue. Thank you so much!

[garye]

Glad to hear that, thanks for testing it out! I'll move the change through the gcloud release process and close this issue when it's finally released.

[garye]

Fixed in the Go library but still needs a gcloud release.

[garye]

Released in gcloud 170.0.0: https://cloud.google.com/sdk/docs/release-notes