Google-cloud-go: Firestore: "Unknown desc" error for transaction on Firestore Emulator

[alxmsl]

Client

Firestore client 768132bba1e7dd4ec0934d4a93eec2705030780a

Environment

Google Cloud SDK:

$: gcloud version
Google Cloud SDK 256.0.0
app-engine-go
app-engine-python 1.9.86
beta 2019.05.17
bq 2.0.46
cloud-datastore-emulator 2.1.0
cloud-firestore-emulator 1.6.2
core 2019.07.26
gsutil 4.41

MacBook:

  System Version:   macOS 10.14.5 (18F132)
  Kernel Version:   Darwin 18.6.0

Expected Behavior

I tried to use transactions on Firestore client and Google Cloud Firestore Emulator. And I've got some strange results. Source code is:

    ctx := context.Background()
    client, err := firestore.NewClient(ctx, "project-test")
    if err != nil {
        panic(err)
    }

    print_counter := func() {
        d, err := client.Collection("counters").Doc("a").Get(ctx)
        if err != nil {
            log.Printf("error on get key: %v\n", err)
            os.Exit(-2)
            return
        }
        log.Println(d.Data())
    }

    increment := func(){
        var c int64
        ref := client.Collection("counters").Doc("a")
        err := client.RunTransaction(ctx, func(ctx context.Context, tx *firestore.Transaction) error {
            doc, err := tx.Get(ref)
            if err != nil && status.Code(err) != codes.NotFound {
                return err
            }
            v, err := doc.DataAt("value")
            if err != nil && status.Code(err) != codes.NotFound {
                return err
            }
            c, _ = v.(int64)
            c += 1
            return tx.Set(ref, map[string]interface{}{
                "value": c,
            }, firestore.MergeAll)
        }, firestore.MaxAttempts(3))
        if err != nil {
            log.Printf("error on run transaction: %v\n", err)
            return
        }
    }

    // Sequential run is working fine
    increment()
    increment()
    print_counter()

    wg := sync.WaitGroup{}

    // Sequential run is working fine
    for i := 0; i < 2; i += 1 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            increment()
        }()
        wg.Wait()
    }
    print_counter()

    // Concurrent run is creating error 'Unknown desc'
    for i := 0; i < 3; i += 1 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            increment()
        }()
    }
    wg.Wait()
    print_counter()

Command to run emulator:

$: CLOUDSDK_CORE_PROJECT=project-test gcloud beta emulators firestore start --host-port=127.0.0.1:8081

And I wait to have value=7, but...

Actual Behavior

It is working so unstable. Almost all times it returns unobvious errors Unknown desc. Sometimes it executes concurrent transaction correctly

$: GO111MODULE=off FIRESTORE_EMULATOR_HOST=127.0.0.1:8081 go run main.go
2019/08/05 17:01:48 map[value:2]
2019/08/05 17:01:48 map[value:4]
2019/08/05 17:03:18 error on run transaction: rpc error: code = Unknown desc =
2019/08/05 17:04:18 error on run transaction: rpc error: code = Unknown desc =
2019/08/05 17:04:18 error on run transaction: rpc error: code = Unknown desc =
2019/08/05 17:04:18 map[value:4]
$: GO111MODULE=off FIRESTORE_EMULATOR_HOST=127.0.0.1:8081 go run main.go
2019/08/05 17:09:25 map[value:6]
2019/08/05 17:09:25 map[value:8]
2019/08/05 17:10:25 error on run transaction: rpc error: code = Unknown desc =
2019/08/05 17:11:25 error on run transaction: rpc error: code = Unknown desc =
2019/08/05 17:11:25 map[value:9]

Firestore emulator output is:

$: CLOUDSDK_CORE_PROJECT=project-test gcloud beta emulators firestore start --host-port=127.0.0.1:8081
Executing: /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator start --host=127.0.0.1 --port=8081
[firestore] API endpoint: http://127.0.0.1:8081
[firestore] If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:
[firestore]
[firestore]    export FIRESTORE_EMULATOR_HOST=127.0.0.1:8081
[firestore]
[firestore] Dev App Server is now running.
[firestore]
[firestore] Aug 05, 2019 5:01:48 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Aug 05, 2019 5:03:18 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
[firestore] INFO: operation failed: transaction timeout
[firestore] Aug 05, 2019 5:04:18 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
[firestore] INFO: operation failed: transaction timeout
[firestore] Aug 05, 2019 5:04:18 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
[firestore] INFO: operation failed: transaction timeout
[firestore] Aug 05, 2019 5:09:25 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[firestore] INFO: Detected HTTP/2 connection.
[firestore] Aug 05, 2019 5:10:25 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
[firestore] INFO: operation failed: transaction timeout
[firestore] Aug 05, 2019 5:11:25 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
[firestore] INFO: operation failed: transaction timeout

[jadekler]

Thanks for filing this issue. I strongly suspect that this is an emulator issue, since we haven't had similar reports filed about the actual service. Would you mind re-filing the issue on the firestore issue tracker? https://cloud.google.com/support/docs/issue-trackers

I'm also happy to file it there on your behalf if you'd like, though the former is easier for you to track.

Closing for now: the emulator does not live in this repo. Happy to re-open if this turns out to be a library problem instead of an emulator problem.

[alxmsl]

Thank you for your advice!
Here is link to follow up if somebody has the same problem: https://issuetracker.google.com/138953088

[alxmsl]

Hi, @jadekler . I've got this one comment from Cloud SDK team member: https://issuetracker.google.com/issues/138953088#comment2

Looks like support of the Firestore Emulator is a client-side feature. Would you like to re-open this one issue or should I create feature request?

[jadekler]

Looks like they're only targetting node.js with the emulator. That's a bummer. :/ Unfortunately, there remains not much we (Go client team) can do here. You could file an issue to update the emulator to work for other languages (at the same issue tracker against the firestore team), or you could create your own minimal emulator by implementing FirestoreServer.

The latter may seem daunting but in practice it's fairly straightforward if you're just trying to mock it enough to run a few tests. (as opposed to building a full emulator that replicates the backend)