Google-cloud-go: Can't connect to Cloud Datastore Emulator

[philippgille]

I can't connect a simple Go application that works fine with GCP to the Cloud Datastore Emulator.

The Go SDK explicitly mentions that it's supported, and doesn't list any additional requirements or prerequisites (like some --no-legacy flag for gRPC compliance or something similar that I found on the web when looking for a solution). See https://github.com/googleapis/google-cloud-go/blob/f35fa89a7117b680d7eafbded858d35d5d59a30c/datastore/doc.go#L483.
It only links to this document, which also doesn't mention any requirements besides the DATASTORE_EMULATOR_HOST environment variable, which I set.

Other sources of info:

• https://cloud.google.com/sdk/gcloud/reference/beta/emulators/datastore/
• https://cloud.google.com/sdk/gcloud/reference/beta/emulators/datastore/start

Client

Using the datastore sub package:

package main

import (
    "context"
    "os"
    "time"

    "cloud.google.com/go/datastore"
)

func main() {
    err := os.Setenv("DATASTORE_EMULATOR_HOST", "localhost:8081")
    if err != nil {
        panic(err)
    }
    dsClient, err := datastore.NewClient(context.Background(), "foo")
    if err != nil {
        panic(err)
    }

    // Let's use AllocateIDs() as connection test.
    // It works when working with GCP, but not with the emulator.
    tctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    keys := []*datastore.Key{
        &datastore.Key{
            Kind: "bar",
        },
    }
    _, err = dsClient.AllocateIDs(tctx, keys)
    if err != nil {
        panic(err)
    }
}

Describe Your Environment

Local Windows 10 Pro with Docker for Windows.

Running the emulator with Docker:

docker run -it --rm -p 8081:8081 google/cloud-sdk gcloud beta emulators datastore start --no-store-on-disk --project=foo

Starts successfully:

Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator create --project_id=foo /root/.config/gcloud/emulators/datastore
[datastore] Dec 13, 2018 7:28:59 PM com.google.cloud.datastore.emulator.CloudDatastore$CreateAction$1 apply
[datastore] INFO: Provided project_id to Cloud Datastore emulator create command, which is no longer necessary.
[datastore] Created new Cloud Datastore project in '/root/.config/gcloud/emulators/datastore'.
Executing: /usr/lib/google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=localhost --port=8081 --store_on_disk=False --consistency=0.9 --allow_remote_shutdown /root/.config/gcloud/emulators/datastore
[datastore] Dec 13, 2018 7:28:59 PM com.google.cloud.datastore.emulator.CloudDatastore$FakeDatastoreAction$9 apply
[datastore] INFO: Provided --allow_remote_shutdown to start command which is no longer necessary.
[datastore] Dec 13, 2018 7:28:59 PM com.google.cloud.datastore.emulator.impl.LocalDatastoreFileStub <init>
[datastore] INFO: Local Datastore initialized:
[datastore]     Type: High Replication
[datastore]     Storage: In-memory
[datastore] Dec 13, 2018 7:29:00 PM io.gapi.emulators.netty.NettyUtil applyJava7LongHostnameWorkaround
[datastore] INFO: Applied Java 7 long hostname workaround.
[datastore] API endpoint: http://localhost:8081
[datastore] If you are using a library that supports the DATASTORE_EMULATOR_HOST environment variable, run:
[datastore]
[datastore]   export DATASTORE_EMULATOR_HOST=localhost:8081
[datastore]
[datastore] Dev App Server is now running.
[datastore]
[datastore] The previous line was printed for backwards compatibility only.
[datastore] If your tests rely on it to confirm emulator startup,
[datastore] please migrate to the emulator health check endpoint (/). Thank you!

Expected Behavior

Successful call of dsClient.AllocateIDs().

Actual Behavior

panic: rpc error: code = Unavailable desc = retry failed with context deadline exceeded; last error: all SubConns are in TransientFailure, latest connection error: <nil>

goroutine 1 [running]:
main.main()
        C:/Users/John/go/src/temp/main.go:32 +0x211
exit status 2

[enocom]

You need to pass a --host-port flag to the emulator command. Without it, the emulator will listen on localhost only within the container.

Try this:

docker run -it --rm -p 8081:8081 google/cloud-sdk gcloud beta emulators datastore start \
  --no-store-on-disk \
  --project=foo \
  --host-port 0.0.0.0:8081

[philippgille]

Thanks @enocom! That solved the problem.

Does it make sense to add that as a small note in the Docker Hub description? I could create a PR for that.

• https://hub.docker.com/r/google/cloud-sdk/
• https://github.com/GoogleCloudPlatform/cloud-sdk-docker/blob/master/README.md

[enocom]

Glad it worked.

You might open an issue over there. I see it's been a problem before: https://github.com/GoogleCloudPlatform/cloud-sdk-docker/issues/107.