Google-cloud-go: Method not found: google.pubsub.v1.Subscriber/StreamingPull

[johan-lejdung]

Client

PubSub "0.20.0"
Grpc "v1.11.1"

Environment

MacOS with following docker-compose:

version: '2'
services:
   pubsub:
      image: adilsoncarvalho/gcloud-pubsub-emulator
      ports:
        - "8590:8085"

Problem

Can someone please help me figure this out, I feel like I have no idea what is going on. The following code return this error:

FATA[0000] PubSub projects/<<project>>/subscriptions/<<sub>> FAILED to start: rpc error: code = Unimplemented desc = Method not found: google.pubsub.v1.Subscriber/StreamingPull

Code:

func (ps *PubSub) ListenClusterRecieve() {
    ctx := context.Background()

    err := ps.subscription.Receive(ctx, func(ctx context.Context, m *pubsub.Message) {
        msg := &payload.msg{}
        proto.Unmarshal(m.Data, msg)
        if ps.Process(*msg) {
            m.Ack() // Acknowledge that we've consumed the message.
        } else {
            m.Nack()
        }
    })
    if err != nil {
        log.Fatalf("PubSub %s FAILED to start: %v", ps.queueClusterSub, err)
    }
}

I have been looking for ways to somehow deactive GRPc for pubsub, but I'm not sure that's how it works. I have also been debugging deep in the source-code but haven't quite figured out why this happens. It seems to happen after the this: https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/pubsub/subscription.go#L452

Using the same version of both PubSub and grpc I am able to successfully publish messages on topics, it's just the receiving that doesn't work.

[jadekler]

Hey hey @johan-lejdung 🙂🇸🇪 Could you describe how you're installing the library? For instance, are you doing go get -u cloud.google.com/go/..., or perhaps using govendor or godep?

[rogerhub]

Maybe it's just unimplemented in the emulator? https://github.com/googleapis/googleapis/blob/master/google/pubsub/v1/pubsub.proto says StreamingPull is experimental.

[johan-lejdung]

Hi! @jadekler! I am using dep. This is in my lock-file:

[[projects]]
  name = "cloud.google.com/go"
  packages = [
    "compute/metadata",
    "iam",
    "internal/optional",
    "internal/version",
    "pubsub",
    "pubsub/apiv1",
    "pubsub/internal/distribution"
  ]
  revision = "4b98a6370e36d7a85192e7bad08a4ebd82eac2a8"
  version = "v0.20.0"

[[projects]]
  name = "google.golang.org/grpc"
  packages = [
    ".",
    "balancer",
    "balancer/base",
    "balancer/roundrobin",
    "codes",
    "connectivity",
    "credentials",
    "credentials/oauth",
    "encoding",
    "encoding/proto",
    "grpclb/grpc_lb_v1/messages",
    "grpclog",
    "internal",
    "keepalive",
    "metadata",
    "naming",
    "peer",
    "resolver",
    "resolver/dns",
    "resolver/passthrough",
    "stats",
    "status",
    "tap",
    "transport"
  ]
  revision = "1e2570b1b19ade82d8dbb31bba4e65e9f9ef5b34"
  version = "v1.11.1"

[johan-lejdung]

@rogerhub Yeah, that might be it. How would I be able to run against the emulator then though? I just simply want to receive pubsub messages from the emulator, maybe there is a setting I can turn off somewhere?

[jadekler]

@johan-lejdung I'm a bit confused here. Our code relies on google.golang.org/genproto/googleapis/pubsub/v1, but I don't see that in your lock file. Is that added somewhere else?

(i'm also not super familiar with dep, so I am probably missing something here)

[johan-lejdung]

Sorry, I didn't post my entire lockfile, just the parts I thought were relevant. Here it is:

[[projects]]
  branch = "master"
  name = "google.golang.org/genproto"
  packages = [
    "googleapis/api/annotations",
    "googleapis/iam/v1",
    "googleapis/pubsub/v1",
    "googleapis/rpc/status",
    "protobuf/field_mask"
  ]
  revision = "f8c8703595236ae70fdf8789ecb656ea0bcdcf46"

The entire lock-file is around 290 lines.

Do you think there might be a problem with the emulator and not the library?

[jadekler]

Ah right on. AFAICT streaming pull is working in the emulator - it's tested here and implemented here.

Could you check your code against that test above and let me know if it's perhaps a setup issue? Without having seen exactly that error before, it feels very much like you're missing some proto-generated code (hence why I was curious about your deps) or your setup is different than ours.

edit: (I assume the emulator you're describing is pstest and not this emulator? if not, please let me know!)

[johan-lejdung]

Absolutely! Will do today after work :)

Thank you for spending your time with this!
On Tue, 3 Apr 2018 at 21:19, Jean de Klerk [e-mail] wrote:

Ah right on. AFAICT streaming pull is working in the emulator - it's tested
here
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/pubsub/pstest_test.go
and implemented here
https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/pubsub/pstest/fake.go#L528
.

Could you check your code against that test above and let me know if it's
perhaps a setup issue? Without having seen exactly that error before, it
feels very much like you're missing some proto-generated code (hence why I
was curious about your deps) or your setup is different than ours.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-go/issues/953#issuecomment-378365693,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AI1mxFXa0czbkXpDihBlSPb4RAAGtz8qks5tk8tBgaJpZM4TC4k2
.

[johan-lejdung]

Actually currently I use this pubsub emulator adilsoncarvalho/gcloud-pubsub-emulator. But I swapped to it from another wrapper package. Basically it's just the emulator but in a image. So yeah, I am using that emulator.

Just to be sure I just tried running the pubsub emulator you linked and it actually worked.. This is annoying, I've tried to different images with pubsub one of which I successfully has used before in another project (same setup as here).

Could you elaborate on what pstest is? And maybe recommend another image I can use for docker?

[jadekler]

Ahh ok! So, that emulator I linked is our official emulator that's usually used for real-ish-world-tests, benchmarking, and so on. The emulator you linked I've not seen before. pstest is an in-memory emulator written in Go that's usually used for integration tests.

[johan-lejdung]

Okay thank you. I guess there isn't an official image available somewhere?

Edit: For anyone looking for an image. I got this to work: https://hub.docker.com/r/bigtruedata/gcloud-pubsub-emulator/ it looks to be updated regurarly.

Thank you for the help! Sorry that it ended up being a rather silly problem hehe

[jadekler]

Yep! I believe you can use the gcloud docker image and just run the emulator using gcloud.

(you may have to write your own dockerfile to execute the necessary gcloud commands tailored to your usecase, but it should be minimal work)

[jadekler]

Thank you for the help! Sorry that it ended up being a rather silly problem hehe

We're here to help!

[johan-lejdung]

Thanks!

[jba]

Just a clarification, for posterity:

• The official PubSub emulator is described at cloud.google.com/pubsub/docs/emulator and controlled with gcloud beta emulators pubsub. It runs in its own process and can be accessed from any language.
• The pstest package is an experimental PubSub fake for in-process testing in Go.