Google-cloud-python: Pub/Sub: emulator push to local endpoint failing

[anguillanneuf]

Using the Pub/Sub emulator to push to a local endpoint used to work. But it seems to have stopped working now. Here are the steps to reproduce.

Environment details

1. MacOS
2. Python 3.7.3
3. google-cloud-pubsub==0.41.0

Steps to reproduce

1. git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git, then create a virtualenv and install the necessary packages from pip install -r pubsub/cloud-client/requirements.txt.
2. Start a Pub/Sub emulator in one terminal where the virtualenv is activated:gcloud beta emulator pubsub start.
3. In a separate terminal where the virtualenv is also activated, export the appropriate environment variables. export PUBSUB_PROJECT_ID=my-project-id, then $(gcloud beta emulators pubsub env-init). The latter ensures that PUBSUB_EMULATOR_HOST takes the correct hostname and port number.
4. cd pubsub/cloud-client, then test out the emulator by creating a topic. python publisher.py my-project-id topics create my-topic.

Describe the problem

After successfully creating a push subscription liu that points to http://localhost:3000/apples, I published some messages to the topic that liu is tied to. When I refreshed the page, I got 304 errors, telling me that the web resource had not been modified. In other words, the local endpoint did not receive any of the messages. Normal pull subscription still works though.

[sduskis]

This seems like an emulator problem, not a client issue. Let's discuss this internally.

[anguillanneuf]

@sduskis I just check the gcloud release notes, but didn't see anything for gcloud beta emulators pubsub specifically.

[gronnbeck]

I have the same problem when running tests against Pubsub Push Subscriptions. I get a Java stacktrace from the emulator. It seems like something is missing on its classpath. Hope the stacktrace helps:

[pubsub] Exception in thread "pool-1-thread-2" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter
[pubsub]    at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.toJsonStringForPush(HttpEndpointPusher.java:100)
[pubsub]    at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.push(HttpEndpointPusher.java:53)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.push(PushLoop.java:135)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:116)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73)
[pubsub]    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[pubsub]    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[pubsub]    at java.base/java.lang.Thread.run(Thread.java:834)
[pubsub] Caused by: java.lang.ClassNotFoundException: javax.xml.bind.DatatypeConverter
[pubsub]    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
[pubsub]    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
[pubsub]    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
[pubsub]    ... 8 more
[pubsub] Exception in thread "pool-1-thread-1" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter
[pubsub]    at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.toJsonStringForPush(HttpEndpointPusher.java:100)
[pubsub]    at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.push(HttpEndpointPusher.java:53)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.push(PushLoop.java:135)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:116)
[pubsub]    at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73)
[pubsub]    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[pubsub]    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[pubsub]    at java.base/java.lang.Thread.run(Thread.java:834)

[clemens-smartparking]

Have the same issue with the emulator client using nodejs. @sduskis any update on this issue?

[car12o]

@sduskis I agree with you seems like an emulator problem, is the emulator code open source? If so could you please provide me the repo?

I have exatly the same stacktrace as @gronnbeck

pubsub-server    | [pubsub] INFO: Detected HTTP/2 connection.
pubsub-server    | [pubsub] Exception in thread "pool-1-thread-1" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter
pubsub-server    | [pubsub]     at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.toJsonStringForPush(HttpEndpointPusher.java:99)
pubsub-server    | [pubsub]     at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.push(HttpEndpointPusher.java:52)
pubsub-server    | [pubsub]     at com.google.cloud.pubsub.testing.v1.PushLoop.push(PushLoop.java:133)
pubsub-server    | [pubsub]     at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:114)
pubsub-server    | [pubsub]     at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:71)
pubsub-server    | [pubsub]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
pubsub-server    | [pubsub]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
pubsub-server    | [pubsub]     at java.base/java.lang.Thread.run(Thread.java:830)
pubsub-server    | [pubsub] Caused by: java.lang.ClassNotFoundException: javax.xml.bind.DatatypeConverter
pubsub-server    | [pubsub]     at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
pubsub-server    | [pubsub]     at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
pubsub-server    | [pubsub]     at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
pubsub-server    | [pubsub]     ... 8 more

[tobioboye]

Hi folks!

Any updates on the Java class path issues with the PubSub emulator? When I create a push subscription using the PubSub Python Example Subscriber, I get a "success response below" but get a Java error stack trace.

```$ python3 ./subscriber.py my-local-project create-push "process_unit_of_work" "sub_process_unit_of_work" http://localhost:8080

Push subscription created: name: "projects/my-local-project/subscriptions/sub_process_unit_of_work"
topic: "projects/my-local-project/topics/process_unit_of_work"
push_config {
push_endpoint: "http://localhost:8080"
}
ack_deadline_seconds: 10
message_retention_duration {
seconds: 604800
}

Endpoint for subscription is: http://localhost:8080
```

Error Stack trace:
[pubsub] Aug 28, 2019 2:12:32 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete [pubsub] INFO: Adding handler(s) to newly registered Channel. [pubsub] Aug 28, 2019 2:12:32 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead [pubsub] INFO: Detected HTTP/2 connection. [pubsub] Exception in thread "pool-1-thread-25" java.lang.NoClassDefFoundError: java/time/Duration [pubsub] at com.google.cloud.pubsub.testing.v1.MessageBacklog.pull(MessageBacklog.java:92) [pubsub] at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:111) [pubsub] at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73) [pubsub] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) [pubsub] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) [pubsub] at java.lang.Thread.run(Thread.java:748) [pubsub] Exception in thread "pool-1-thread-26" java.lang.NoClassDefFoundError: java/time/Duration [pubsub] at com.google.cloud.pubsub.testing.v1.MessageBacklog.pull(MessageBacklog.java:92) [pubsub] at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:111) [pubsub] at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73) [pubsub] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) [pubsub] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) [pubsub] at java.lang.Thread.run(Thread.java:748)

[tseaver]

@tobioboye The pubsub emulator isn't managed from this repository, which means we aren't really set to support it. @anguillanneuf Can you suggest a place to report such issues?

[tobioboye]

Thanks @tseaver. I'll await a response about the appropriate place to report the issue I am seeing

[arlobryer]

Hi @tseaver, @tobioboye, @anguillanneuf - just wondering if there's any update on this issue? I've just encountered the same problem and stumbled across this thread trying to figure out what's going on.
Also saw this which seems like the same issue - https://groups.google.com/forum/#!msg/cloud-pubsub-discuss/6SHdoTn16Po/8a09QaGvDwAJ (I see @anguillanneuf is the last person to comment there).

[anguillanneuf]

@arlobryer @tobioboye @gronnbeck @clemens-smartparking @car12o Things are moving finally! Please expect an updated emulator in the 265.0.0 release of gcloud sdk on October 1.

Thanks for all your patience!