Google-cloud-python: pubsub: streaming pull needs keepalive

[pongad]

With default configuration, streaming pull will hang if the stream is idle for a few minutes. This usually happens when the subscription has no more messages. In Go, we set gRPC keepalive to 5 minutes (CL). A similar option should be available in Python, though I'm not sure what the code to pass in the option looks like.

[lukesneeringer]

Does the stream _hang_ or _drop_? If the stream drops, we do not care.

[pongad]

I'm not certain what the differences between "hang" and "drop" are, but I'll describe the symptoms in more detail.

In both Java and Go (and probably others) without keepalive, the stream just never give us new message or notify us of any errors.
In Java, where we give gRPC callbacks (onNext, onError, and onCompleted) to handle events, none of the callbacks are ever called.
In Go, where we call msg, err = stream.Recv() the method never returns, giving us neither messages nor errors.

With keepalive configured, we'll eventually get an error (from experience, either UNAVAILABLE or INTERNAL) and then we can re-create the stream.

[makrusak]

@lukesneeringer: I have the same problem with the latest python pub/sub library.
If there are no new messages on the topic during 2-3 minutes, all will hang and the callback will not be triggered anymore.
I'm searching for solution.

In debug logs I see such lines have been repeating every 0-10 seconds.
After 2-3 minutes without new messages there are no more any "debug" logs and no reaction on new events:

2017-10-19 18:26:51,169 == DEBUG == Snoozing lease management for 4.454791 seconds.
2017-10-19 18:26:55,628 == DEBUG == The current p99 value is 10 seconds.
2017-10-19 18:26:55,629 == DEBUG == Renewing lease for 0 ack IDs.
2017-10-19 18:26:55,629 == DEBUG == Snoozing lease management for 5.356016 seconds.

My code is exactly like here:
Subscriber samples

[makrusak]

@lukesneeringer Still have problem with a hanging connection with latest 0.29.0 pub/sub package. Few minutes after starting the subscriber it stops producing any "debug" logs (in case of absence of messages).
You can view all debug logs here:
https://pastebin.com/wCCCh5Fy