Google-cloud-node: Pubsub feature: get messages count waiting in subscription

[pierre-b]

Hi,

I need to know how many messages are waiting in the subscription, to pop new workers or monitor the health of my system. This is something AWS SQS does too.

Is it in the roadmap actually ?

Thank you

[pierre-b]

FYI: as https://code.google.com/p/cloud-pubsub/issues/detail?id=13 told me, there is a "num_undelivered_messages" endpoint to get that metric.

[ryanseys]

Considering it's "not accurate" I wouldn't use it yet and don't think we should support it in this library until we know it is accurate.

[pierre-b]

This feature could never be accurate because messages are always flowing into the subscription (depending on the publish rate), but it's crucial to know if you need to add more workers to process messages or monitor the health of your workers.

[stephenplusplus]

From https://cloud.google.com/monitoring/api/metrics, they list two interesting ones:

_num_outstanding_messages_: Current number of messages delivered but not yet acknowledged.
_num_undelivered_messages_: Current number of messages pending to be delivered.

Maybe we could open up a generic subscription.getMetrics('num_outstanding_msesages') and anything else from the list: topic.getMetrics('message_sizes'), as these may all be useful.

[stephenplusplus]

Assigned this to @callmehiphop since you had some ideas for how to implement this generically.

[leibale]

what about this one?

[stephenplusplus]

We have a few more things to implement with a higher priority before getting to this one, mostly support for the Prediction and Translate APIs, then promises.

[leibale]

@stephenplusplus if i will create a pull request, you will merge?

[stephenplusplus]

That would be great!

[sahas-]

Great to see you guys working on this already. If its worth, right now I use stackdriver's subscription metrics capabilities to get this information, apparently, stackdriver lags behind and the count is not accurate. Just wanted to add +1 to this request and happy to be a beta tester, if need be. Let me know.

[mpcm]

Adding link to the current related google doc page, for others like me that ended up here looking to learn about msg depth in a sub: https://cloud.google.com/pubsub/docs/ordering#order_of_processed_messages_matters

[stephenplusplus]

@callmehiphop given the large Pub/Sub rewrite that happened since this issue was opened, is this request still relevant / do you have an idea for a reliable implementation?

[callmehiphop]

@stephenplusplus AFAIK the re-write hasn't changed anything in regard to collecting these metrics.

[stephenplusplus]

This issue was moved to googleapis/nodejs-pubsub#8