Google-cloud-ruby: High number of threads created by GCP error reporting gem

[ce07c3]

We use GCP Error Reporting, GCP Logging and GCP Pub/Sub. Our job processing backend based on Pub/Sub would frequently freeze. I "augmented" Thread to understand where threads were being created. In the output, it showed that the calls originated in the GCP Error Reporting gem. The issue was reproducible both locally on Mac OS X as well as on Heroku.

1) Is this expected when there are a lot of errors to report?

Another two questions, if I may ask here:

2) How does the GCP Pub/Sub library rate limit its message stream? E.g. say the callback thread processor doesn't catch up with the rate of incoming messages.
3) If I understood correctly, the stop method of the subscriber is spawning threads; this may not work reliably during SIGTERMs and other signals.

[ce07c3]

I see there's this one: https://github.com/googleapis/google-cloud-ruby/issues/3679. So 1. is fixed in new versions of the gem.

[dazuma]

@ce07c3 Can you verify the fix for (1) in recent versions? (The latest version is 0.32.0.) I ask because I was just looking at the code while tracking down a different issue, and I suspect there might be another reason for large numbers of threads to be created.

[ce07c3]

You’re right, I can still see a high number of threads.

[dazuma]

Can you open separate issues for your other (pubsub-related) questions? It'll make it easier to keep track of what will probably be separate conversations about separate issues. Thanks!

[blowmage]

I had this drafted but unsent. Apologies.

How does the GCP Pub/Sub library rate limit its message stream?

Pub/Sub Subscriber will pull as many messages as will fit in the inventory. The default is 1,000 messages. Once the inventory is full, it will stop pulling messages until the inventory is dropped to around half of the limit, then it will pull some more.

The message is removed from inventory once the message is acked, nacked, or the callback completes. If you are performing additional work in the callback, then you may want to ack the message after the additional work.

[dazuma]

It looks like the middleware is not re-using the global AsyncWriter, but creating a new one for each middleware instance.

[ce07c3]

Pub/Sub Subscriber will pull as many messages as will fit in the inventory. The default is 1,000 messages. Once the inventory is full, it will stop pulling messages until the inventory is dropped to around half of the limit, then it will pull some more.

The message is removed from inventory once the message is acked, nacked, or the callback completes. If you are performing additional work in the callback, then you may want to ack the message after the additional work.

@blowmage thanks! I assume the threshold is the 0.8 multiplication where it should resume streaming. I moved a related question to #4152. :)

[dazuma]

google-cloud-error_reporting 0.32.1 includes the potential fix for thread count.

[ce07c3]

Thanks!