Nodejs-pubsub: Not working in cloud function

[codyzu]

I am unable to get the client to work in a cloud function. Sometimes messages are never acknowledged (even though ack() is being called) and the cloud function can get "stuck" where no newly published messages are ever fetched until the cloud function is redeployed.

Reading the cloud function docs, background activities should not be started.

Is the node.js pubsub client not compatible with cloud functions because it sends acknowledgments in the background and re-uses connections from a pool? I have a StackOverflow question asking the same question in more detail.

Thanks!

Environment details

• OS: Google Cloud Function
• Node.js version: 6.11.5
• npm version: _not documented by Google_
• @google-cloud/pubsub version: 0.16.2

Steps to reproduce

1. Create a Google Cloud Function using the pubsub client to pull messages from a subscription
2. Deploy the function
3. Publish messages, execute the function, and repeat

[theacodes]

Hi @codyzu. Using pull within cloud functions is not a supported use case, instead, you should use push for your subscriptions. See https://cloud.google.com/functions/docs/calling/pubsub

Closing this, but feel free to comment if you have further questions.

[codyzu]

Thanks @jonparrott for your response!

We use pull because we want PubSub to "queue" messages, that can be serviced in bulk at a regular interval.

In detail, the function is triggered (pushed to) by a PubSub Topic (T1). T1 is published to by cronjob, every 5 minutes (for example). The messages have no interesting payload for the cloud function.

An independent process publishes messages ("jobs") asynchronously to the "queue" PubSub Topic (T2).
Upon executing (triggered by T1), the cloud function pulls from the "queue" PubSub Subscription (S2) to service the queued messages.

Upon discovering that the node.js pubsub client is not compatible with cloud functions, we switched the PubSub service REST API (for dequeuing messages). However, it has its own set of unique challenges.

It seems we are using PubSub in a new way? Do you see a flaw to our design? The advantage is that we can service our asynchronous "jobs" (topic T2) in bulk at regular intervals to save resources.

[theacodes]

/cc @kir-titievsky

I would say that yes, the second part of that is an unusual use case for pubsub. I would suggest maybe using a persistent database to queue up bulk jobs.

[codyzu]

Sorry to be stubborn :stuck_out_tongue: and BIG thanks for your time!

From an architectural standpoint, it is interesting because cloud function never has to access any permanent storage (DB, cloud storage, etc...). All the data to service the jobs (some API calls in our case) is stored in the PubSub message payload.

This adds an interesting decoupling in our system. Also, it allows the cloud function to be scaled without having to manage the distribution of queued jobs between instances (PubSub guarantees the messages are delivered to a single subscriber)

Maybe we made a mistake, but it does not seem obvious why cloud functions cannot be compatible with pubsub pull. We are currently using the REST API's to pull the subscription.

[theacodes]

I mean, I'm not saying you can't, I'm just saying it's a little unusual. (@kir-titievsky can maybe speak a bit more to it).

[kir-titievsky]

@codyzu I think a combination of App Engine & with Task Queues and App Engine Cron would be a much better integrated set of tools for implementing what you've outlined.

That said, here are a couple ways to get this working with Pub/Sub with mild hacks:

• Don't wait to do the work. Have T2 trigger the CF as soon as the jobs arrive. Optionally use really low quota. I assume there is a reason you want to wait. Could you tell us?
• If you must, have your chon job toggle a subscription to T2 between pull and push every 5 minutes. Turn it to push at minute 5n, let all the job get distributed to workers, turn it back to pull at minute 5n+1.

[codyzu]

Thanks @kir-titievsky for your thoughtful response!

I assume there is a reason you want to wait. Could you tell us?

Yes :smile_cat: there are 2.

1. The primary reason is that the jobs are "scheduled". Each one is set to execute sometime between now and 7 days (the maximum retention of a PubSub message).The scheduling is unique per job (defined by user settings). When the cloud function pulls the messages, it looks at the scheduling field in the message payload and determines which messages are ready to be serviced during the current execution cycle. Jobs ready, are serviced and ACK'ed. Jobs not ready, are not ACK'ed and left to be serviced later.
2. The API's we call are rate-limited but support bulk operations, so by waiting 5 minutes (this interval may be tuned), we potentially service multiple jobs in bulk and reduce the number of API calls.

a combination of App Engine & with Task Queues and App Engine Cron would be a much better integrated set of tools

This is because that toolset is better suited to working together?

We chose Kubernetes CronJobs instead of App Engine Cron because we already have a production k8s cluster. Adding a CronJob to k8s seemed quicker, simpler, and added less new maintenance burden given our existing architecture.

We chose PubSub to "queue" our jobs again because we already use it and cloud functions in our architecture. Accepting the 7 day maximum lifetime of messages, PubSub _theoretically_ fit all of our needs: a sort of "time-based" priority queue. Plus the ability to simply add more cloud functions that pull from the same subscription adds a super simple way to scale the number of workers.

On paper, our design seemed innovative and yet elegantly simple. However, given the discussion here and in my two SO posts, here and here, the impression I have is that the community consensus is that we are using PubSub in a way this it is not intended. However, it is not clear _what_ is not intended.

Do you think we would be better off using services from the GAE? Why?

Thanks again for your time and discussion!

[codyzu]

I wrote an article on our blog discussing in more detail _why_ we use PubSub the way we do (and the challenges we encountered). It may be more clear than my description above.

Don't hesitate to add any questions or feedback over there also...