Google-cloud-dotnet: Pub/Sub concurrency control
We are using the C# client to subscribe to one of our Google Pubsub topics. We used the StartAsync API and we had flood issues - we got more requests then our server can handle. We didn't find a way to back-off the requests using this API.
In the docs, we can see the concurrency control feature, which is implemented only for the Java SDK: https://cloud.google.com/pubsub/docs/pull#concurrency-control
We ended up using the Pull API:
subscriberClient.PullAsync(subscriptionName, returnImmediately: false, maxMessages: 1)
Since then, we can see in the Google Console that the amount of requests to GooglePubSub grew with many errors. I guess it uses a pulling implementation with many requests.
Is there a better way to control the number of processed messages using the C# client?
ZahiC
All 2 comments
Subscriber flow-control is supported by using Settings.FlowControlSettings.
For example, setting FlowControlSettings to new FlowControlSettings(1L, null) will only allow one message to be processed at a time. That is, it'll ensure that the concurrency level of the function you provide to StartAsync is <= 1.
FlowControlSettings defaults to allowing a concurrency level of 10,000, which is why you were seeing such a flood of messages.
chrisdunelm
on 8 Nov 2017
Thanks! We will try that
ZahiC
on 12 Nov 2017
Related issues
AliSheikhTaheri
路
6Comments
giammin
路
6Comments
Talento90
路
6Comments
y-nishine
路
5Comments
gabrielheming
路
7Comments
Most helpful comment
Subscriber flow-control is supported by using
Settings.FlowControlSettings.For example, setting
FlowControlSettingstonew FlowControlSettings(1L, null)will only allow one message to be processed at a time. That is, it'll ensure that the concurrency level of the function you provide toStartAsyncis <= 1.FlowControlSettingsdefaults to allowing a concurrency level of 10,000, which is why you were seeing such a flood of messages.