Librdkafka: What are the tradeoffs/downsides in reducing queue.buffering.max.ms?
I found that when I made this number small as compared to the default, performance quadrupled. I realize there is no free lunch however so wanted to better understand what the downside is of doing this. Is it an increase in CPU? (Client side or server side?)
winbatch
All 5 comments
Please read this section of the librdkafka introduction:
https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#high-throughput
It explains the key aspects of throughput and latency.
If your application has special send patterns you might need to configure the two two config properties "batch.num.messages" and "queue.buffering.max.ms".
edenhill
on 28 Dec 2013
Did this answer your question?
edenhill
on 1 Jan 2014
yes and no. Was wondering if it increases cpu on either the client machine
or broker host
On Wed, Jan 1, 2014 at 5:24 PM, Magnus Edenhill [email protected]:
Did this answer your question?
—
Reply to this email directly or view it on GitHubhttps://github.com/edenhill/librdkafka/issues/41#issuecomment-31431655
.
winbatch
on 1 Jan 2014
Using a smaller value for "queue.buffering.max.ms" means less messages will be sent in each produce request, thus the request's overhead will be higher per-message than if more messages are sent.
It will not affect the producer side much, but the broker will only handle one produce request at the time (there are multiple requests in-flight though), so the fewer messages in a request, the lower performance.
But it also depends on your usage pattern; what rate you are producing messages at, and if they are produced in bursts.
You may need to optimize your "batch.num.messages" and "queue.buffering.max.ms" properties according to INTRODUCTION.md. That is:
queue.buffering.max.ms- how long librdkafka will wait forbatch.num.messagesto be produce()d by the application before sending them to the broker in a produce requestbatch.num.messages- the maximum number of messages that will go in a single produce request to the broker.
When either one of these two properties are reached a produce request will be sent.
If you need high throughput: both these properties should be high (default),
if you need low latency: decrease queue.buffering.max.ms to your maximum accepted queue latency
edenhill
on 1 Jan 2014
Thanks for the info, closing
winbatch
on 4 Jan 2014
Related issues
mirroj
·
3Comments
roopabayar
·
5Comments
feng1o
·
3Comments
QueryCity
·
7Comments
lxyscls
·
4Comments
Most helpful comment
Using a smaller value for "queue.buffering.max.ms" means less messages will be sent in each produce request, thus the request's overhead will be higher per-message than if more messages are sent.
It will not affect the producer side much, but the broker will only handle one produce request at the time (there are multiple requests in-flight though), so the fewer messages in a request, the lower performance.
But it also depends on your usage pattern; what rate you are producing messages at, and if they are produced in bursts.
You may need to optimize your "batch.num.messages" and "queue.buffering.max.ms" properties according to INTRODUCTION.md. That is:
queue.buffering.max.ms- how long librdkafka will wait forbatch.num.messagesto be produce()d by the application before sending them to the broker in a produce requestbatch.num.messages- the maximum number of messages that will go in a single produce request to the broker.When either one of these two properties are reached a produce request will be sent.
If you need high throughput: both these properties should be high (default),
if you need low latency: decrease queue.buffering.max.ms to your maximum accepted queue latency