Librdkafka: Does "queue.buffering.max.ms == 0" mean that there is no batch?

[lxyscls]

Description

Four devices work as Kafka clients. The rate is about 1000/pps per device, and every packet's size is about 500 bytes. Two of them have packet loss. "Local: Message timed out" are printed in the delivery callback.

I have found that there are many data buffered in the TCP Send-Q at the device, and there are no data buffered in the Kafka broker's TCP Recv-Q.

So I set "queue.buffering.max.ms" to 1000 and "compression.codec" to "snappy". And by now, there is no packet loss.

I am curious about that why the default value of "queue.buffering.max.ms" is 0, does it mean that there is no batch? And if not, which property works?

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka version (release number or git tag): 0.11.4
• [x] Apache Kafka version: 0.11.x
• [x] librdkafka client configuration: default
• [x] Operating system: MIPS 2.6.21.7
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[rnpridgeon]

I suspect the lack of sufficient buffering performed by the client is leading to request queue saturation on the broker side. Subsequent requests will incur compounding latency costs as the network threads struggle to keep up ultimately leading to message timeouts as you have describe. Increasing batching as you have done here limits the number of requests made to the broker increasing overall throughput.

The broker exposes it's response queue via JMX with the following mbean which may be helpful in your investigation:
-kafka.network:type=RequestChannel,name=RequestQueueSize

Anytime this reaches queued.max.requests(default:500) network threads will block trying to add additional requests to the queue. As long as the network threads are blocked here they can't loop back to pull responses off the response queue so you'll see increased latency on both ends. This is not the only reason requests may take longer than expected but inspected broker health first is like the right approach to solving this mystery.

[lxyscls]

@rnpridgeon Thank you for the reply.

But I am doubt that kafka brokers can't afford a traffic which is less than 4000pps?

[rnpridgeon]

It's easy enough to verify/refute with broker JMX monitoring so I'd argue its not a bad place to start. If the problem had persisted even with batching enabled I would be more inclined to start with diagnosing the producer.

Also you mentioned using snappy compression. Is this enabled with queue.buffering.max.ms=0 as well? Typically you want to amortize the cost of this compression over larger message sets when possible to avoid bottlenecks.

[lxyscls]

@rnpridgeon Yeah, you are right! I will try Yahoo's kafka manager or JMX mbean.

Btw, I do not enable the snappy with "queue.buffering.max.ms=0".