Librdkafka: Upgrading from 0.9 to 1.0 greatly impacts Kafka performance itself

[aurelienrb]

Hello,

We upgraded our client lib from 0.9.5 to 1.0.0 and then observed a massive impact on our Kafka instances : +500% of CPU increase!

• the impact is on Kafka server itself, there's no noticeable change on our C++ producers (using librdkafka)
• we do use gzip compression
• we do NOT use SSL
• we removed the config broker.version.fallback config = 0.8.2.2 used for 0.9.5
• we added request.required.asks = 1

That's the only settings we use. Again, upgrading the lib did not have measurable impact on our C++ producers. We are running out of idea about what can cause such a huge impact on Kafka itself.

Any idea?

• [x] librdkafka version 1.0.0
• [x] Apache Kafka version: 0.10.2
• [x] Operating system: Ubuntu 16.04

[edenhill]

Try setting linger.ms=50

[aurelienrb]

Thanks for the tip :smile: With that setting, the impact is reduced on Kafka instances: we now get about 60% of increase server side, which is much less but still pretty high.

We noticed our C++ clients (producers) have a huge increase on network bandwidth (both in and out). This explains the impact on the server.

Setting linger.ms=50 does not seem to have an impact (although it does on the server CPU). It's hard to get precise numbers because the metrics accumulates all the network traffic (we run a web servers), but to give you an idea, when we upgrade librdkafka, the overall traffic becomes:

• 3x times higher in input
• 2.5x times higher in output

So it suggest a massive increase to the network traffic related to the communication with the Kafka servers (we send [a few hundreds .. a thousand] messages per second on each producer, it represents less than 10K messages/sec on average server side).

[edenhill]

How many partitions are you producing to?

It would be interesting to see what the typical batch message count is, since larger batches improve performance and reduce overhead.
You can do this by monitoring the topics.batchcnt metric in https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md#topics

[aurelienrb]

We use 5 partitions / 8 brokers. Our C++ producers write on 2 topics.

We did a mistake reading the network bandwidth: there's no noticeable change when we update the lib. But the impact server side is real. Therefore I'm confused we can't see any impact client side.

We are going to upgrade more progressively the lib version and enable the metrics: are they available for 0.9.5?

[edenhill]

The batch metrics were added in v1.0.0.

The increased server load is indeed a bit strange.

[polomarcus]

Hi,
@aurelienrb 's team and me performed others tests and we could observe the same load increase on our brokers with version 0.11.6 or 0.11.1 of the lib :arrow_right: It made us think that V1.0 was not responsible for our performance problem.

In 0.11, queue.buffering.max.ms 's default value was changed from 1s to 0. By resetting it to 1s we could fix our performance issue on our brokers even with v1.0.0 :smile: :heavy_check_mark:

Producer:
The default value of queue.buffering.max.ms was changed from 1000ms to 0ms (no delay). This property denotes the internal buffering > time (and latency) for messages produced.

Thanks for your great work on the library

[aurelienrb]

Yes it works now, thanks again for you help.
I'm closing the issue :smile: