Librdkafka: On-demand broker threads

[edenhill]

This is a coalescing issue for the following common issues:

• Too many connections to broker (connections maintained even though they are unused) ✅
• Too many threads (causes CPU core contention and higher memory usage)
• Blocking requests ✅

Requirements:

• Sparse connections - just connect to the brokers we need to talk to ✅
• Sparse threads - just create threads for brokers we need to talk to
• Purge old brokers - remove brokers no longer reported in metadata
• Multiple connections to the same broker to avoid HOLB (e.g., Fetch blocking Commit) ✅
• Portable IO scheduler (current Windows implementation does not support fd-based wakeups, leading to higher latency or higher CPU usage)
• Pluggable IO scheduler?

[Ishiihara]

@edenhill Do you plan to work on this soon?

[edenhill]

@Ishiihara It is on the backlog and work will start after the next release (june).

[gduranceau]

@edenhill
Hi Magnus,

I'm pleased to see improvements planned in this area. I was wondering if you considered going even further and use a single reactor thread to handle events from all the broker sockets? In environments with very large Kafka clusters where clients (especially producers) may target all partitions of the topics defined, that would avoid spawning dozens of librdkafka threads in each client.

Also, do you plan to apply these changes for bootstrap broker threads/connections as well?

[edenhill]

@gduranceau
The idea, which is not set in stone, is to use a single IO thread (or possibly one for read and one for write, depending on the facilities the OS provides) that will be highly host-OS specific, to be as optimized as possible.
Then there'll be N worker threads doing the actual parallel-friendly heavy work, batching, compression, etc.

The number of worker threads will be a combination of on-demand and hard configuration limits and depends on the number of partitions (level of parallelism) and the throughput load.

SSL is an unknown here, can we maintain performance using SSL with just a single IO thread that will have to do all the SSL shuffling too?

[gduranceau]

@edenhill
Very good. I like the design very much.

I'm not really worried with SSL. I worked on an enterprise service bus serving 300k tps, and SSL was never a concern: symmetric encryption is pretty fast.

One suggestion for the I/O thread (coming from that experience as well): if a single thread is a bottleneck, you can actually support spawning multiple I/O threads (number based on configuration). The key is to make sure to assign each connection to one of these threads only, for all its lifetime, and for all its operations, so as to avoid any sort of concurrency issue. So basically, when a connection is created, one of the I/O threads is chosen with round robin policy, and it remains associated to the connection until the connection terminates.

[billygout]

@gduranceau Is this ready and tested? I think I really need this guy, for too many open fd's on the kafka broker side.

[gduranceau]

@billygout The patch is complete, but so far, this has not been integrated in any production running application on my side yet.

[billygout]

@gduranceau thanks! You might just have me as a prod tester in a Kafka producer application.

[gduranceau]

@billygout Cool! Don't hesitate to report problems. I will also have to deploy this in a producer application in the weeks to come.

[jbvmio]

@gduranceau @billygout How has testing been going with this? (connect.on.demand)

[billygout]

I have not tried this yet. Sorry.

On Thu, May 10, 2018, 12:17 PM jbvmio [e-mail] wrote:

@gduranceau https://github.com/gduranceau @billygout
https://github.com/billygout How has testing been going with this?
(connect.on.demand)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/edenhill/librdkafka/issues/825#issuecomment-388156710,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABoBEnXoBWSimMR2WFh0Xi0RlJcDFzarks5txJJSgaJpZM4KQvQA
.

[gduranceau]

I tested this in a variety of scenarios, but not in a producer application just yet. For info, there's at least one limitation that I'm aware of with my implementation, which I commented in PR https://github.com/edenhill/librdkafka/pull/1659 (on May 15).

[chaloulo]

+1 for this enhancement.

[hackerwin7]

+1 for this.

[amitt001]

@edenhill I see that enable.sparse.connections flag is added to librdkafka. Are these two features present in the latest version?

• Sparse connections - just connect to the brokers we need to talk to
• Sparse threads - just create threads for brokers we need to talk to

Also, I am wondering how is it going to behave for a Producer. The "Introduction" docs say that:

"Random broker selection
When there is no broker connection and a connection to any broker is needed, such as on startup to retrieve metadata, the client randomly selects a broker from its list of brokers"

My assumption is that its only true for consumer. As initially when a producer is configured it's not producing to any topic so it connects to all the brokers by default. I am seeing the same behavior in my tests as well. As soon as I am configuring the producer with two brokers I see 6 new threads. But the strange behaviour is that the thread count is not going down even when I start producing to a topic.

[edenhill]

@amitt001 The next version of librdkafka, v1.0.0, will feature sparse connections, it will only connect to the brokers it needs to communicate with. But there will still be one thread per known broker, i.e., no sparse threads at this point, that requires a larger re-design of the librdkafka internals and is not scheduled for the short term.
The non-connected threads will be mostly idle though.

[fuyijun1989]

@edenhill My team has a scenario that dozens of clients run in one machine and each connects to 300+ brokers, and due to the 1 thread to 1 brokers design, the thread number goes up to 10K, resulting in severe cpu burn on context switch. If we won't get a "sparse thread" config in the near future, could you give us some advices to mitigate the situation? Maybe some minor code change or config change.
Many thanks.

[edenhill]

We're preparing our Q1 backlog and I'm doing my best to get sparse threads on there.
The idea is to only spin up a broker's thread when it is actually needed (to housekeep partitions or communicate with a broker), and then kill them after some idle period again. This should remedy the situation for when a producer or consumer only needs to communicate with a subset of brokers in the cluster, which is usually the case. E.g., the partition leaders for partitions being fetched or produced to.

[umashankartm]

@edenhill, we are having an application that connects to multiple Kafka clusters. we are seeing 1000s of threads being created. Do you have any timeline on when can we expect this feature?
Thanks you..

[billygout]

@umashankartm the following PR was already merged years ago: https://github.com/edenhill/librdkafka/pull/2019

You'll find it referenced in release notes from back then with accompanying config param documentation.

If you are using that already, and still are lots of connections, perhaps your Kafka cluster is 1000's of nodes large, and you are consuming from or producing to at least one partition led by each broker? Otherwise, no idea.

[umashankartm]

@billygout thanks for your quick response.. my understanding was that only sparse connection was merged not sparse threads. Sparse connection does help to some extent but issue we are facing right now is with respect to the number of OS threads. Moreover, i dont see any documentation related sparse threads in the latest code base. Am I missing something?

[billygout]

@umashankartm good point. I don't remember whether the threads were cut down along with the connections, but even before that change, I seem to recall the number of threads being one per broker, plus a small number of special purpose threads. Also, if you're running on Linux, note that librdkafka uses the thread naming feature, and it may help diagnose if you can show the thread names, by, for example, running top In thread mode for your process PID with top -H -p <PID> and then hitting the c key, which should toggle the display of the COMMAND column to show thread names.