Librdkafka: Connection State

[twaters]

The event callback can tell me when the connection to the broker has failed. But is there a way to know when the connection is re-established? The only thing I can see at the moment is a debug log.

[edenhill]

Nope, not currently.

What this usually boils down to is some event mechanism for monitoring availability of the leader brokers for your desired partitions, eg #137

I guess that's what you are after?

[twaters]

Yep, something like that is exactly what I'm looking for.

[jonathanwang2005]

+1. It will be very useful to have an event notification when a broker is available after ALL_BROKERS_DOWN.

[edenhill]

We probably want these events:

• BROKER_ADDED - hostname, id
• [BROKER_REMOVED (brokers are currently not removed from librdkafka) - hostname, id, reason]
• BROKER_UP - hostname, id
• BROKER_DOWN - hostname, id, reason
• ALL_BROKERS_DOWN - reason
• PARTITION_LEADER_UP - topic, partition, hostname, id, reason
• PARTITION_LEADER_DOWN - topic, partition, hostname, id
• GROUP_COORDINATOR_UP - hostname, id
• GROUP_COORDINATOR_DOWN - hostname, id, reason

[briot]

I like the proposal with the various kinds of events.

The scenario I currently have is the following: I have two brokers, and a topic with multiple partitions, but replication_factory=1 (the problem can be generalized with more brokers and higher replication factory).

When one of the two brokers goes down, I am getting a msg_timeout error (as expected) for my high-level consumer. But the error callback does not provide any information as to which broker is down. So my program must assume they are all down, and let listeners know they will not be getting new messages for the time being. Would be nice to have more detailed information, and the proposed PARTITION_LEADER_DOWN error would be useful !

Thanks