Kafkajs: How to connect back to the Kafka broker after the connection is lost

[can-kayacan]

Kafkajs does not detect when Kafka broker is for any reason (e.g. restarting) becomes unavailable. Sending messages to a disconnected Kafka broker does not give any error. It continues this behavior even after the Kafka broker is available back again.

Is there a way to detect if the Kafka broker is down or how can we make sure that the connection is healthy and we can send messages safely?

[Nevon]

That is not how KafkaJS works. If KafkaJS makes a request and doesn't get an answer within a timeout period, it will throw an error and retry. The client re-establishes connection if the connection is closed. If it fails to reconnect, it will retry a number of times and eventually crash.

Could you provide some logs to prove what you are claiming? You can run with a higher log level to get more information about what's going on, by running with the environment variable KAFKAJS_LOG_LEVEL=debug.

[can-kayacan]

Well it is working as you described.

I was trying to stop the kafka docker with "docker pause" which was not really stopping the docker container that's why Kafkajs was still sending messages correctly. After correctly stopping the container it started to work like you described. Thank you for the help.

[robodude666]

Based on the behavior I've observed, if you your client is setup like so:

const kafka = new Kafka({
  logLevel: logLevel.ERROR,
  brokers: ['broker1:9092', 'broker1:9092', 'broker3:9092'],
  clientId: 'example-producer',
})

and the current broker goes down (broker1 for example), kafkajs will endlessly try to reconnect to that same broker until it comes back up.

I'd expect it to give up after a certain period of time and try another broker. If this is the expected functionality, am I doing something wrong with the setup? I'm using the producer.js and consumer.js examples.

EDIT: I just realized my topic's replication is set to 1. Would this be the reason kafkajs doesn't attempt to connect to any other broker because broker1 is the only one that has this topic/partition?

[Nevon]

Would this be the reason kafkajs doesn't attempt to connect to any other broker because broker1 is the only one that has this topic/partition?

Without actually reading the code, I'm not 100% sure, but it could be the case. There are two reasons to connect to a broker:

1. We don't know enough about the cluster, so we need to get metadata ("On which broker can I find topic foo?", "What's the host and port for broker with nodeId bar?"). This can be done against any broker. By default, we'll use any broker we're already connected to. If we're not connected to any broker, we try to connect to all the brokers we know about (through previous metadata calls) in random order until one connects. If all those fail, we try all the brokers that you configured the client with, in the order you specified them.
2. We need to communicate with the leader for a certain partition.

If you have replication set to 1, I could see how we would endlessly try to connect to that broker. Try setting replication to 2 when you create the topic, and then kill the leader for that topic-partition and see what happens. The other broker should become the leader for that topic-partition and we should connect to that one instead.

[robodude666]

I tried creating 3 brokers and setting my replication factor to 3. I was able to kill 1 broker and the consumer reconnected to another broker. However, after killing a 2nd broker all consumers stopped working. Bringing up the first broker caused all consumers to resume.