Node-rdkafka: Does node-rdkafka consumer retry internally while consuming via consumer stream?

[prashanthmadduri]

Hello @webmakersteve,

We are implementing kafka consumer stream like var stream = KafkaConsumer.createReadStream this.

1. We are interested to know if the node-rdkafka module manages retries internally or not.
2. Are there any errors in consumer which we should attempt a custom retry for reconnecting or is it not required all together?

@webmakersteve Any help will be much appreciated. Thank you.

[codeburke]

The point of the ConsumerStream is to give a continuous stream of messages. In order to do that it does have to deal with some retry logic that you can see here:
https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer-stream.js#L166-L176
https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer-stream.js#L252-L264

https://github.com/Blizzard/node-rdkafka/blob/master/lib/kafka-consumer-stream.js#L180-L188
Any errors will be bubbled up via the error event. If you do not have an error handler then it will cause the stream to throw an Exception and die.

I've been running a Consumer Stream in production for a couple years now and from my experience it makes sense to setup an error event listener to log the errors to your logs, but ultimately I haven't had to add any extra logic in handling the errors for consumers. In the README it also says:

Most errors are not necessarily fatal, but the ones that are will immediately destroy the stream.

Hope this helps!

If this answered your question then please close the Issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.