Node-rdkafka: Reconnect after losing connection to broker

[srikav]

Environment Information
OS: Docker alpine linux
node-rdkafka version- 2.3.2]

Steps to Reproduce
1) start kafka and zoo keeper
2) start producer and consumer
3) stop kafka and zookeeper
4) producer will show broker failure errors
5) restart kafka and zoo keeper
6) the producer and consumer seems like they reconnect, but they do not consume or produce data to topics.
-- seems like it is hung in a state.

node-rdkafka Configuration Settings
initProducer: {
'metadata.broker.list' : process.env.KAFKA || 'localhost:9092',
'group.id' : 'podlink-reporter',
//'dr_cb' : true,
//'event_cb' : true,
'debug' : 'topic,metadata,msg,fetch,cgrp'
},
initConsumer: {
'metadata.broker.list' : process.env.KAFKA || 'localhost:9092',
'group.id' : 'podlink-reporter',
'enable.auto.commit' : true,
'debug' : 'topic,metadata,msg,fetch,cgrp'
},
log.txt
log.txt

Additional context

[jsnoble]

Any headway with this? We are facing the same issue. After complete broker disconnection it seems to hang. When the connection comes back the consumer does not get any more data

[bobrowadam]

Same issue here

[webmakersteve]

As far as the producer is concerned, disconnects should be managed fully transparently. If you're seeing the producer stop producing after a disconnect when the brokers come back up I would recommend you add delivery reports and logging around errors and successes to ensure that data is still flowing in that capacity.

I can't tell anything significant from the logs other than there was a disconnection.

[gnitsenko93]

@webmakersteve does node-rdkafka manage re-connections internally?

I mean the best approach (based on you advice) is:
(1) detect connection failure
(2) reconnect on a service layer (restart service/service worker/or whatever else).

But if after the step (1) we do not want to do anything will node-rdkafka make attempts to re-establish connection with brokers?

[webmakersteve]

@gnitsenko93 correct, node-rdkafka manages re-connections internally. The only disconnection it does not manage is the initial disconnection, which I am considering changing.

[webmakersteve]

You should use produce or consume constantly and dumbly, just assuming it will work eventually, and node-rdkafka will reconnect to brokers as they come back online.

[deyceg]

@webmakersteve on reconnect to the broker will you emit another ready event?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[mvtm-dn]

Same issue. Looks like after some attempt to reconnect node-rdkafka just stop infinity loop in flow mode.

[hugomd]

@mvtm-dn Did you end up finding a solution for this? I'm having the same issue

[mvtm-dn]

Yep. I stop using flow mode