Node-rdkafka: consumer is not getting messages after some time of running

Created on 15 Oct 2018 · 11Comments · Source: Blizzard/node-rdkafka

Environment Information

OS: Docker of carbon-alpine
Node Version 8.11.4
NPM Version 5.6.0
node-rdkafka version 2.3.2

Im running 3 instances of app that listening for a specific events. at certain point, 2 of the 3 instances is not getting the events anymore.

when inspecting the kafka, I see that:

TOPIC                 PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                           HOST             CLIENT-ID
forgetPasswordInit    0          5558            5558            0               man-mailing-api-A-04abb206-9754-496b-984a-f740d1bdde73 /192.168.202.223 man-mailing-api-A
forgetPasswordInit    1          -               5472            -               man-mailing-api-B-39c418ef-5257-4db2-a923-ec1357f89c5e /192.168.202.221 man-mailing-api-B
forgetPasswordInit    2          -               5526            -               man-mailing-api-C-171c352c-5f27-42e6-8847-8147aa4bbf0b /192.168.202.222 man-mailing-api-C

As you can see, the CURRENT-OFFSET is shown only for the instance that listening.

Any suggestions of how to debug it?
When restarting the instances everything is back to normal

stale

Source

idangozlan

👍5

All 11 comments

Are you committing messages? Any firewalls closing your connections?

krystianity on 16 Oct 2018

It's not firewall and not connectivity issue (I guess), since the 3rd instance is still alive.
Im using the following options:

Kafka.KafkaConsumer({
  'client.id': hostname(),
  'group.id': 'some-group-id',
  'metadata.broker.list': ....,
});

idangozlan on 16 Oct 2018

I had a similar issue.
In my case, i was not able to consume all messages even though i had all 3 partitions up and running. And it is not an issue with the data either.

raghu64 on 6 Dec 2018

Also have ran into this issue. Any way of going about debugging or checking connectivity of the consumer? Have tried using the event_cb and commit_offset_cb options.

With event_cb, I was only able to log event.error events, where I'd get this error, which didn't seem to break the connection:

{ [Error: Local: Broker transport failure]
  origin: 'local',
  message: 'broker transport failure',
  code: -1,
  errno: -1,
  stack: 'Error: Local: Broker transport failure' }

Also I could log offset_commit_cb, which gives something like:

[
  {topic:"topic-one", partition:0, offset:5431}
  {topic:"topic-two", partition:0, offset:4271}
]

But nothing on checking the status of the consumer.

UPDATE:
event.error events seem to be loggable without any config. Other events can be debugged by setting the debug config for the Consumer to 'broker,consumer' (see https://github.com/edenhill/librdkafka/blob/v0.11.5/CONFIGURATION.md for the full list), which allowed me to receive event.log events:

        consumer.on('event.log', (e) => {
          console.log('event.log:', e);
        })

        consumer.on('event.error', (e) => {
          console.log('event.error:', e);
        })

brickbite on 21 Dec 2018

👍1

@brickbite It looks like I have solved the same problem recently -- https://github.com/Blizzard/node-rdkafka/issues/544

nervgh on 24 Dec 2018

hi @nervgh, thanks for linking me to that. originally I was thinking it might be related to issue #424. I'll give your solution a shot and come back if there's something else to report.

brickbite on 27 Dec 2018

🎉1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale[bot] on 27 Mar 2019

Multiple issues related to the same topic and no solutions. The stale bot keeps closing this issue on and on because no activity, but the problem is still there