Kafkajs: ERROR: The group is rebalancing, so a rejoin is needed

[oemergenc]

Hi,

i need some help with the error message from the title. I read this issue, which seems to be related:

https://github.com/tulios/kafkajs/issues/226

I also read the documentation here, which also seems to be related:
https://kafka.js.org/docs/faq#what-does-it-mean-to-get-rebalance_in_progress-errors

However, it doesnt matter what i try the errors are remaining. It only occures when i redeploy my application and unfortunately it always triggers our alerting system, which is annoying.

Here is the log from my application:

{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:32:41.900Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":16,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:32:41.901Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":3560}"}
{"loglevel":"error","msg":"ERROR [Connection] Response SyncGroup(key: 14, version: 1) {\"timestamp\":\"2020-04-06T10:33:41.906Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":18,\"size\":14}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:35:41.919Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-0b9827ef-9e82-4343-96d7-3bee2053c449\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[11,15,19,23,27,3,31,35,39,7],\"my-topic-2\":[11,15,19,23,27,3,31,35,39,7]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":180018}"}
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:37:17.024Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":45,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:37:17.025Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":2618}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:37:41.916Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-69d3bcd8-a1d9-428f-8742-142ccb674b67\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9],\"my-topic-2\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":24891}"}

I already tried to increase the following config parameters.

maxRetryTime,sessionTimeout, rebalanceTimeout,heartbeatInterval

Neither helped, any hints would be greatly apprechiated.

[Nevon]

It only occures when i redeploy my application and unfortunately it always triggers our alerting system, which is annoying.

As the documentation suggests, this is a normal occurrence during deploys:

This is a normal occurrence during, for example, deployments, and should resolve itself.

Whenever a member leaves or joins the consumer group (such as when you redeploy), a rebalance is needed, in order to assign work across all members of the consumer group. The members become aware of this when the broker responds to an API call with an error indicating that they need to resync.

During deploys, you are adding and removing nodes from the group, which means that the group cannot continue processing messages during that time (or at least they will be frequently interrupted when they have to stop and resync), so it is correct that your application isn't actually doing its job during that time. On the other hand, if the issue persist after your deployment is over, then it means that your application still isn't working and should probably trigger an alert.

I already tried to increase the following config parameters.

Take a look at the documentation for these parameters to understand further what they do. maxRetryTime and heartbeatInterval have nothing to do with this, and changing their values without understanding what they do could have bad consequences. In particular increasing heartbeatInterval could mean that the application heartbeats too infrequently and may cause the broker to assume that it is dead and kick it out of the group (depending on the value of sessionTimeout.

[oemergenc]

Hi,

thank you for your fast answer and the clarifying words.

Is there any chance you can think about, so that i can get rid of those errors? It would also be sufficient for me to log them as warn or info, however i have no clue how to do that.

Thanks in advance.

[Nevon]

The thing is, they are errors, which is why they are logged as such. It's just that sometimes they are expected errors. However, only you have the context to determine when it is fine (during deploys) and when it isn't. I would suggest that you adjust your alerting to not alert during deploys.

If this is not possible for whatever reason, such as if there's no way for your monitoring system to know that there is a deploy ongoing, then one workaround I can think of is to use your own custom logger that specifically looks for "The group is rebalancing, re-joining" and logs it as a warn rather than an error. I don't recommend it, but it is a possible solution.

[oemergenc]

Yes, writing my own logger which ignores errors is a solution i would like to avoid.

I think we need to live with those errors. We just recently switched over to KafkaJS from kafka-node and also used node-rdkafka before and at least those client do not log this kind of information on error level. Maybe this is also something this implementation can think about.

Closing this issue for now. Thank you for your fast feedback.

[cristianbriscaru]

I too believe that rebalancing should not be logged as error given that it is part of the normal protocol/workflow for kafka when a consumer joins/leaves a consumer group. No other libraries for kafka log this as an error ( springboot, kafka-node,...) . I believe this should be at worst logged as a warning if not a info.