Kafkajs: Response heartbeat key 12 | Error : The coordinator is not aware of this member

[esuarezz]

{"level":"ERROR","timestamp":"2018-09-17T12:35:16.757Z","logger":"kafkajs","message":"[Connection] Response Heartbeat(key: 12, version: 0)","broker":"52.50.141.155:28094","clientId":"kafkajs","error":"The coordinator is not aware of this member","correlationId":106,"size":6}
{"level":"ERROR","timestamp":"2018-09-17T12:35:16.757Z","logger":"kafkajs","message":"[Runner] The coordinator is not aware of this member, re-joining the group","groupId":"x-group","memberId":"kafkajs-b39167d2-1a03-437c-a0e9-04ea1130fa16","error":"The coordinator is not aware of this member","retryCount":0,"retryTime":313}

Sometimes I get this error and can be there for long time, then it is working fine.. or sometimes it is not consuming directly because it is showing this... the configuration always is the same, so not understand why sometimes is going fine, and other times not

[Nevon]

Hey Elio,

The error you're encountering is because your consumer has not communicated with the brokers in more than sessionTimeout milliseconds (see sessionTimeout in https://github.com/tulios/kafkajs#-options-1).

Basically, in order for the coordinator to know which consumers in the consumer group are healthy, each consumer needs to send a heartbeat to indicate that it is alive. The sessionTimeout option sets after how long the coordinator should consider the consumer unhealthy if it has not sent a heartbeat. How often your consumer heartbeats is configured with the heartbeatInterval option. Note, however, that heartbeatInterval only sets the minimum amount of time that has to pass between each heartbeat. If you take a look at the implementation of eachMessage, you can see that by default we will heartbeat after processing each message, provided that at least heartbeatInterval milliseconds has passed:

https://github.com/tulios/kafkajs/blob/9fa9f447e9805654b7b79cb6f9b101c4a3acf84d/src/consumer/runner.js#L129-L158

In other words, what may be happening is that processing each message is taking longer than your sessionTimeout. If so, when the consumer finally finishes processing the message and gets the opportunity to heartbeat, the coordinator will consider it to be unhealthy and you will get the error message you're seeing.

The way to solve this is to think about how long processing each message will take (taking into account any retries or backoff strategies you may be employing), and then setting the sessionTimeout to a bit more than that.

I haven't found any great documentation on this, but this should give you a decent high-level overview: https://www.opsclarity.com/understanding-kafka-consumer-lag/

[tulios]

Just complementing the reply, your heartbeat interval should be at least 1/3 of your session timeout.

[ashercoren]

@Nevon , you said that the will heartbeat after processing each message, provided that at least heartbeatInterval milliseconds has passed. If a message gets processed very fast, and the next message will only arrive sometime later, can that cause a heartbeat to be missed as well?

[Nevon]

If a message gets processed very fast, and the next message will only arrive sometime later, can that cause a heartbeat to be missed as well?

No. It doesn't matter if there are messages to process. On each Fetch request, which happens continuously in a loop, we'll heartbeat if enough time has passed since the last heartbeat.

[alexsyd]

@Nevon does this error mean our Streams app doing some time consuming tasks (which it shouldn't)? As far as I understand Streams library is doing the heartbeat calls by default and only doing something wrong in our code would cause such error, right?

[Nevon]

@alexsyd I don't know what streams library you are using, so I can't answer that. In KafkaJS, it depends on how you have configured your client, but you can see exactly what we are doing in my previous comment.