Kafkajs: Group Instance ID support

[deswaan]

What about supporting the Kafka group instance id feature?
This prevents the rebalancing to occur if you keep the same id when you start a new consumer.

[Nevon]

For reference, I believe this is the KIP that is being mentioned: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

These are the relevant PRs for the Java client:

1. https://github.com/apache/kafka/pull/6177
2. https://github.com/apache/kafka/pull/6650
3. https://github.com/apache/kafka/pull/6666
4. https://github.com/apache/kafka/pull/6714
5. https://github.com/apache/kafka/pull/7122

The general gist is to add a new instanceId to the consumer config, which should be included in several APIs, including JoinGroup, LeaveGroup, SyncGroup, OffsetCommit, Heartbeat and DescribeGroup, which means implementing support for those updated API versions. The logic surrounding leaving a consumer group has to also change, as we should not issue a LeaveGroup request if instanceId is set to a non-null value (i.e. this is a static member).

Although I haven't found the Java implementation of it, I also believe that the partition assignment needs to be updated to assign based on the instance id rather than member id, in order to avoid reassigning partitions during rolling restarts.

[tulios]

Just mapping some of the work required, we would need to support protocols JoinGroup v5, SyncGroup v3 (potentially up to v5), OffsetCommit v4, v5, v6, and v7 (potentially v8), Heartbeat v3 and v4, LeaveGroup v2 and v3

[thiagopsnfg]

hello friends, any updates on this issue?

[Nevon]

hello friends, any updates on this issue?

To my knowledge, no one is working on this currently. In #888 I upgraded most of the APIs to their required versions, but nothing has been done beyond that.

[thiagopsnfg]

It's a pity, you did an excellent job @Nevon. Can you measure if it would be too much work for us to try to propose a PR?

[Nevon]

For this kind of change, most of the work is probably in understanding what needs to change. Making the change is usually not too bad. I don't have a full picture of the work, as it's been quite a few months since I last looked at this, but I think it can be done in steps.

1. Implement KIP-320. This is probably gonna be an epic in and of itself. The reason we need this is because we need to support Fetch v12 in order to get the currentLeaderEpoch field, which we need in order to support OffsetFetch v6-7, which in turn we need because it includes the groupInstanceId field that's needed for KIP-345.
2. Add support for OffsetFetch v7
3. Change the behavior of a consumer leaving the group to not issue a LeaveGroup request if the instance id is set.
4. Expose a way of configuring the instance id on the consumer.
5. Update the partition assigner to take into account the instance id (the KIP has a good example of how)

I have not really mapped out the work for KIP-320, so I don't know how big or small it is, but it's definitely the first step needed. We have another issue open for that #818