Librdkafka: KafkaConsumer can act as a Partition Assignor

[kenxxu67]

Description

In a consumer group, I need to do smart partition assignment so that the consumers can reduce cache rebuilding when there are assigned new partitions.

Python client seems to have this functionality.

How to reproduce

Only see RebalanceCb in the latest librdkafka code base, which is only notifying the consumers the rebalance decision made by Kafka

Checklist

Please provide the following information:

• [ ] librdkafka version (release number or git tag):
• [ ] Apache Kafka version:
• [ ] librdkafka client configuration:
• [ ] Operating system:
• [ ] Using the legacy Consumer
• [ ] Using the high-level KafkaConsumer
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

The assignment functionality is currently not pluggable (but it is prepared to be pluggable).
With enough interest we'll bump this up the roadmap.

[kenxxu67]

Do you know when you might have this with the current interest level? We have some design decision need to be made pretty soon base on this. Would appreciate any information.

[edenhill]

I think this is the second feature request for pluggable assignors in 1.5 years, so it is not high on the list unfortunately.
At this rate I wouldn't expecting seeing it before august.

A workaround in the meantime is to write your own assignor and making your own fork & build of librdkafka. Just copy one of the existing assignors (e.g., https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_range_assignor.c) and add it somewhere here: https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_assignor.c#L520

[kenxxu67]

Thanks for the quick response. I will look into it and get back to you.

[jeffwidman]

@edenhill is there a place I can track the current state of support for pluggable custom assignor algorithms for librdkafka/the other confluent bindings? Googling only showed me this issue so it'd be good to have a breadcrumb trail to whatever issue now tracks this...

[edenhill]

Hey @jeffwidman ! Not sure we have a proper master issue for it, maybe you'd like to create a new one asking for pluggable assignor support so we can properly track demand?