Librdkafka: Feature request: Support pluggable partition assignment algorithms for consumer

[jeffwidman]

Please add support for pluggable partition assignment algorithms for the consumers.

I was in a design review today where we will need this for a handful of upcoming services.

This was originally requested back in https://github.com/edenhill/librdkafka/issues/1135.

Using this ticket to track progress on this feature.

[chris-zen]

I would like to implement a service like Kafka Streams Interactive Queries on top of librdkafka. After looking at the source code of both projects, the only critical part that I miss in librdkafka is the ability to provide a custom assignor to be able to manage standby replicas of local states.

According to @edenhill here, here and the code here, it is possible to configure custom assignors internally, but it is not exposed at the API level, and only the builtin range and roundrobin assignors are allowed.

I was wondering if there is any plan to implement it, how difficult would it be, or what are the chances to get mentoring for contributing.

[chris-zen]

Some more context on why is so relevant to be able to customise the assignor. This is the case for Kafka Streams, where there are many consumers with state, which makes rebalances quite expensive when not done properly. Currently Kafka Streams uses one assignor called Sticky that minimises state migration between instances (and also accounts for standby replicas).

There is a recent KIP-429 to make the protocol more incremental, that gives an interesting view on the current state and the relevance of the matter.

[edenhill]

Let's look at making the assignor API public when we implement KIP-429 (et.al) to make sure all use-cases are covered.

[Buttered]

Is there any update on this feature?

[rhoban13]

I am also interested in using this feature. Is it likely the proposed api will be as @edenhill reference here. Is making this api public something you might accept a PR for, or is the above ref larger change doing to nullify such a PR?

[dshivashankar1994]

+1 for the feature

[edenhill]

What is your use-case for supplying a custom partition assignor?

[dshivashankar1994]

I wanted to ensure that only one consumer is active for a consumer group. The topics i am using have only one partition and we recommend our users to create a single user per consumer group. But if by mistake there are more consumers joining, there is a problem.
Currently I am using RebalanceCb to handle things. Say a,b subscribe to set of 10 topics, then only one of the consumer will receive all the topics while the other will not start (fail). This works fine.

Problem is, consumer A subscribes to 2 topics, consumer B subscribes to 10 topics out of which 2 are overlapping. Not after the rebalanceCb, consumer A will start receiving in 2 topics and B in remaining 8 topics. B is not even aware that it is not receiving the messages because of A. To avoid this, I wanted to implement my own partition assignment algorithm.

[ChadJessup]

@edenhill - would you consider a PR implementing a public API for custom assignors? It looks like a similar pattern to custom Partitioners would work?

[edenhill]

@ChadJessup I think a good start would be a PR containing only the public API (rdkafka.h), but I suggest to wait with that until the Incremental rebalancing PRs have been merged since there will be internal changes to the assignor APIs, see the incrreb branch.