Confluent-kafka-dotnet: Custom Partitioners

[mhowlett]

this is a frequently requested feature. librdkafka now provides a built in partitioner that is compatible with the java client partitioner: murmur2_random, but there are other use cases for a custom partitioner. we plan to implement this by adding a ConfigureTopics method to the producer which will have the restriction that it must be called before the topic is first used. This method will take a config dictionary, where one of the properties that can be set is a partitioner delegate.

[gunjanarora]

When can we expect this feature to be available?

[mhowlett]

after the 1.0 API is stable, probably towards the end of the year.

[rjrizzuto]

I also would like this feature. I have an existing application that does streams processing using NEsper. I would like to switch to Kafka for the pub/sub bus. I think mapping event categories to topics makes the most sense for that, as opposed to putting all categories in a single topic. The issue I have is that I need to ensure that partitions in each topic are mapped to the same consumer by key so that the consumer can still join streams based on that key. I essentially need the same partitionor that Kafka Streams uses.

[rjrizzuto]

I see that 1.1.0 is out in Nuget. Is there any ETA on custom partitioners yet?

[rohitshrivastava04]

any news on this feature?

[rjrizzuto]

.Net is the red haired step child still.

[Berezhnyk]

I am waiting for this feature too...

[mhowlett]

It's lower down the priority list since it's easy to work around (just explicitly specify the partition on produce, determined via a class called MyFancyPartitioner if you want). We do plan to implement this still, and it's not hard so it'll get done.

[mhowlett]

it's actually a roadmap item in our Q1 planning, so I just discovered, (but don't rely on that, it'll be something that'll get bumped if other things slip).

[rjrizzuto]

Setting the partition in the producer doesn't seem to allow for a custom rebalance algorithm in the comsumers, like a custom partitioner would.

[DobrovAlexey]

Hi @mhowlett, is there any news on this feature?

[mhowlett]

custom partitioners are in 1.6.2

[lambdakris]

Hi @mhowlett is there documentation for this feature yet? If not, is there a comparable example in java or another lang that I could take a look at and try to adapt?

[mhowlett]

have a look at the integration tests for example use: https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/test/Confluent.Kafka.IntegrationTests/Tests/Producer_CustomPartitioner.cs