Confluent-kafka-dotnet: Assert hit in round robin assignor

[mhowlett]

via: @wwarby https://github.com/confluentinc/confluent-kafka-dotnet/issues/1366#issuecomment-668669487

I've just run into massive instability after upgrading some high throughput C# apps from v 1.4.4 to 1.5.0. In my case the error thrown is

dotnet: rdkafka_roundrobin_assignor.c:97: rd_kafka_roundrobin_assignor_assign_cb: Assertion `next < rd_list_cnt(&eligible_topic-> members)' failed

... and results in an all out application crash. Rolled back to 1.4.4 and it solved the problem.

In my case, I'm consuming data from topics that already exist and I don't have topic auto-creation enabled - I'm running multiple instances of the C# app, each instantiating a single consumer to join the consumer group.

[mhowlett]

cc: @edenhill

[ralmsdeveloper]

Hello @mhowlett,
Will we have a patch for this?
I have the same problems.

[mhowlett]

magnus is currently out of the office - he was making changes to this, so i'll let him investigate when he gets back. For now, revert to 1.4.4.

[ralmsdeveloper]

Yes, it worked for me!
I will try to find the reason for the bug, I will be happy to contribute!

[mhowlett]

thanks! suggest looking at librdkafka changes since 1.4.4 that touch the round robin code.

[mhowlett]

additionally: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1366#issuecomment-673198582

upon calling consume the first time, it throws an exception, but if you try the second time it works

[wwarby]

Any ETA on a fix for this?

[edenhill]

We'll hopefully release v1.5.3 today or early monday.

[edenhill]

Monday

[plachor]

pre-release or stable ?

[plachor]

@mhowlett will you pre-release v1.5.3-RC5 ? I'm looking for checking if my issues are gone after update.

[mhowlett]

1473

[mhowlett]

https://www.nuget.org/packages/Confluent.Kafka/1.5.3-RC5

[plachor]

@mhowlett, @edenhill seems that my issues with endless loop in round-robin assignor are gone. Also my tests are green.

Once you release stable version I will check if round-robin is distributing partitions properly among existing members in case of uneven subscriptions for topics with non-const partitions count. Any ETA for stable release ?

[edenhill]

librdkafka v1.5.3 is out, dotnet client should be out later today.

[plachor]

Great thnx

[plachor]

Round-robin assignor is now working as intended :)