Confluent-kafka-dotnet: Subscribe to many topics and async processing of messages

[Marusyk]

Description

I have a single consumer which subscribe to all wanted topics (let say 5). Then I need to process all messages from one topic one by one (sync), but in parallel(async) between topics. When consumer receive a message, it should redirect it to the corresponding thread by topic. Looks like 5 topics = 5 processing thread.

Is it ok approach, or should I create a consumer for each topic? Now all topics have 1 partition, but it will be extended in the future.

If it is ok to have single consumer for many topics, then could you please suggest how to implement it? There are any common patterns or something like that in this package?

Thank you in advance.

[mhowlett]

your suggestion (one consumer, 1 thread per topic, each managing a queue of messages that is populated by the consume loop) sounds good to me.

in the future we may improve the consumer API for this use case.

[Marusyk]

But how can I implement this? Should I create some manager, that will get numbers of Subscriptions from consumer, then create thread for each, then check if (consumeResult.Topic == "A") => to thread A ...

Are there any solutions?

[mhowlett]

something like that, yes. we don't have any examples yet.