Confluent-kafka-dotnet: (Question) Need help in implementing following functionalities.

[roughblot]

I have been trying to implement Kafka on an MVC Web Application. I'd like to implement following functionalities :

• Display the consumer list (List of all consumers consuming from specific topic) for specific Broker.
• Display the consumer lag for each consumer.
• How can we find total number of partitions for specific topic ?

Thank you for your time.

[mhowlett]

You can list consumer groups and their constituent members using the ListGroups method on Producer or Consumer. Unfortunately, the member data is currently not deserialized in the C# client and there is no built in way to discover member topic / partition assignments from this data.

All topics and the total number of partitions per topic can be found using the GetMetadata method on a Producer or Consumer instance. You can then get high watermarks for all topic/partitions using the QueryWatermarkOffsets method on a Producer or Consumer. If you have an instance of a consumer in a consumer group, you can now use the Position method to get the current offsets, and thus consumer lags.

I think the client can't currently do what you would like. An admin client is in the pipeline (across languages), which may.

[roughblot]

@mhowlett

• Tried using the GetMetaDatamethod and to my surprise it takes allTopics as argument of type bool which cannot be scaled down to a specific partition. It returns brief information on all the topics and partitions for a broker. Can you suggest me any way out ?

I shall surely wait for the Admin client, which may solve almost all our issues.

Thank you.

[mhowlett]

When used on a Producer, you can specify a specific topic (on a Consumer you can't currently).

There is an open PR (#63) with API documentation. Maybe this will help:

https://github.com/confluentinc/confluent-kafka-dotnet/pull/63/files#diff-80ae38da0f3b0eb17f006b13962bcc62R721

Note that we are marking this API as unstable in the upcoming release. The functionality will remain going forward, but we may change how it is exposed to make the API clearer.

[edenhill]

The GetMetadata() APIs should provide the same functionality regardless of client. If the consumer is missing a topic-specific one we should really add that now.

[mhowlett]

on the consumer we don't have functionality to make the SafeTopicHandle's already (like we do on the producer) so it would be a bit of effort/refactoring to make it work.

[edenhill]

Ah, okay. too bad

[roughblot]

@mhowlett No issues. I wanted that functionality on consumer end for some reason.
Anyways, Thank you for your time.

[mhowlett]

as a workaround you can create a producer as well and use that - you don't actually need to use it to produce anything. you'll have extra TCP connection(s) open to brokers, but you're unlikely to notice any negative perf impact.

[roughblot]

@mhowlett We're able to fetch what we needed. Thanks for all the help.