Librdkafka: Support for ListConsumerGroupOffsets?

[jeffwidman]

I'd like to add a kafka consumer lag input plugin to telegraf, and for that I need a Golang client that supports the ListConsumerGroupOffsets call.
But as best I can tell this isn't currently available in librdkafka's AdminClient implementation (and therefore unavailable in confluent-kafka-go).

So this ticket is to track adding it.

Note that I'm a total n00b to C, so I may be just misreading the librdkafka code and this call may already be there...

One design comment, since I just added this method to the kafka-python client:

• A common scenario in monitoring situations is to fetch the offsets/calculate consumer lag for many/all consumer groups in the cluster.
• ListConsumerGroupOffsets is a very chatty call because it will first lookup the group's coordinator. So it requires two network round trips.
• However, when listing consumer groups, if a broker returns a consumer group, you know it's the coordinator. (I don't think this is part of the official Kafka spec, but nevertheless I doubt it will change anytime soon).
• If you allow ListConsumerGroups to have an optional param of specifying a single broker and ListConsumerGroupOffsets to have an optional param of specifying a known group coordinator than you can leverage this to nearly halve the number of network calls required to fetch offsets for all consumer groups:
  
  
  
  1. Loop through all brokers, calling ListConsumerGroups against each one.
  
  
  2. For the consumer groups returned, call ListConsumerGroupOffsets, using the group coordinator identified above.
  
  

Java doesn't currently support these overrides, but they were useful enough that we chose to include them in kafka-python, even though in general we try to follow the Java clients very closely: https://kafka-python.readthedocs.io/en/master/apidoc/KafkaAdminClient.html#kafka.admin.KafkaAdminClient.list_consumer_group_offsets

Similarly, I added optional support for specifying which partitions you want the offsets for. If omitted, then all known offsets are returned (requires KIP-88 which landed in brokers >= 0.10.2).

You can see how I then leveraged this when patching the Datadog check here:
https://github.com/DataDog/integrations-core/pull/2730/files#diff-deed983f73a04522ac621d7f6d0c8403R248

[jeffwidman]

If no one else picks this up, I'll take a stab at this...

[edenhill]

Hey Jeff!

The plan is to add the missing Admin API pieces early this year.
As for all-consumer-groups-in-cluster optimizations: that's an interesting problem.. and solution :)
Will look into it more and ping you when the time comes.

[plachor]

Hi @edenhill any plans to extend admin client with this API in upcoming releases? Possibility to query consumer offsets would enable new interesting scenarios like automation of scaling, monitoring of lag before maintenance actions like re-partition and etc.

[solarhell]

Any updates?

[accelerated]

Querying consumer offsets is a very useful feature! Please consider implementing without this workaround

[KrzysztofBranicki]

Hi @edenhill, can you share what are the plans regarding this feature in terms of work planning? It would be really great to have feature parity with Java client in this area. Currently it is real pain because we need to develop dedicated app/microservice in Java just to be able to read consumer group offsets, and we are not a Java software house.

[edenhill]

We'll be looking at catching up the Admin API in Q3

[plachor]

Is it still considered for Q3?