Librdkafka: Consumer group, multiple topics, topic churn

[blandings]

Description

We have a consumer group consisting of consumer processes, each process running as a daemon. The idea is to have this group subscribe to a bunch of topics. As long as the complete list of topics is specified at the time of topic_conf creation, a consumer group consuming from multiple topics works.

However, if there is a churn in topics; topics come and go, is there a way to signal those consumers to refresh their subscription for topics? Can I add and delete topics from the subscription list?

Do consumers take topic names specified as wild cards? So, I could, for example, setup a consumer group to subscribe to all topics with names like 'ConsumeFromMe_*'? This may be one way to allow for topic churn.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [11.4] librdkafka version , but will/can move to 11.6 if required to support our use case
• [ 1.0.0] Apache Kafka version: but will/can move to 2.0.0 if required to support our use case
• [ack=all ] librdkafka client configuration: (possibly not relevant to my question here)
• [ Redhat 7.2] Operating system: <REPLACE with e.g., Centos 5 (x64)>
• [ NA] Provide logs (with debug=.. as necessary) from librdkafka
• [NA ] Provide broker log excerpts
• [ N] Critical issue: Not critical, but urgent

[blandings]

Hello, are there any suggestions/opinions on this one?

[edenhill]

The cluster topic list is refreshed every topic.metadata.refresh.interval.ms (def 5 minutes) to pick up any subscribed topics that have been created, or deleted, since the last time.
If this results in a changed subscription the consumer will rejoin the group.

Wildcard topics are supported using regex, which is used when a topic name starts with ^.
It is fine to mix wildcards and exact topics, like so:
"mytopic", "^myregex.*", "^another[0-9]$", "static"

of subscribed topics is refreshed

[blandings]

Thank you for the very precise answer. We are trying this at our end. We will let you know. Also, is this documented already?

[edenhill]

Apart from the API docs for subscribe(), which mentions wildcard regex support, there is nothing. We'll see to it.