Kafka-node: Support for consuming a wildcarded topic

[lightswitch05]

We often define our topics on the format <service>.<app>.<type>. This works great for libraries that offer wildcard support. When I try to subscribe to a topic using a wildcard eg. finance.*.error, I get a TopicsNotExistError. Three questions:

1. Is there any interest in supporting wildcards in topics?
2. Are there any major technical hurdles that would prevent wildcarded topics from being added to topicExists and other functions that would be affected?
3. Are there any thoughts on how wildcard support would best be implemented and how it would affect the options of the Consumer objects

[hyperlink]

Roadmap wise

The module is currently implemented to the 0.8 version of kafka. Latest versions of kafka is backwards compatible so that's why the module still works on 0.9 (and probably .10 as well). The only exception is SSL. We added that recently. I would love to see us supporting the latest protocol.

The high level consumer needs to be updated to support 0.9 type of consumers. And this would include the wildcard support you mentioned. It would also include the new broker co-ordinator which I believe should resolve many of these rebalancing issues we currently have with our high level consumer.

So I would love to see us adding more support for 0.9 and 0.10. I don't know if other collaborators have similar goals. Maybe @haio or @estliberitas can chime in.

Getting wildcard feature today

You can get the wildcard behavior outside of this module by grab all the topics matching the wildcard out of zookeeper and consume on all them.

[mcasimir]

@hyperlink

You can get the wildcard behavior outside of this module by grab all the topics matching the wildcard out of zookeeper and consume on all them.

Can u please clarify how? What is the procedure? Retrieve the list and keep doing addTopic?

[lightswitch05]

@hyperlink thanks for adding this to the 0.6.0 milestone! I looked at adding support for it myself, but all the async calls in the zookeeper library are a little difficult to follow. I thought about trying to clean up the implementation up a bit by adding bluebird for promise support, but I don't know if there is any interest in promises for this library.

[hyperlink]

np @lightswitch05! I'm going to take a stab at implementing it next week. Though bug fixes will still take top priority.

Regarding bluebird are you thinking of introducing to the private calls in module or as part of the API?

In the past I've hesitated making drastic changes to the code since our test coverage is so bad. We've got a little better but I feel it's better to focus on better tests and better support for kafka 0.9+ and if we can improve the performance that would be great. I've always envisioned this module as a low level interface to kafka. In our app we create our own higher level abstractions around it. I don't know if others feel the same way.

[lightswitch05]

Bluebird would be a purely internal replacement of async. A good example is Zookeeper.prototype.getConsumersPerTopic. It has a huge async.series block with callbacks all over the place. Another is Zookeeper.prototype.addPartitionOwnership. A lot of the callbacks are just to carry on results or errors, but it make the processing path a bit hard to follow - at least for me. Promises would simplify a lot of it. Not only would it make the code a bit more readable, but I believe it would allow some of the large methods to be more easily broken down into smaller more maintainable (and testable) methods.

But also, since adding bluebird is a purely internal change - it is pretty low priority and does add a lot of risk with little return until testing has improved. As a newcomer to this project, those large async blocks are intimidating when considering ways to add new functionality. Just a thought, if you and the other project owners have any interest in promises in place of async - I could create an example branch.

In the meantime, I desperately need wildcard support. If there is anything I can help out with please let me know. I'm still learning a lot about kafka and zookeeper, but I can at least write unit tests.

[nathanph]

:+1: Interested in seeing wildcard support.

Obligatory #420 #yoloswag

[lightswitch05]

I saw 0.6.0 was released and was excited this might have been complete, but it doesn't look like its in 0.6.0 :(

[hyperlink]

Yes, sorry I wasn't able to implement this... PRs Welcome!

[teckays]

👍 bump

[dioptre]

Maybe use nats as a replacement for kafka rn.

[crobinson42]

@hyperlink what do we need to do to implement this? I'll contribute where I can

[crobinson42]

For reference: https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717