Quarkus: Kafka Streams - Config - Topics should be a list

[gsmet]

Right now, it's a string but it should rather be a proper list in config:

https://github.com/quarkusio/quarkus/blob/master/extensions/kafka-streams/runtime/src/main/java/io/quarkus/kafka/streams/runtime/KafkaStreamsRuntimeConfig.java#L37

@gunnarmorling can you have a look? Or maybe you had a particular reason to do it this way?

[lhauspie]

Can I work on this issue after @gunnarmorling answered ?

[gunnarmorling]

+1 for making this a list. No specific reason that isn't one yet, an
oversight probably. @Logan, feel free to grab this one. Thanks a lot!

[lhauspie]

Thanks. I'll work on it.

PS: my handle is @lhauspie ;)

[gunnarmorling]

Thanks. I'll work on it.

👍

PS: my handle is @lhauspie ;)

Yeah, sorry, I was replying from my phone, didn't have the handle and didn't think that @ with your name would hit another user... :)

[gunnarmorling]

Btw. just in case you're interested in doing more contributions to the Quarkus Kafka Streams extensions, there's a few things we could talk about. E.g. built-in support for metrics and health checks, if the corresponding Quarkus extensions are enabled. It's on my list of things to work on, but would be more than glad about any help. As said, just in case, simply ignore this message otherwise :)

[lhauspie]

Btw. just in case you're interested in doing more contributions to the Quarkus Kafka Streams extensions, there's a few things we could talk about. E.g. built-in support for metrics and health checks, if the corresponding Quarkus extensions are enabled. It's on my list of things to work on, but would be more than glad about any help. As said, just in case, simply ignore this message otherwise :)

It would be a pleasure. Feel free to give me some pointer to issues to start discussion.

[lhauspie-adeo]

As the topics String is a comma-separated list of topic names, KafkaStreamsTopologyManager.java parse this topics String to a Set by using this piece of code :

        Set<String> topicsToAwait = runtimeConfig.topics
                .map(n -> COMMA_PATTERN.split(n))
                .map(Arrays::asList)
                .map(HashSet::new)
                .map(Collections::unmodifiableSet)
                .orElseGet(Collections::emptySet);

So my question is: how is it possible to make topics to be a List (or a Set) in KafkaStreamsRuntimeConfig.java without parsing a comma-separated String ?

@gsmet do you have a use case where you need to retrieve the List (or Set) of topics ?

[machi1990]

KafkaStreamsRuntimeConfig.java

Hi @lhauspie-adeo

I guess changing

@ConfigItem
public Optional<String> topics;

to

@ConfigItem
 public List<String> topics;

In the config file should do the job.

[lhauspie-adeo]

Oh. OK, I did it, I just have to test before creating the PR.

Thanks

[machi1990]

Oh. OK, I did it, I just have to test before creating the PR.

Thanks

You are welcome. Thanks for the contribution :-)

[lhauspie]

Sorry, I got the wrong account again.
Could you assign this issue to @lhauspie, not @lhauspie-adeo

[gunnarmorling]

It would be a pleasure. Feel free to give me some pointer to issues to start discussion.

As a starting point, you could take a look at this discussion here: https://github.com/quarkusio/quarkus/issues/4375. This is about providing means for extensions such as ours here to optionally expose metrics and health checks, based on the presence of certain capabilities provided by other extensions. The actual metrics and checks to provide are discussed in the blog post linked in that issue. The idea is to further simplify the user's live by not requiring them to implement the glue code shown in the post but having this done once within the extension itself.