Quarkus: KStreams extension does not support configure it in code

[lordofthejars]

Describe the bug

KStreams extension does not support configure in code as if you try to configure KStreams programmatically, you get:

java.lang.RuntimeException: Failed to start quarkus
    at io.quarkus.runner.ApplicationImpl.doStart(ApplicationImpl.zig:198)
    at io.quarkus.runtime.Application.start(Application.java:89)
    at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:90)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:61)
    at io.quarkus.runtime.Quarkus.run(Quarkus.java:38)
    at io.quarkus.runner.GeneratedMain.main(GeneratedMain.zig:30)
Caused by: io.quarkus.runtime.configuration.ConfigurationException: One or more configuration errors has prevented the application from starting. The errors are:
Configuration key "quarkus.kafka-streams.application-id" is required, but its value is empty/missing: Property quarkus.kafka-streams.application-id not found
Configuration key "quarkus.kafka-streams.topics" is required, but its value is empty/missing: Property quarkus.kafka-streams.topics not found

Expected behavior

To be able to configure kstreams programmatically.
Really critical for command line mode.

Actual behavior

An exception is thrown.

To Reproduce
Steps to reproduce the behavior:

1. Create a plain project with kstreams extension
2. Create the quarkus main
3. Package and run

[geoand]

cc @gunnarmorling

[gunnarmorling]

@lordofthejars, why is it you need to start Kafka Streams programmatically?

[lordofthejars]

Command line application where user set the information as argument. CLI tool.

[gunnarmorling]

Hum, and there's no application.properties in CLI mode? Why not?

[lordofthejars]

there is, but I want to do a CLI tool: my-tool -b broker-host -t my_topic

[gunnarmorling]

Ok, but then you could have application.properties in your CLI tool JAR.
What am I missing?

>

[lordofthejars]

that if it is a CLI tool I don't want that the user needs to edit an
external file. Imagine that to do an ls -ll I should edit
application.properties and add that I want to use -ll. Or in oc login
--token I should create an application.properties file setting the token.
It has not much sense for a CLI tool.

Missatge de Gunnar Morling [e-mail] del dia dt., 5 de
maig 2020 a les 9:00:

Ok, but then you could have application.properties in your CLI tool JAR.
What am I missing?

>

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/quarkusio/quarkus/issues/9058#issuecomment-623887622,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALSMYMSA65DA5Y2FPI6HYDRP62RHANCNFSM4MY5D43A
.

[geoand]

@lordofthejars as a workaround can't you set properties with -Dquarkus.kafka-streams.topics=whatever?

[lordofthejars]

Yes I can do it but it is this, just a workaround, if you think about in
terms of developer experience is not great. Maybe adding a flag that
disables the configuration validation might be the best option so this is
set internally in the CLI tool.

Notice that this is going to be a problem in more extensions.

Missatge de Georgios Andrianakis [e-mail] del dia dt., 5
de maig 2020 a les 9:17:

@lordofthejars https://github.com/lordofthejars as a workaround can't
you set properties with -Dquarkus.kafka-streams.topics=whatever?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/quarkusio/quarkus/issues/9058#issuecomment-623893785,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALSMYMDRES3PW6CZ56T5D3RP64RZANCNFSM4MY5D43A
.

[geoand]

Yes, it's definitely a more general problem that we needto think about.

cc @maxandersen

[gunnarmorling]

Maybe adding a flag that disables the configuration validation might be the best option so this is set internally in the CLI tool.

I don't think that's the right approach. Rather have a way for populating Quarkus properties from more idiomatic CLI arguments than "-D=..."; you still should boostrap Kafka Streams via the extension.

[maxandersen]

somewhat related discussion is at https://github.com/quarkusio/quarkus/issues/8871
and then https://github.com/quarkusio/quarkus/issues/6497 feature request which is pending.

not sure if that will fix your issue though @lordofthejars as I assume you might want to expose a simpler/more human friendly "api" to the user instead of setting 'raw' properties ?

[lordofthejars]

@maxandersen Exactly that's my point, suppose I want to implement oc in Quarkus. I would expect to be able to create oc login --token=XXX and not oc login -Dtoken=XXX

[maxandersen]

i guess with a static main you should be able to wire in custom overrides to application.properties similar to what unit testing does - @geoand @stuartwdouglas does this sounds feasible to add to the Quarkus.run() ?

[geoand]

Custom overrides meaning that one would be able to use myProp instead of say quarkus.some-extension.something?

That sounds feasible to me.

[gsmet]

Can't you just populate system properties in your main? I would expect Quarkus to consume them or am I mistaken?

[gsmet]

Hum, maybe it's a bit too late in the process though. I suppose Quarkus is already started when you can do things.

[stuartwdouglas]

So the way we should deal with this is via some kinda of annotation based way of consuming/processing command line arguments, including the ability to bind them to config properties. Something like @ConfigParam(short="t", long="topic", usage="Specifies a Kafka Topic to connect to", repeatable=true, configProperty="quarkus.kafka-streams.topics")

This should be part of a generic module that makes it easy to parse command line params and print usage instructions etc. @maxandersen have you looked into this at all, I know there was some talk about picocli?

[maxandersen]

Hum, maybe it's a bit too late in the process though. I suppose Quarkus is already started when you can do things.

if you have your own main quarkus haven't started yet - but it would be nice to see if we can have a way without requiring a main.

On picocli - i haven't looked into this specific usecase but I know picocli author is working on a quarkus extension and will be posting on quarkus-dev soon on it.

about the annotation binding approach - I'm not an instant fan; but I think picocli could be enhanced to allow to use picocli annotations as normal and then have it apply settings based on other annotations.

like:

@Option(names={"-t", "--topic"}, description="Specifies a Kafka Topic to connect to", arity="*")
@ConfigBinding("quarkus.kafka-streams.topics")
List<String> topics;

if we could let picocli (or any other similar framework build up these properties) and then have quarkus process the resulting "beans" and pickup @ConfigBinding then that could work in combination of more free form filling of system.properties if more advanced processing is needed.

[remkop]

I posted on the dev list and @stuartwdouglas and I briefly discussed this. My example in that thread uses command mode, but this is not required. Quarkus can simply use picocli internally to (as @maxandersen suggested) populate the @Option-annotated fields and then process the resulting beans and pick up @ConfigBinding next. There are various ways to accomplish this, happy to discuss more details.

Picocli out of the box supports most (if not all) of the features mentioned under Requirements in https://github.com/quarkusio/quarkus/issues/6497 .