Spring-boot: Provide auto-configuration for Reactor Kafka

[jntakpe]

Spring Boot provides a great helper configuration KafkaProperties class for Kafka.

This class includes two spring-kafka module classes :

import org.springframework.kafka.listener.ContainerProperties.AckMode;
import org.springframework.kafka.security.jaas.KafkaJaasLoginModuleInitializer;

Due to those imports, in order to leverage Spring Boot's Kafka auto configuration, it is required to import spring-kafka module which is largely increasing the application size (our projects are using Reactor Kafka).

IMHO, the configuration that depends on spring-kafka should be moved to spring-kafka module, allowing Spring Boot's projects to leverage automatic type safe Kafka configuration without importing spring-kafka.

[wilkinsona]

Thanks for the suggestion. The whole auto-configuration, of which KafkaProperties is a part, depends on Spring for Apache Kafka. In other words, it's intentionally auto-configuration for Spring for Apache Kafka and not for Kafka itself. The auto-configuration will back off completely in the absence of Spring for Apache Kafka. Also, you shouldn't try to use KafkaProperties in your own code. As noted in the documentation, all configuration properties classes are for internal use only:

Similar to auto-configuration classes, @ConfigurationProperties classes available in Spring Boot are for internal use only. The properties that map to the class, which are configured via properties files, YAML files, environment variables etc., are public API but the content of the class itself is not meant to be used directly.

I don't think it makes sense for Spring Boot to offer configuration for Kafka without using Spring for Apache Kafka. Our opinion is that it makes sense to use the latter when interacting with Kafka in a Spring application. If you have use cases where that's not true, please share them here and we can reconsider or pass them on to the Spring for Apache Kafka team for them to look at.

[jntakpe]

@wilkinsona, all our applications are microservices that relies on Spring Boot + Spring Webflux. All those microservices fully leverages Reactive programming so it was pretty obvious for us to connect to Kafka embracing reactive libraries. That's why we are using reactor-kafka to communicate with Kafka. Even if Spring-kafka supports some async types, IMHO it's not fully embracing reactive programming like reactor-kafka does. We are very pleased by all the effort made so for by Spring teams to push Java reactive support forward, it seems logical that Spring Boot enables easy Kafka configuration through KafkaProperties class letting users choose the fully reactive implementation reactor-kafka or the classic spring-kafka.

[wilkinsona]

Thanks. So it sounds like what you're really looking for is auto-configuration support for Reactor Kafka. Can you share some details of how you're currently using Reactor Kafka and how you're using KafkaProperties to configure it?

[jntakpe]

Actually that's pretty straightforward, we inject KafkaProperties to create both KafkaSender and KafkaReceiver beans. It looks like, the following sample :

@Bean
fun sender(properties: KafkaProperties): KafkaSender<String, String> {
     val options = SenderOptions.create<String, String>(properties.buildProducerProperties())
     return KafkaSender.create<String, String>(options)
}

Hence both KafkaSender and KafkaReceiver are configured using regular Spring properties.

The full reactor auto configuration support would be great. Nevertheless, being able to simply inject KafkaProperties without being forced to add spring-kafka to the classpath is our main concern

[wilkinsona]

Thanks for the additional information.

Nevertheless, being able to simply inject KafkaProperties without being forced to add spring-kafka to the classpath is our main concern

This (alone) almost certainly won't happen. As I mentioned above, Boot's own @ConfigurationProperties classes should be considered internal so you should not be injecting KafkaProperties. Given that the only supported use of KafkaProperties is by the auto-configuration for Spring for Apache Kafka, and its needs are met by the current arrangement, it will be difficult to justify modifying KafkaProperties as you have described.

[jntakpe]

So you intent to provide a reactor-kafka auto configuration mechanism ?

[wilkinsona]

We don't know yet, hence this issue still being labelled as waiting for triage. We can certainly consider it though. In the meantime, I would strongly encourage you to use your own configuration properties and @ConfigurationProperties class for the configuration of Reactor Kafka.

[jntakpe]

Yes that's what I intent to do. Thanks !

[wilkinsona]

Having discussed this with the Reactor team, we've decided not to add auto-configuration for Reactor Kafka at this time. If the situation changes, we can re-open this issue and reconsider. Thanks for the suggestion, @jntakpe.