Generator-jhipster: Migrate from spring-cloud-stream to native Apache Kafka client

[Dabz]

As seen face-to-face today with @jdubois , @framiere and I propose to leverage our Confluent expertise to provide better Kafka support in JHipster.

Overview of the feature request

JHipster is currently generating the Apache Kafka source code with Spring Cloud Stream, we propose to rely on the native Apache Kafka client instead.

Motivation for or Use Case

Spring Cloud Stream is always lagging a few version behind Apache Kafka and it makes the generated source code harder to read.

While we do see value in Spring Cloud Stream, from our experience, only advanced users would really benefit from this kind of framework and it is treacherous for beginners and intermediate users. While it helps, it hides too much semantic and blocks people from understanding the behaviour of their applications (e.g. automatic retry, error handling, idempotency, etc...).

[jdubois]

As we all agree here, let's do this!

[Tcharl]

I may be the only one that use rabbitmq. Not that important, Kafka is a pretty nice beast :-)

[DanielFran]

@pascalgrimaud Can this issue be closed?

[pascalgrimaud]

For now, we migrated from Spring Cloud Stream to Spring Kafka. If I understand well, we should use directly the native client.
So I don't know.

Maybe @ctamisier you have an advise on this ?
Otherwise, I think we can close and improve later if needed.

[ctamisier]

I think there are some improvements in progress about the docker image properties.
https://github.com/Dabz/generator-jhipster/commit/b3fe482fad4b357f117f292ad0fb90e218e7d3ec
maybe this ? :)

I have some work in progress (80% done) using kafka native client (producer and consumer) and test containers, that can be a "drop-in" replacement of what already exists in JHipster, I'll submit it later as a PR.

Then I'm still wondering if a producer or consumer should be directly included in a JHipster web application rather than in an "external" component dealing directly between kafka and the underlying database.

If it does make sense we can even think about integrating kafka streams and let the developer taking advantage of advanced concepts (kstream, ktable, joins, windowing,...) or not.

[pascalgrimaud]

@ctamisier : I just implemented the sync data between microservices for my current project. I'll improve the Kafka support probably next months. We used Spring Kafka, which works well for our simple use case.
Do you plan to migrate to Kafka Native client ?
We need to discuss about it when possible :-)

[ctamisier]

Yes this is the plan and recommended by the confluent expertise.
I won't be able to keep working on this this current month..
However I put the "effective" project here: https://github.com/ctamisier/clement (see the 'start' commit) and I've still some work in progress about running tests (some issues with application.yml loading and properties management).

[pascalgrimaud]

@ctamisier : no urgent here, it will be done in time. I already saw the amount of work you did, so I'm putting a bug bounty on this. Don't hesitate to ping me if you need help, when you start the integration in JHipster next month :-)

[cbornet]

I get that we remove Spring-Cloud-Stream. But here it seems we are also removing Spring-Kafka. What's wrong with Spring-Kafka ?

[ctamisier]

From my point of view:

• It is still about "lagging a few version behind Apache Kafka". The current spring-kafka 2.2.8.RELEASE uses kafka-clients 2.0.1. (Nov 08, 2018)
• And still hides too much semantic

[pascalgrimaud]

fixed by https://github.com/jhipster/generator-jhipster/pull/10379

[pascalgrimaud]

Lot of work done here to achieve it by @ctamisier and @clement26695
I'm increasing the bounty, so you can share: 200 & 100
Is it ok for you ?

[ctamisier]

Sure ! thank you very much !

[robmosca]

Hey! I was trying to follow the tutorial on using Kafka with JHipster on the JHipster website but it seems to still refer to the previous implementation, the one using Spring Cloud Stream. Is it possible to create a new tutorial on the current implementation using Spring Kafka?
I am a newbie with Apache Kafka and I hoped using JHipster would simplify my learning on how to integrate it in my applications :)

[pascalgrimaud]

@robmosca : thanks for this.
Indeed, our documentation is outdated, we need to update this.
Anyway, this tutorial is already generated directly by JHipster. So you can generate a new application, with Kafka. You'll have :

• configuration already done in application yml
• kafka docker compose file to start the broker
• a rest end point to test kafka
• a generated producer
• a consumer as example

[robmosca]

Indeed, after writing to you I tried to use the code, the way I found it the most intuitive... and it just worked! 😊
You have done an amazing work. Really easy to use. Thanks!

[mraible]

Is this tutorial out of date? https://www.jhipster.tech/using-kafka/

[pascalgrimaud]

@mraible : I confirm the documentation is out of date. Anyway, the tutorial is not needed anymore as everything is directly coded inside JHipster.