Librdkafka: Accessing the rockdb state store

[pablocastilla]

Hi!

First of all thanks for your hard work.

As a future feature I think it would be great if we could access to the state store (rockdb) in a transactional way. That can help a lot in implementing service logic in a exactly once way like it is shown here:
https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/

I understand implementing the whole stream processor is impossible, but just with accessing the storage in the transaction we could do a lot and we could move from at least once to an exactly once scenario.

Thanks!

[benstopford]

Yeah, we've discussed this a bit in the past but it hasn't gone beyond chat. It's a good idea though imo.

There are a couple of reasons why I like the approach. Firstly it's useful in its own right: a simple, useful database that could be backed by a changelog topic. Secondly it plugs a hole in the KSQL+consumer pattern. By that I mean the pattern where you use KSQL to do joins and aggregates then pass events to a stateless microservice that implements business logic. While this works if the business logic is truly stateless (i.e. everything it needs is in the event) often that business logic will require some form of stateful operation (marking things done etc.) and pulling in an external DB somewhat defeats the point.

[pablocastilla]

Hi @benstopford !

Glad you have answered, I am a big fan of your blog posts.

I have reached this idea by doing a PoC with dotnet and kafka, in the readme you have the conclusions, maybe you want to have a look.
https://github.com/pablocastilla/kafkiano

Thanks!