Confluent-kafka-dotnet: Question: Exactly-Once Semantics

[geofflamrock]

Description

Apologies if this isn't the correct forum to ask this, given it's not an issue as such but more a question.

With Kafka 0.11, there is support for an application to produce/consume with exactly-once semantics using transactions, are there any plans to add support for this? Thanks.

How to reproduce

N/A

Checklist

N/A

Please provide the following information:

• [ ] Confluent.Kafka nuget version:
• [ ] Apache Kafka version:
• [ ] Client configuration:
• [ ] Operating system:
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[treziac]

Github is also for question :)

This will have to be implemented by librdkafka, this is already on roadmap.

[geofflamrock]

Fantastic thanks, do you by chance know of a github issue in librdkafka that is tracking this at all so I can follow it? I did a quick search and couldn't find anything. Thanks.

[treziac]

Tracked in https://github.com/edenhill/librdkafka/issues/1308

[geofflamrock]

Thanks, should I close this issue now or leave it open to track support for this in the .NET client?

[mhowlett]

librdkafka v0.11.0 (and hence the dotnet client v0.11.0) understands the new message format added in 0.11.0 but idempotence and transaction functionality is yet to come. this is on the roadmap but we don't have an ETA for you at this point.

[ayoung]

I am waiting for this as well. Keep us updated.

[adrianwright]

I am waiting for this too.

[mhowlett]

we haven't started working on it but we will soon. it should be here within 6 months (it's a big feature).

[ZebpayArjunVachhani]

@mhowlett is it still in road map or is it implemented?

[mhowlett]

it's absolutely going to be implemented. idempotent producer will be done before transactions. can't yet guarantee a release date, but it's a current open task.

[Tenseiga]

+1

[AceHack]

Any update?

[mhowlett]

idempotent producer is in -beta2, and will be stable in 1.0 which will should be out on the timeframe of a couple of weeks. we're starting work on transactions immediately after.

[RomainAn]

Also waiting transactions, gonna wait

[tomnagly]

Hello, any update?

[mhowlett]

1.0 has idempotent producer. The consumer side of transaction support is underway here: https://github.com/mhowlett/librdkafka/tree/isolation1

[tomnagly]

Thank you @mhowlett
It is not clear to me if I can use it on dotnet client?

[mhowlett]

yep, simply set the EnableIdempotence property to true, and you're good to go!