Confluent-kafka-dotnet: Transaction or Unit of Work Support for Kafka.NET

[PingPongSet]

Description

Is transaction or unit of work supported for Kafka .NET?

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version:
• [ ] Apache Kafka version:
• [ ] Client configuration:
• [ ] Operating system:
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[mhowlett]

we have just started working on this. probably 3-6 months away.

[PingPongSet]

What other features, or enhancement are under development within one year?

Is there a roadmap for next releases that is available publicly?

[PingPongSet]

Can transaction or UOW span async producer or consumer?

[YogirajA]

@mhowlett any update on this? Thanks!

[mhowlett]

our focus will switch EoS after 0.11.5 is released.

[PingPongSet]

@mhowlett EoS, do you mean end of support? But the feature is not even available currently.

[rnpridgeon]

Exactly Once Semantics.

https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

[mhowlett]

Just discovered this: https://github.com/dotnetcore/CAP - have not used it, but looks like it might be good if you have relatively low throughput requirements. but with 0.11.5 now out, @edenhill's switched focus to implementing idempotent producer / transactions in librdkafka.

[hafeezre]

Has there been any progress on Transaction support?

Our server-side requires it.

[mhowlett]

not yet

[gpaniagua013]

Still not progress? :)

[mhowlett]

the first step is adding the capability for the consumer to ignore aborted transactions (isolation.level config parameter), this is underway.

[fipil]

Hi, any progress with the transactions support?
Matt, can I implement it myself (create a fork)? Is the transaction support already in the latest librdkafka?
Thanks.

[mhowlett]

the capability for the consumer to ignore messages in aborted transactions (in librdkafka) is in the final stages of review, but not yet merged. work hasn't started on the producer side of transactions yet.

[rohan-bhosale]

when can we see an update i.e the merged code

[mhowlett]

you can track progress here: https://github.com/edenhill/librdkafka/pull/2377
there's been a fairly substantial refactor, hoping to add a commit for that today to the PR.

[rohan-bhosale]

you can track progress here: edenhill/librdkafka#2377
there's been a fairly substantial refactor, hoping to add a commit for that today to the PR.

Thanks Matt.

[smg-bg]

Any updates about this?

[mhowlett]

I may push a WIP PR today (word count written in .net with transactions) - i have it working already locally. but you'll need a custom build of librdkafka and the api subject to change and it's alpha quality. we're pushing hard on this feature though ATM, should be early next year.

[JohnTicehurst]

I have a few questions about Kafka Transactions
Will there be changes to the latest api to handle Transaction support ?
For the Producer will it be as simple as a BeginTransaction / CommitTransaction bounding the .Produce or .ProduceAsync
Will the Consumer have to change or is that handled with an IsolationLevel setting ?

Thanks,
John

[mhowlett]

The consumer isolation setting is ReadCommitted by default, so you don't need any adjustments, however you need to be sure not to commit offsets using the consumer (you need to turn off auto commit, which is on by default).

The producer just has additional methods, yes, no additional changes for transactions, though i've added a cancellation token parameter to ProduceAsync in that PR, sort of not related, just useful in the example.

The semantics of the txn API are not as good as they could be (following the java api). see the discussion in KIP-447 https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics , however it's workable. I think the WordCount example is getting pretty good now in PR #1147 with comments. we'll make changes later following the java client after kip-447 is implemented.