Confluent-kafka-dotnet: Question: Is there a plan for transactions for the producer?

[longility]

Is there a plan for a transactions for the producer?

Something like the following:

producer.beginTransaction();
  for (ConsumerRecord record : records)
    producer.send(producerRecord(“outputTopic”, record));
  producer.sendOffsetsToTransaction(currentOffsets(consumer), group);  
  producer.commitTransaction();

Referenced from: https://www.confluent.io/blog/transactions-apache-kafka/

We are in need of this, and we are seeing if we should do a quick workaround or if we should wait. Unless someone can guide us on what that could look like, we might be able to contribute.

[mhowlett]

yes, we're targeting end of December and it should be available in preview form much sooner. don't hold us to that timeline though - it'll be out when we have confidence it is stable enough.

[longility]

@mhowlett that sounds awesome. Is there an item that we can keep track of or see progress?

[mhowlett]

watch the librdkafka PRs. it's not open yet.

On Fri, Oct 18, 2019 at 1:45 PM Long Mai [e-mail] wrote:

@mhowlett https://github.com/mhowlett that sounds awesome. Is there an
item that we can keep track of or see progress?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/confluentinc/confluent-kafka-dotnet/issues/1091?email_source=notifications&email_token=AAAEVZVXDZSAP3HOWMNQI2TQPIN77A5CNFSM4JCLNDP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBV5CNI#issuecomment-543936821,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAEVZQRKR43FNPZXDXDNZ3QPIN77ANCNFSM4JCLNDPQ
.

[edenhill]

https://github.com/edenhill/librdkafka/issues/1308

[noahdav]

@edenhill @mhowlett Can we offer any help to get this added sooner than end of year? We have people who can work on this if together if needed.

[edenhill]

@noahdav The implementation is pretty much complete and we have full transactional functionality.
We're currently working on writing tests, there's a lot of intricate error cases that needs proper handling.

We really do appreciate your help; I believe the most valuable thing right now is testing, in particular to cover real world use-cases. So my suggestion would be for you to start using the new transactional API in your applications and report back any issues you may find.

The txns branch is not yet ready for wider use but it will be cleaned up this week and should be ready to go by next monday.

[noahdav]

@edenhill This is great news. We can test this this at scale. Please let me know when we can start using this branch.

[noahdav]

@edenhill Is this branch ready to be consumed for testing. We would like to start testing this at scale. Please let me know which branch we should start to use.

[noahdav]

@edenhill please let us know when we can start testing with this branch.

[smg-bg]

@edenhill I can help with testing as well

[edenhill]

Finalizing final error handling and test cases this week, will be ready for review and use by friday.

[mhowlett]

i've already integrated the current librdkafka txn branch with .net and have a working example, I hope to get that tidied up and a WIP PR pushed later today.

[mhowlett]

https://github.com/confluentinc/confluent-kafka-dotnet/pull/1147

[alexpirine]

Hello,

We're in dire need of EOS, and therefore we also need transactions to implement that.

Do you know if this is going to be released anytime soon now?

[mhowlett]

1.4.0-RC1 is on nuget, and includes transactions. It should be pretty reliable. An EOS processing example is here: https://github.com/confluentinc/confluent-kafka-dotnet/tree/1.4.x/examples/Transactions

There will be an 1.4.0-RC2 today with changes unrelated to transactions. There will be an RC3 to reflect these changes: https://github.com/edenhill/librdkafka/pull/2728 and this will impact the API slightly, but shouldn't stop you from working with 1.4.0-RC1.

We typically release following a successful 2 week soak test of the latest RC.