Librdkafka: EOS support

[edenhill]

Support for Exactly-Once-Semantics as introduced in Apache Kafka 0.11.0.

The message format is already supported which means librdkafka may be used
with EOS-supporting clients (official Java clients) without a performance penalty.

To track popularity, please add a +1 reaction to this comment if this is a feature you need

EOS is made of up three parts:

• Idempotent producer - guarantees messages are produced once and in order (enable.idempotence=true)
• Producer Transaction support - commit or fail a set of produced messages and consumer offsets across partitions (BeginTransaction(), SendOffsetsToTransaction(), etc)
• Consumer transaction support - filter out messages for aborted transactions, and wait for transactions to be committed before processing (isolation.level=read_committed)

We'd be interested to know which functionality you are planning to use, use reactions to vote:

• 😆 - Idempotent Producer - supported in librdkafka v1.0.0 ✅
• 🎉 - Producer transactions - supported in v1.4.0 ✅
• ❤️ - Consumer transactions - supported in v1.2.0 ✅

[kiranrj]

We were primarily looking for a transaction aware consumer. So as you mentioned, exposing read_uncommitted=false as a property would really help.
Thanks in advance .

[vladcrisan13]

@edenhill do you have an estimated timeline for releasing this feature?
Also, can you suggest a reliable work-around that would ensure exactly once message consumption until this feature is released?

Thanks a lot!

[edenhill]

We're aiming for Q2.

If there was a reliable work-around this feature wouldn't be needed, right? ;)

[vladcrisan13]

That's true, but it was worth a shot :)
Thanks again!

[XiaoyaoFan]

We also need the EOS feature, and thanks for putting efforts on it! 😀 Would you mind to share the current development status? Thanks!

[edenhill]

@XiaoyaoFan Idempotent producer is scheduled for Q2, EOS for Q3

[cmeiklejohn]

Is the idempotent producer still on track for Q2?

[edenhill]

The plan is now to release all EoS features (idempotent, transactions, EoS) in the same release, which will be in Q3.

[IranjeetSingh]

We are eagerly waiting for the release that will contain EOS features.

[thebigredgeek]

Any update here?

[edenhill]

@thebigredgeek We're soon beginning work on the Idempotent producer.

[edenhill]

Please vote on what EOS functionality you are planning on using by adding a reaction to the top comment on this issue.

[edenhill]

Idempotent producer support is now available for testing on the idempotence branch.
Enable by setting enable.idempotence=true in the producer configs.

[BertusGreeff]

For our scenarios we are interested primarily in EOS to solve the Zombie Fencing problem as described here: https://www.confluent.io/blog/transactions-apache-kafka/

Does the work that is in the idempotence branch support transactional.id and is it enough to implement Zombie Fencing?

[sachnk]

+1 for EoS (specifically transaction support).

[ianwsperber]

Are you able to offer an update on the expected timeline? Appreciate the work that's being done here 😄

[edenhill]

librdkafka v1.0.0 with idempotent producer support is scheduled to be released within about 1 month.
Early release candidates will be made available within two weeks, we strongly encourage community testing to vet out any corner cases.

[edenhill]

@BertusGreeff Theoretically it should be possible to do zombie fencing with idempotent producers, given a transactional.id, but it'll need a bit of implementation on the client and testing to know for sure.

[ianwsperber]

Great to hear @edenhill ! Is there any news about producer or consumer transaction support?

[edenhill]

@ianwsperber Consumer transaction support will follow the 1.0.0 shortly. Producer transactions will be late Q4 or 2019.

[sean-buchanan]

+1 for transaction/eos support.

[logonod]

+1 for transaction/eos support.

[esalahov]

+1 for transaction/eos support.

[mirsahibmovsumov]

+1 for transaction/eos support.

[Tenseiga]

+1 for transaction support

[GeorgeShagov]

edenhill commented on Oct 4

librdkafka >> v1.0.0 with idempotent producer support is scheduled to be released within about 1 month

Hello
Any prgrs with this?

thx

[edenhill]

@GeorgeShagov We're on the final leg of the release cycle, pulling in the last fixes and doing final verification. An RC will go out tomorrow and the final release friday or monday.

[GeorgeShagov]

Thank you, Magnus, this is a good news indeed

[edenhill]

Two new issues have recently surfaced which blocks the v1.0.0 release until early next year:

• #2163 - idempotence: out of order issue after timeout
• #2164 - consumer: stalled message consumption after cluster failure

We encourage people to try out the v1.0.0-RC5 release candidate in the meantime.

[maver1ck]

Hi,
Any update on this ?

[sreejithp]

@edenhill , wondering if the release of the Idempotent producer support is around the corner?

[edenhill]

It is! Everything is looking great and we're just finishing up some packaging issues prior to the release.
Will have one of the final RCs later this week hopefully.

[edenhill]

librdkafka v1.0.0 has now been released, including idempotent producer support.

[BertusGreeff]

Congrats, we’ve been looking forward to this milestone!

[maver1ck]

@edenhill
Great news, but I have the question:
Which of the original points are supported ?
I assume idempotent producer.
What about transactions ?

[edenhill]

@maver1ck We now support the idempotent producer, not transactions.

See release notes here:
https://github.com/edenhill/librdkafka/releases/tag/v1.0.0

[andyczerwonka]

In the end, we're looking for feature parity in the .NET client (when compared to the Java client) and the .NET client depends on librdkafka.

[edenhill]

@andyczerwonka What features exactly are you missing from the .NET client?

[andyczerwonka]

@edenhill sorry, should have been more explicit. We have a use case where transaction support would really help. A batch of messages, in this context, is logically a single document, although the addressable atomic unit is still a single message.

[pavel-agarkov]

I'm using Kafka for microservices communication. In many cases there is an external counterpart for some internal event or group of them and they have to be dispatched all or none otherwise current bounded context will be out of sync with other contexts. Yes, it is possible to dispatch only internal events and then some other process will read them back and dispatch external events. But internal events are very pure and don't contain all information necessary for external integration events so the whole aggregate has to be resurrected in order to produce a proper external event which is definitely suboptimal.

[AlexeyUzhva]

Are there any defined plans for transactions support?

Here for bunch of use cases it would be extremely useful and mitigation of their absence looks unpretty and not at all optimal.

[JiangXiao2019]

From the source code librdkafka-1.0.0-RC7, I found that isolation_level is set to READ_UNCOMMITTED for consumer mode. I want read committed messages only, how should I do?

[edenhill]

@JiangXiao2019 we're aiming to add READ_COMMITTED support during 2019 Q2 or Q3.

[AlexeyUzhva]

@edenhill does it also mean full transaction support, or just READ_COMMITTED / READ_UNCOMMITTED switch?

[edenhill]

It means the consumer will be transaction-aware, filtering out uncommitted/aborted transactions.

[AlexeyUzhva]

Crossing the fingers that transaction-aware producer is also somewhere on the road :D

[JiangXiao2019]

@edenhill OK, got it, thank you.

[JiangXiao2019]

@edenhill It's OK for me if consumer can filter out uncommmitted transactions. It seems that aborted transaction has been filtered out in current version, Am I right? Thank you!

[edenhill]

@JiangXiao2019 The current version performs no filtering of transactions.

[JiangXiao2019]

@edenhill And then I have a question, my app use simple consumer mode, it pull messages from kafka with specified offset. The producer was developed by customer, and they used kafka transaction. When I consume the message which offset=100, librdkafka returns offset=101 to me. I checked the kafka server configure, no [compose] options was set. So I think this maybe caused by aborted transactions, messge with offset=100 maybe marked as "aborted" by kafka server.

[edenhill]

I'm not aware of any such filtering functionality on the broker.

[JiangXiao2019]

in rdkafka_broker.c，
/* Ignore aborted transactions for now */
if (AbortedTxCnt > 0)
rd_kafka_buf_skip()

[edenhill]

That code ignores the list of aborted transactions, not the aborted transactions themselves

[JiangXiao2019]

@edenhill Could you help me to know what caused this issue? "When I consume the message which offset=100, librdkafka returns offset=101 to me".
read 100----> returns 101
read 101----> returns 101 too
This happed 500 thousand times in half a day.

[edenhill]

@JiangXiao2019 This issue is to track EOS popularity. Please open up a separate issue and provide logs with debug=cgrp,fetch enabled.

[harshadvichare]

@edenhill

Once we have updated librdkafka to 1.0.0 versions accordingly step mentioned in document. While producing we are getting the following error message.
Broker: Message format on broker does not support request

retries = 3
acks = all
retry.backoff.ms = 100
max.in.flight.requests.per.connection = 1
enable.idempotence = true
broker version = 0.11.0.0

[edenhill]

@harshadvichare This issue is to track EOS popularity. Please open up a separate issue and provide logs with debug=broker,feature,protocol enabled.

[edenhill]

HELP WANTED

To fully understand the user requirements and prioritize accordingly we would like to ask you all to explain how you are aiming to use EOS and Producer Transactions specifically.

So please take a moment to write down your use-case, this will help us know what to implement and when.

[maks-rafalko]

We are working on the application with exactly-once guarantee (well, at least we try to achieve it).

There are several microservices that can read from and write to Kafka. Java microservice (and its internal implementation) supports read_committed isolation level, but PHP microservice does not, because at the moment of writing this comment read_committed is not yet implemented in librdkafka, but fortunately WIP.

So, different parts of the application can read committed or not yet committed data, while we must read always only committed data.

[robfr]

We are consuming data from an unreliable source and publishing to kafka. Should we encounter an interruption, we have the possibility to retry the entire stream of data, but we cannot restart from where we were last (no gaurantee data is in the same order or that states have not changed). As it is, we load the entire source into memory/disk and then publish from that "more reliable" source. Ideally we should add some consumer logic to deduplicate in case of a publication error too.

Producer Transations would be an elegant solution to allow us to consume the source, publish immediately and cancel should we encounter a problem with the source and keep our consumer application simple.

Thanks for your work on librdkafka!

[Tenseiga]

We are working on payment domainwhere the whole payment process shld be in
a kafka transaction. And eos is an absolute must.

On Wed, 31 Jul 2019, 17:15 robfr, [e-mail] wrote:

We are consuming data from an unreliable source and publishing to kafka.
Should we encounter an interruption, we have the possibility to retry the
entire stream of data, but we cannot restart from where we were last (no
gaurantee data is in the same order or that states have not changed). As it
is, we load the entire source into memory/disk and then publish from that
"more reliable" source. Ideally we should add some consumer logic to
deduplicate in case of a publication error too.

Producer Transations would be an elegant solution to allow us to consume
the source, publish immediately and cancel should we encounter a problem
with the source and keep our consumer application simple.

Thanks for your work on librdkafka!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/edenhill/librdkafka/issues/1308?email_source=notifications&email_token=ABLKBNHHQGQ4DDG5FJ6HCV3QCFJZTA5CNFSM4DSCPCNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3GUAEA#issuecomment-516767760,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLKBND4NE34S33TZA424NTQCFJZTANCNFSM4DSCPCNA
.

[sachnk]

We have ETL workflows where we consume data from kafka, transform it, and then produce a new message back onto kafka. We have several hops where we do this consumption, transformation, and production. Some of the hops are in Java, where we get access to EOS semantics, but some of the hops use librdkafka, which means those hops are at-least-once semantics. As soon as any hop in the path is at-least-once, the entire path becomes at-least-once.

Having EOS semantics / producer transaction will eliminate the need for us to build business logic that de-dupes along our ETL path... allowing our entire ETL path to be EOS, regardless of whether we're using Java or not.

[sean-buchanan]

To get our events into Kafka, we had publish them in batches. Just want to ensure that either all or no events get into the store so we don’t have any duplication.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

---

From: Magnus Edenhill [e-mail]
Sent: Wednesday, July 31, 2019 1:53:47 AM
To: edenhill/librdkafka [e-mail]
Cc: sean-buchanan [e-mail]; Comment [e-mail]
Subject: Re: [edenhill/librdkafka] EOS support (#1308)

HELP WANTED

To fully understand the user requirements and prioritize accordingly we would like to ask you all to explain how you are aiming to use EOS and Producer Transactions specifically.

So please take a moment to write down your use-case, this will help us know what to implement and when.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/edenhill/librdkafka/issues/1308?email_source=notifications&email_token=AFAWNTE3DVEZ5I3FUWZW52LQCFAIXA5CNFSM4DSCPCNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3GNEVY#issuecomment-516739671, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFAWNTDMGPHESOX7VTEHJYDQCFAIXANCNFSM4DSCPCNA.

[AlexeyUzhva]

We are also publishing events in batches. These events are on high load path (up to 50k events per second), so we can't afford duplicate checks for each via external source. So we're doing duplicate check for batches only, which is less. Though far from being ideal.

Also since we cannot pack batches on input side by any other meaningful criteria rather then just a timestamp, we need additional repartitioning after decomposing batch to meaningful events. To enable processing by their groups/accounts.

[GeorgeShagov]

Hello Magnus

1. Thanks for your great work!
2. LATOKEN CEX Exchange Trading Flow is based on Kafka.
3. In two words: Double Executions/Fills are Huuuge Problem.

thx
George

On 31/07/2019 10:53, Magnus Edenhill wrote:
>
>

HELP WANTED

To fully understand the user requirements and prioritize accordingly
we would like to ask you all to explain how you are aiming to use
EOS and Producer Transactions specifically.

So please take a moment to write down your use-case, this will help us
know what to implement and when.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/edenhill/librdkafka/issues/1308?email_source=notifications&email_token=AJNIJON3XD4ZLHJB3ECXM5DQCFAIZA5CNFSM4DSCPCNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3GNEVY#issuecomment-516739671,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJNIJOIB3DMP7ZRYFWUFIKDQCFAIZANCNFSM4DSCPCNA.

[BertusGreeff]

Our application uses Kafka to transfer state between two micro-services. We use Compacted Topics and it is critical that the topic always contains the latest message for each key.

Our primary need for Producer Transactions is to implement Zombie Fencing semantics as described here: https://www.confluent.io/blog/transactions-apache-kafka/

Client support needed to enable Zombie Fencing is the ability to associate a transactional.id with a producer and send a single message in a transaction.

[nleeuskadi]

we need transaction support to develop our system which is composed of "read-process-write" kafka clients. meaning that the majority of our software components read date from Kafka topics, process it and push the result in other kafka topics.
We need the "read-process-write" to be atomic

[ObviusOwl]

I'm writing an ETL system for emails (SMTP) in python providing a easy to use plugin interface. I would like to split the monolith into micro-services and replicas with kafka as backbone. EOL and full transactional behaviour would be a strong requirement to implement a reliable distributed producer-consumer architecture.

Confluent's python library relies on librdkafka: confluentinc/confluent-kafka-python#357

[alexpirine]

We're processing payments, we'd like to avoid double-spends.

[edenhill]

Available in v1.4.0, enjoy!