Confluent-kafka-dotnet: Commiting offsets one by one vs commiting offsets in batch from perspectives of Performance.

Created on 27 Jul 2020 · 1Comment · Source: confluentinc/confluent-kafka-dotnet

Description

I would like to confirm which is the best way of committing offsets manually from the perspectives of Performance.

Commit the offset one by one i.e. once you consume and done with processing, commit the offset
Commit the offsets in a batch i.e. consume 10 messages, process them and commit the offset for these 10 at one go.

How to reproduce

Checklist

Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[ ] Confluent.Kafka nuget version.
[ ] Apache Kafka version.
[ ] Client configuration.
[ ] Operating system.
[ ] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[ ] Critical issue.

question

Source

BibhutiBhusanDora

Most helpful comment

if you consume 10 messages, you only need to commit once corresponding to the final message, not each one, and this will be 10x as efficient as committing after every message.

side note: the best way to achieve at least once semantics is to set AutoOffsetStore to false and AutoOffsetCommit to true, and use StoreOffsets to mark offsets as eligible to commit. A background thread will periodically commit them at interval AutoCommitIntervalMs.

mhowlett on 27 Jul 2020

👍2

>All comments

if you consume 10 messages, you only need to commit once corresponding to the final message, not each one, and this will be 10x as efficient as committing after every message.