Php-rdkafka: Offset commit problem

[godtou]

• PHP version: 5.4.6
• librdkafka version: 0.9.1
• php-rdkfaka version: 1.0.0

I comment at issue86

[godtou]

OK, I found a global configuration $conf->set("enable.auto.offset.store", "false"); in librdkafka. This controls the offset store. The $topic->offsetStore method depend on this configuration.If this configuration is set to True(the default value is), the method make no affect. I check it in the test code in librdkafka.

I test both in high and low consumer.

While low level consumer doesn't have the offset commit method, it should set auto.offset.commit to be true. The high level consumer can commit and store offset by itself.

However I didn't find auto.offset.store in kafka's configuration. Since I'm new to kafka.I feel confused about these two configurations. Could you explain the difference between these two configurations?

Thank you

[godtou]

After I checked the code of librdkafka, I knew that:
rd_kafka_offset_store just set rktp->rktp_stored_offset while rd_kafka_commit_offset commit this offset to the broker or file.
Still confused with why auto.offset.store can affect the offset store and auto.offset.commit has no affect.

[arnaud-lb]

Hi @GodTou

If you want to commit offsets manually, the only thing you have to do is to set auto.commit.enable to false in the topic's configuration.

Note, however, that auto.commit.interval.ms (in the consumer conf) still affects how often the offsets are written to the offset storage. By default it's 60 seconds on librdkafka 0.8.

So, basically, $topic->offsetStore() will store the offset in memory, and librdkafka will periodically store the in-memory offset to the offset storage, at a frequency defined by auto.commit.interval.ms .

[godtou]

Thanks for your reply.

I already know what store and commit really do.

And literally， auto.offset.commit controls the commit , enable.auto.offset.store controls the store

If you don't set enable.auto.offset.store to be false, the offset still auto store in memory, and librdkafka will periodically commit the in-memory offset to the offset storage, at a frequency defined by auto.commit.interval.ms

The configuration enable.auto.offset.store can be found in here

It is a global configuration and the default value is true.

This based on my tests.

[xzt1981]

HI，can I use a high level consumer to manually commit offset?
and what does KafkaConsumer::commit do?

Thank you very much!

[liuchenrang]

I have the same situation. @arnaud-lb @GodTou , how to fix this problem.

[KaelenProctor]

I ran into this same issue as well. I found that setting 'enable.auto.commit' to 'false' on the RdKafka\Conf instance, instead of the RdKafka\TopicConf, worked when using the high-level consumer. Then the offset wasn't updated unless $consumer->commit() was called on the messaged processed.

[xiexiangnow]

这里的坑不少，这里分享下高层消费者手动提交的方式，在很多的地方都是说明的设置$topicConf->set('enable.auto.commit', 'false');然后应该设置$conf->set('enable.auto.commit', 'false');这个设置然后没有一个地方说明，望采纳！

[hrchu]

@GodTou enable.auto.offset.store controls the offset to be stored into broker instead of in memory, isn't it?

[godtou]

@hrchu Long time past, everything has changed. Please read https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
for some help, or just read the librdkafka source code.