Kafkajs: Question about auto committing

[gilesbradshaw]

If I am using auto commit - is there a chance that a message will be committed when I haven't actually handled it?

eg

await consumer.run({ eachMessage: async ({ topic, partition, message }) => { await takeaverylongtimeifever() } )

could messages get auto committed before takeaverylongtimeifever is done?

[tulios]

@gilesbradshaw, the consumer won't commit in 2 cases:
1) user code throws an error (takeaverylongtimeifever throws)
2) the session times out, so the consumer has to rejoin the group

In general, you are in control. If you throw inside eachMessage it will interrupt the auto-commit

[gilesbradshaw]

Thanks for prompt reply :)

So as long as takeaverylongtimeifever fails before [session] is up it will never get committed .. ?

If that's right - what defines [session]? I can see options for requestTimeout and connectionTimeout..

[tulios]

Sorry for the long delay, when you create the consumer you can configure the session timeout, the heartbeat interval, etc. Take a look here https://kafka.js.org/docs/consuming#a-name-options-a-options

[tulios]

@gilesbradshaw do you still need help with this issue?

[gilesbradshaw]

Hi - I need to do a few experiments - basically to reassure myself that it's impossible for messages to be auto committed without them having been handled. For instance if they were queued up waiting to be handled. The crucial issue is whether I can guarantee all messages are handled. I'll get back to you if I find any issues - thanks for your help.

[Nevon]

This seems to be resolved for now, so I'm going to close this. Feel free to re-open it if you run into any issues.