Kafkajs: Add way to commit offset with Consumer

[JaapRood]

In various other clients, like the official KafkaConsumer, arbitrary offsets can be committed through an interface like consumer.commit. So far, the only way to do this in KafkaJS is through commitOffsetIfNecessary function exposed as part of the eachBatch mechanism.

This limitation prevents the committing of offsets before a message has been consumed, which is a time and place where in stream processing tasks it's desirable to do so. While in most cases a consumer.seek operation is desirable, there are situations where you just want to set the progress made and not consume any messages at all.

Another example is actually present in the admin client's 'setOffsets' method, where a seek and then processing of messages must happen in order for the auto commit to eventually commit the right offset.

Proposed solution

Introduce a consumer.commit or consumer.commitOffsets method, like the official KafkaConsumer does. Like consumer.seek, it could limit it's calling to only after consumer.run has established the internal consumerGroup.

[JaapRood]

In a thread on slack, @tulios expressed some concerns about how the fetching mechanism might be affected:

Another problem is that the consumer loop only fetches the offsets in certain conditions, it always increments the offsets and commits, so if you commit outside of the loop it might be quickly overridden

But also seemed confident the following would fix that:

when resuming a consumer it should load the offsets again

[JaapRood]

As with #373 I'm happy to take give this a go, but will be delayed in doing so as we're still trying to see how much of our stream processing tasks we can get running against KafkaJS, before going back and fixing this. I'll be sure to announce it here when I get around to it, hoping others will too if they get around to it.

[Hann]

Hi. Is there a release plan included this feature?
I found the feature on the docs but It seems to be unreleased.
https://kafka.js.org/docs/consuming#a-name-manual-commits-a-manual-committing

[JaapRood]

The release isn't out yet, but it should be in the next one. Meanwhile, you can depend on the commit sha (see https://docs.npmjs.com/cli/install). There is no build step for the repo and it's how I run earlier releases.

@tulios just wanted to tag you here to indicate another case where an automatic pre-release might smoothen things over. Also, on the disconnect between documentation and the current stable release, maybe we should be marking features with the version at which they're available? Might get a bit messy :/

Update: actually just found #279, that should address the docs disconnect as well.