Confluent-kafka-go: Kafka consumer gets stuck after exceeding max.poll.interval.ms

[im-abeer]

Description

When the consumer does not receives a message for 5 mins (default value of max.poll.interval.ms 300000ms) the consumer comes to a halt without exiting the program. The consumer process hangs and does not consume any more messages.

The following error message gets logged

MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 255ms (adjust max.poll.interval.ms for long-running message processing): leaving group

I see that ErrMaxPollExceeded is defined here but unable to find where it is getting raised.

If any such error is raised, why does the program not exit ?

Checklist

Please provide the following information:

• [X] confluent-kafka-python and librdkafka version (confluent_kafka.version(master) and confluent_kafka.libversion(1.0.0)):
• [X] Apache Kafka broker version: v1.1.0
• [X] Client configuration: { "bootstrap.servers": "my.kafka.host", "group.id": "my.group.id", "auto.offset.reset": "earliest", "enable.auto.commit": false }
• [ ] Operating system:
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

Unless you are using the channel consumer (which you shouldn't use), you need to call Poll() or ReadMessage() at least every max.poll.interval.ms-1.

[edenhill]

See the "max.poll.interval.ms is enforced" chapter here: https://github.com/edenhill/librdkafka/releases/v1.0.0

[keyan]

Hello @edenhill, I’m running into a similar issue as the original poster, I’m using a -1 timeout but calling in an infinite loop, e.g.:

for {
        msg, err := consumer.ReadMessage(-1)
        if err != nil {
                <do some logging>
        }
}

My producer stopped writing messages for a few minutes and I logged this:

Application maximum poll interval (300000ms) exceeded by 164ms

Subsequently my producer was back up but the consumer seemed to be hanging on ReadMessage(-1) indefinitely.

According to the doc you linked to:

requires the application to call rd_kafka_consumer_poll()/rd_kafka_poll() at least every max.poll.interval.ms. Failure to do so will make the consumer automatically leave the group […] and not rejoin the group until the application has called ..poll() again

I’d expect that my consumer did indeed leave the group, but the subsequent call to ReadMessage() should have made the consumer rejoin the group and continue to see new messages.

---

Is this a configuration issue? Would using a shorter timeout for ReadMessage() resolve this?

Or is this a manifestation of https://github.com/edenhill/librdkafka/issues/2266?

librdkafka version: 1.0.0
confluent-kafka-go version: v1.0.0
Client configuration: ["auto.offset.reset": "earliest", "enable.auto.commit": false]

[edenhill]

This looks like https://github.com/edenhill/librdkafka/pull/2365/commits/80e9b1ee0c0e477c02f18b130c4302240711a88d which is fixed in librdkafka 1.1.0.

[csrgxtu]

i am having this issue too, how to fix this anyway?

[szalapski]

I am having this issue with librdkafka 1.5.0, exactly as keyan said. Can anyone help?

[rogaha]

having this this issue as well with v1.4.2. Any hints?

[rogaha]

my code:

for {
        select {
        case <-ctx.Done():
            err = kc.Close()
            if err != nil {
                r.Logger.Error(err)
                panic(err)
            }
            r.Logger.Info("Done")
            return nil
        default:
            msg, err := kc.ReadMessage(100 * time.Millisecond)
            if err == nil {
                if err := p.Perform(ctx, msg.Value); err != nil {
                    r.Logger.Error(err)
                }
            }
        }
    }

[keyan]

To clarify, are you all seeing that your consumer won't rejoin on the subsequent ReadMessage() call? Just seeing the poll interval exceeded message is not abnormal or unexpected.

FWIW, after upgrading to the v1.1.0 client and also changing from a -1 to a sane large timeout, I stopped after rejoining issues. But leaving the consumer group still happens as is expected.

[jgao54]

Using v1.5.2. Also calling ReadMessage(-1) in an infinite loop, and not seeing rejoining after consumer leaving group, worked around it by setting timeout to be less than max.poll.interval.ms instead of -1, but wondering why it's not rejoining as expected.

[rogaha]

I'm using v1.5.2 and the issue persists. The worker is "dead" after receiving the "leave group" error:

%4|1614171022.957|MAXPOLL|rdkafka#consumer-10| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 123ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171022.960|MAXPOLL|rdkafka#consumer-8| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 481ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171022.980|MAXPOLL|rdkafka#consumer-2| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 423ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171023.957|MAXPOLL|rdkafka#consumer-3| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 500ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171024.447|MAXPOLL|rdkafka#consumer-7| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 84ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171025.955|MAXPOLL|rdkafka#consumer-5| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 195ms (adjust max.poll.interval.ms for long-running message processing): leaving group
%4|1614171025.961|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 55ms (adjust max.poll.interval.ms for long-running message processing): leaving group

Restarting the worker fixes the problem. It seems to be a queue management issue.

[edenhill]

What means of consuming messages are you using? Channel consumer? ReadMessage? Poll?

[rogaha]

ReadMessage

[edenhill]

And you are calling ReadMessage() more often than the max.poll.interval.ms (30s) ?

[rogaha]

yes:

for {
        select {
        case <-ctx.Done():
            err = kc.Close()
            if err != nil {
                r.Logger.Error(err)
                panic(err)
            }
            r.Logger.Info("Done")
            return nil
        default:
            msg, err := kc.ReadMessage(100 * time.Millisecond)

[gabrielhartmann]

@edenhill Why is this issue closed? I'm seeing the same thing as everyone else, also calling ReadMessage quickly in an infinite loop.

I personally see this issue associated with a broker rebalance. On another thread prior to the stuck consumer symptom everyone else reports I see this:

msg="failed to commit message with error: Broker: Group rebalance in progress" consumer=default

[AdbReverse]

I am having the same issue as well

[AdbReverse]

There is something weird in the code.
Is it possible that we are constantly getting ev=nil return value from Poll()?
Then, the function would stuck on an infinite loop