Node-rdkafka: Managing timeouts on client connect

[jmwilkinson]

There is a common scenario when using docker compose to create a set of services that some will come up before others. This introduces a problem where the client for kafka can potentially try to connect before the cluster is actually active.

This scenario is described in issue #28.

Situation 2

1. Provide node-rdkafka two brokers in a comma separated list. Both brokers are down.
2. Call .connect to attempt to connect to the metadata.broker.list, one of which is down.
3. ready event is not called.
4. Start broker both brokers that were previously down.
5. node-rdkafka never attempts to reconnect.

The default timeout before deciding it is impossible to connect appears to be 30 seconds. I am trying to determine what parameter needs to be changed to lower that timeout, as it would be nice to retry much more frequently- timeout after 2 seconds, wait a few seconds, try again, etc.

I have done the following:

var start = Date.now();

let client = new kafka.Producer({
    'metadata.broker.list': brokers,
    'metadata.request.timeout.ms': 2000,
    'socket.timeout.ms': 2000,
    'session.timeout.ms': 2000
});

client.connect({
            'metadata.broker.list': brokers,
            'metadata.request.timeout.ms': 2000,
            'socket.timeout.ms': 2000,
            'session.timeout.ms': 2000
        }, (err, data) => {
            console.log(`cb, -> ${err}`)
            reject(err);
            let calculatedTimeout = (Date.now() - start) / 1000;
            console.log(`Timeout in seconds: ${calculatedTimeout}`);
        });

This achieves the result:

cb, -> Error: broker transport failure
Timeout in seconds: 30.019

I put the timeout parameters in both places, but neither have worked. So I do not believe I have the correct parameter to adjust the timeout, or I am still not setting it correctly, or this is not supported.

How can I reduce the connection timeout from 30 seconds to two seconds? Is this possible?

[jmwilkinson]

As an addendum to this, it appears that putting in the socket timeout socket.timeout.ms appears to help, but it doesn't actually matter what the value of the timeout is. The timeout failure will occur 10 seconds after starting the kafka broker, or 30 seconds after no contact. Without this parameter, it will fail after 40 seconds.

Very confused as to what is happening here.

[webmakersteve]

We do a similar thing in a docker environment. The only way to do it right is to create a script that blocks until Kafka is ready for connections. Otherwise, you will need to make your code resilient to boot up connection failures, which should generally cause the app to close (in my opinion).

Once you have successfully connected to node-rdkafka, the intent is that you never need to connect again. Disconnections will be managed internally so long as you have not manually called disconnect. When brokers go down, it will try to reconnect on an interval and your messages will be buffered (in the case of a producer) until it doesn't want to hold anymore.

However, there is a special case. The initial connect method will not retry. It tries to fetch metadata to ensure the connection is appropriately listening. If it can't get it within the metadata timeout set in the function (or 30 seconds by default), it will bail out and you will never be connected.

So you will either need to recursively call connect when connect gives an error in the callback, or make a script to block until kafka is available for connections. This can be done with nc if you just care that the port is open and the dns resolves.

[jmwilkinson]

If it can't get it within the metadata timeout set in the function (or 30 seconds by default),

What function are you referring to? I do recursive calls, I just want to shorten the timeout for the connection attempt to fail. I thought the metadata.request.timeout.ms would be used, but as stated I've tried using it and it makes no difference.

[webmakersteve]

That configuration parameters tells librdkafka how long to treat metadata as "fresh". The connect function has a second parameter that allows you to specify a metadata timeout.

https://blizzard.github.io/node-rdkafka/current/KafkaConsumer.html

connect({ timeout: 10000 }, (err, metadata) => {});

[jmwilkinson]

Ahh, that clarifies, things. Perhaps the documentation could be updated to reflect what the metadata options actually are? The current incantation leaves one guessing what to put there.

metadataOptions object Options to be sent to the metadata.

[webmakersteve]

I agree! I actually made a commit yesterday to update the docs. They will get pushed up to the gh-pages branch on the next release.

Is it okay if we close the issue or do you need help with anything else?

[jmwilkinson]

That's great news. I've put the timeout parameter in and verified that it works as expected. As always, thanks for your help.