Kafka-node: BrokerNotAvailableError

[healtho]

Bug Report

Get the below issue sometime with kafka producer

{ BrokerNotAvailableError: Broker not available
    at new BrokerNotAvailableError (<project-folder-node-modules>/kafka-node/lib/errors/BrokerNotAvailableError.js:11:9)
    at Socket.<anonymous> (<project-folder-node-modules>/kafka-node/lib/client.js:728:37)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at TCP._handle.close [as _onclose] (net.js:567:12) message: 'Broker not available' }
{ BrokerNotAvailableError: Broker not available
    at new BrokerNotAvailableError (<project-folder-node-modules>/kafka-node/lib/errors/BrokerNotAvailableError.js:11:9)
    at Socket.<anonymous> (<project-folder-node-modules>/kafka-node/lib/client.js:728:37)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at TCP._handle.close [as _onclose] (net.js:567:12) message: 'Broker not available' }

Environment

• Node version: v8.11.1
• Kafka-node version: 2.6.1
• Kafka version:0.9.1
• Number of Brokers:2
• Number partitions for topic:2

Sample Code used

var client = new Client(kafkaConfig, id);

var producer = new Producer(this.client, kafkaConfig.options);

var that = this;
producer.on('ready', function () {
    /** handler */
});

producer.on('error', function (err) {
    /** handler */
})

Thanks for your help! let me know for any details required

[leadhkr]

+1 having the same issue as the OP. Running v2.2.2

[hyperlink]

Is kafkaConfig a string to your zookeeper host?

[gitcloned]

Yes, this is how the kafkaConfig looks like:

{
        zookeeper: '192.168.77.55:2181,192.168.77.56:2181,192.168.77.52:2181',
        ip: '192.168.77.61:2181',
        brokers: '192.168.77.60:6667,192.168.77.61:6667,192.168.77.58:6667'
    }

[ashwinkhai]

Hi All,
Facing a similar issue for similar code. Was there any resolution?

[manesioz]

Also facing this issue!

[jaiswarvipin]

My docker-compose file

version: '3'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      ES_JAVA_OPTS: "-Xms512m -Xmx3000m"

and Producer code is

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.KafkaClient({kafkaHost:"localhost:9094"}),

    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'topic1', messages: 'hi', partition: 0 },
        { topic: 'topic1', messages: ['hello', 'world', km] }
    ];

client.createTopics(topicsToCreate, (error, result) => {

                echo ("------------------------KAFAK--------------------")
                console.log(error);
                console.log(result);            
            });

Getting below error, while creating the topic before sending pay-load to topics

{ BrokerNotAvailableError: Broker not available (loadMetadataForTopics)
    at new BrokerNotAvailableError (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\errors\BrokerNotAvailableError.js:11:9)
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:21)
    at KafkaClient.loadMetadata (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:876:8)
    at KafkaClient.getController (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:267:8)
    at KafkaClient.sendControllerRequest (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:1219:8)
    at KafkaClient.createTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:935:8)
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\index_behind_kong.js:60:11
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:370:18
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:12)
    at RetryOperation._fn (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:360:12) message: 'Broker not available (loadMetadataForTopics)' }
undefined

I have gave the delay of 5 seconds before calling to createtopic method, however no luck found.

Kindly assist.

[jaiswarvipin]

Finally got the solution. I was doing on mistake. By default Apache Kafka create the 3 replication, this mean its by default create 3 broker as well. However aboe YAML create only one broker, and looknig for other 2 which is not created. hence we are getting the same error.

Fix

version: '3'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_LISTENERS: INTERNAL://:9092,OUTSIDE://:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      ES_JAVA_OPTS: "-Xms512m -Xmx3000m"

then KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 tell the Apache kafka that we Wanted to create only one replication and its work

Happy coding :)

Thanks & Regards
Jaiswar Vipin Kumar R.