Node-rdkafka: Does rdKafka expose the topic creation from the c++ client?

[corybill]

Hello, I was wondering if topic creation is exposed as well as the ability to configure partitions and replication factor on create.

Also, is topic altering exposed as well as the ability to alter the number of partitions when altering a topic.

I see a topic creation function in the producer but it also says that it only creates / manages this object in V8. So I am not sure this is actually creating the topic in Kafka.

[webmakersteve]

A little confused about in what capacity you mean "topic creation". If you mean topic creation, as in making a topic on a list of brokers by connecting to Zookeeper, then no. It does not support that because librdkafka does not support that.

https://github.com/edenhill/librdkafka/blob/master/src-cpp/rdkafkacpp.h#L1111 shows a Topic::create method, but this topic is a topic handle. It is a reference to a topic for use in producing (or in the old consumer client, consuming). This does not make a topic on your broker, but is just a way to reference an existing topic in your code. This functionality is indeed exposed.

const kafka = require('node-rdkafka');

const producer = new Kafka.Producer(...);
const topicConfig = {};

producer.connect()
   .on('ready', () => {
      const topic = producer.Topic('topicName', topicConfig);
   });

This lets you set a few settings about how the topic is used, such as acks. You can see a full list here https://github.com/edenhill/librdkafka/blob/2213fb29f98a7a73f22da21ef85e0783f6fd67c4/CONFIGURATION.md

[corybill]

Thank you, that answers my question. We have a service right now that wraps the Confluent Command Line tools which allows us to create / alter topics topics in Kafka through a rest endpoint. We were wondering if we could remove that service from our world if we moved to rdKafka. We will need to keep it, if / when we make the move to rdKafka. Thanks for the insight.

[edenhill]

@corybill When full KIP-4 support is added to librdkafka you'll be able to create and delete topics programmatically from your client.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations

[corybill]

Nice, thanks for the update @edenhill!

[homer6]

@edenhill It looks like KIP-4 support is nearly complete. After it has been completed, do you expect much of a delay on the librdkafka side?

[sureshappana]

Hi, I wonder now we have a function call or anyway to create topics automatically from producer side (if not exists) using node-rdkafka