Confluent-kafka-dotnet: Not able to connect kafka with dotnet client

[Amitssh]

Hi, we are new to confluent-kafka-dotnet, and recently started testing it as our Kafka consumer client.

It would be great if you can assist me on how to configure connection with kafka with kafka dotnet. Could I be able to configure the connection string?

[treziac]

bootstrap.servers should be like this
"kafkaHost1:9092,kafkaHost2:9092,10.0.75.1:9093" => comma separated list of host:port
You don't have to provide all the brokers, just a subset to initialize the connection to brokers. The client will automatically connect to all brokers in the cluster from this subset. Be sure to provide at least 2 or 3 in case one is down. You can provide all if you want, but don't mix brokers from different cluster

You can take a look at the AdvancedConsumer sample:
https://github.com/confluentinc/confluent-kafka-dotnet/blob/master/examples/AdvancedConsumer/Program.cs

Basically, you need a dictionary config, which is based on librdkafka settings (all config values can be found here : https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

Group.Id and bootstrap.servers are compulsary for the consumer.

[Amitssh]

I provided bootstrap.servers as "localhost:2181, localhost:2183"
However, I'm not able to get the Topic and the brokerlist in the output. It would be nice if more detailed ways of kafka testing step is provided.
Also, do I need to keep running kafka server simultaneously?
Thank you much.

[treziac]

2181 is likely zookeeper. You need to provide kafka brokers

[treziac]

Confluent.Kafka (and the wrapped C library librdkafka) know nothing about zookeeper - they only dialog with brokers. You might want localhost:9092

Also, do I need to keep running kafka server simultaneously?

What do you mean? You should always have your zookeeper and server up yes. Typically they will run as service on a specific machine in production, and you will have a cluster of zookeeper (typically 3 server) and a server of kafka brokers (typically 3 to as many as you want). In Dev you can simply have one zookeeper and one broker.

[Amitssh]

I provided bootstrap.servers as "0.0.0.0:9092, 0.0.0.0:135, 0.0.0.0:80"
Outcome is zookeeper trying to establish connection but not able to establish. At my consumer console getting error as
"Error: localhost:2181/bootstrap: Connection Closed
Error: 1/1 brokers are down"

Please help me out resolving this error.
Thank you.

[treziac]

Did you follow the quickstart on either confluent or Kafka with the console producer/consumer? Once you manage to make it work here, it will be easier to come back to a more advanced client

[Amitssh]

On the Kafka with the consumer/producer console I'm able to communicate. But, when coming to confluent console, connection is not getting established and zookeeper/confluent consumer is always connecting to localhost:2181 but the same is refused.

[treziac]

Can you copy here the command you use for console consumer/producer, both with Kafka and confluent binaries? The binary should be almost identical between Kafka and confluent. Also please provide your os and Kafka/confluent version.

[Amitssh]

command for kafka consumer & producer

kafka-console-producer.bat --broker-list localhost:9092 --topic test
kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

confluent console running on debug mode in VS2017 as:

Advance producer: brokerlist = "localhost:9092"
Advance consumer: brokerlist= "localhost:2181"

OS: Windows server 2012 R2 datacenter (as of now)
Kafka ver: kafka_2.12-0.10.2.1
confluent ver: Confluent.kafka 0.9.5

[treziac]

Kafka scripts are not supported for windows on confluent platform, I plan to correct this if possible with confluent guys to make them working as kafka package does ( @mhowlett =D )

Try to use kafka-console-consumer with --bootstrap-server instead of --zookeeper (you should have a warning saying it's obsolete) ie
kafka-console-consumer.bat --bootstrap-server localhost:9092--topic test
To simplify, Confluent.Kafka use this new consumer implementation with broker

If this is working, try again Confuent.Kafka with bootstrap server at "localhost:9092" only
If you are running Confluent.Kafka on an other machine than your kafka server, be sure to check the console consumer/provider works also on this machine (no firewall issues)

Also, did you try the producer? It's simpler to use first, to check everything works fine

[Amitssh]

@treziac Thanks so much for your time and support. I am successfully able to establish connection.