Confluent-kafka-dotnet: Issue with plain authentication

[javsalgar]

Description

I tried using the example producer to connect to my Kafka broker using SSH forwarding and the connection gets stuck for a long time and nothing is written.

Partition: -1, Offset: Invalid [-1001]

In my server, I can produce using

/opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --producer.config /opt/bitnami/kafka/conf/producer.properties --topic testtopic

With producer.config like this:

bootstrap.servers=localhost:9092                               
compression.type=none          
security.protocol=SASL_PLAINTEXT                               
sasl.mechanism=PLAIN   

I also have this jaas configuration for the Producer

org.apache.kafka.common.security.plain.PlainLoginModule required 
username="user" 
password="E725YQacu7AA";" 

I tried the following configurations:

            var config = new Dictionary<string, object> { { "bootstrap.servers", brokerList }, 
                                                          { "security.protocol", "SASL_PLAINTEXT" },
                                                          { "compression.type", "none"},
                                                          { "sasl.mechanism", "PLAIN"}, 
                                                          { "sasl.username", "user" },
                                                          { "sasl.password", "E725YQacu7AA" }

This first did not work. The second:

            var config = new Dictionary<string, object> { { "bootstrap.servers", brokerList }, 
                                                          { "security.protocol", "SASL_PLAINTEXT" },
                                                          { "compression.type", "none"},
                                                          { "sasl.mechanism", "PLAIN"}, 
                                                          { "sasl.jaas.config", @"org.apache.kafka.common.security.plain.PlainLoginModule required username=""user"" password=""E725YQacu7AA"";" }

Returned this error:

Unhandled Exception: System.ArgumentException: Java JAAS configuration is not supported, see https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka for more information.
   at Confluent.Kafka.Impl.SafeConfigHandle.Set(String name, String value) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Impl/SafeConfigHandle.cs:line 116
   at Confluent.Kafka.Producer.<>c__DisplayClass23_0.<.ctor>b__2(KeyValuePair`2 kvp) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 282
   at System.Collections.Generic.List`1.ForEach(Action`1 action)                                                               
   at Confluent.Kafka.Producer..ctor(IEnumerable`1 config, Boolean manualPoll, Boolean disableDeliveryReports) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 282
   at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer, Boolean manualPoll, Boolean disableDeliveryReports) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 1044
   at Confluent.Kafka.Producer`2..ctor(IEnumerable`1 config, ISerializer`1 keySerializer, ISerializer`1 valueSerializer) in /tmp/confluent-kafka-dotnet/src/Confluent.Kafka/Producer.cs:line 1069
   at Confluent.Kafka.Examples.SimpleProducer.Program.Main(String[] args) in /tmp/confluent-kafka-dotnet/examples/SimpleProducer/Program.cs:line 47

Is there anything that I am missing?

How to reproduce

• Launch a Kafka Cluster with SASL Plain authentication enabled. For example here: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/bitnami.kafka-cluster
• Set up a SSH reverse tunnel to 9092.
• Try to launch the example Producer consumer with the authentication settings above

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version:
• [x] Apache Kafka version: 1.0.1-1
• [ ] Client configuration:
• [x] Operating system: Debian 8
• [ ] Provide logs (with "debug" : "..." as necessary in configuration)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

Run with "debug": "broker,security,protocol" to see what is going on

[javsalgar]

After enabling debug, I saw the issue. Sorry if this was something trivial but I am not experienced with Kafka:

 7|2018-04-11 09:40:43.333|rdkafka#producer-1|CLUSTERID| [thrd:main]: sasl_plaintext://127.0.0.1:9092/bootstrap: ClusterId updat
e "" -> "wCMiF1miRXqh5UEmhzlz8w"                                                                                               
7|2018-04-11 09:40:43.333|rdkafka#producer-1|BRKMAIN| [thrd:sasl_plaintext://kafkatest-kafka-1:9092/1001]: sasl_plaintext://kaf
katest-kafka-1:9092/1001: Enter main broker thread                                                                             
7|2018-04-11 09:40:43.333|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://kafkatest-kafka-1:9092/1001]: sasl_plaintext://kaf
katest-kafka-1:9092/1001: broker in state INIT connecting
7|2018-04-11 09:40:43.333|rdkafka#producer-1|BRKMAIN| [thrd:sasl_plaintext://kafkatest-kafka-0:9092/1002]: sasl_plaintext://kaf
katest-kafka-0:9092/1002: Enter main broker thread
7|2018-04-11 09:40:43.333|rdkafka#producer-1|CONNECT| [thrd:sasl_plaintext://kafkatest-kafka-0:9092/1002]: sasl_plaintext://kaf
katest-kafka-0:9092/1002: broker in state INIT connecting
7|2018-04-11 09:40:44.949|rdkafka#producer-1|BROKERFAIL| [thrd:sasl_plaintext://kafkatest-kafka-0:9092/1002]: sasl_plaintext://
kafkatest-kafka-0:9092/1002: failed: err: Local: Host resolution failure: (errno: Bad address)

So, the cause was:

• When returning the metadata after the first access, the client will try connecting to all the brokers using their hostname. Therefore, if not all the brokers are accessible from your client (using the domain name), then it will not work.

So the solution was to ditch the SSH tunnel and have the brokers be externally accessible in port 9092 and add their hostnames to my /etc/hosts (or get public domains for each host).

After that it was working, so closing the issue. Thanks for everything.