Confluent-kafka-dotnet: ProduceAsync works locally but after deploying it won't

[leopinheirohk]

Description

Hi,

I'm creating a web api in c# .net core (v 2.1) which will be a kafka producer.

For that I have installed Confluent's .NET Client for Apache Kafka (v 1.3.0-PRE1).

After configuring and running locally I can successfully produce messages to Kafka. However after deploying my web api in Azure, it keeps waiting for a response for ProduceAsync until it times out.

Operation System: Windows 10

How to reproduce

1 - Create Kafka in https://confluent.cloud/. Only one cluster and one topic.
2 - Create a .net core web api V2.1 in visual studio 2017
3 - On the POST method configure and produce a message as follow:

            var config = new ProducerConfig
            {
                BootstrapServers = "BROKERNAME",
                ApiVersionRequest = true,
                BrokerVersionFallback = "0.10.0.0",
                ApiVersionFallbackMs = 0,
                SaslMechanism = SaslMechanism.Plain,
                SecurityProtocol = SecurityProtocol.SaslSsl,
                SaslUsername = "MYUSERNAME",
                SaslPassword = "MYPASSWORD"
            };

           producer = new ProducerBuilder<string, string>(config).Build();

           var deliveryReport = producer.ProduceAsync("topicName", new Message<string, string> { Key = Guid.NewGuid().ToString("N"), Value = Newtonsoft.Json.JsonConvert.SerializeObject(message) }).GetAwaiter().GetResult();

4 - Verify that it works fine locally.
5 - Publish the web api in Azure
6 - Verify that it times out in ProduceAsync

Checklist

Please provide the following information:

• [X] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [X] Confluent.Kafka nuget version.
• [ ] Apache Kafka version.
• [X] Client configuration.
• [X] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [X] Critical issue.

[mhowlett]

this will just be connectivity problem. you/we can get more insight by setting the Debug config property to all (and perhaps setting a custom log handler in the builder class). Feel free to paste the output of that here.

also, you should be able to get rid of the:

                ApiVersionRequest = true,
                BrokerVersionFallback = "0.10.0.0",
                ApiVersionFallbackMs = 0,

we should update those instructions.

[leopinheirohk]

Hi, sorry, but what kind of debug would be helpful for you?
I cannot see anything very useful on the outputs since there are only two scenarios:
1 - Successful when I do an asynchronous call with ProduceAsync (but no messages produced)
2 - Timeout when I do a synchronous call with ProduceAsync (no messages produced)

I have enabled some network trace in Azure and that is what generates

It seems like Azure (10.0.0.241:49818) and Kafka(40.119.203.68:9092) are able to communicate but I'm not sure why ProduceAsync doesn't work/timeout.

Another thing I did was to ping the Confluent Server within my azure web app and it seems it can establish communication

I have even enabled Application Insights in Azure but I couldn't find anything useful there.

Thanks,

Leo

[mhowlett]

if you get the full librdkafka debug logs (set the Debug config property to all), you'll get detailed logging of everything the client is doing - we'll most likely be able to determine what the problem is from that, or at least know specifically where to look next.

Some other things to try: Are you able to perform any other operation. e.g. create a topic or successfully do a metadata request with the AdminClient? Have you tried the Produce method (as opposed to the ProduceAsync method) - are you able to produce successfully using that?

[leopinheirohk]

Hi,

I have set the debug property as below:

            var config = new ProducerConfig
            {
                BootstrapServers = brokerName,
                SaslMechanism = SaslMechanism.Plain,
                SecurityProtocol = SecurityProtocol.SaslSsl,
                SaslUsername = "MYUSERNAME",
                SaslPassword = "MYPASSWORD",
                Debug = "all"
            };

However I don't see anything being logged related to Kafka.
Where is it supposed to generate information?
I Debug = "all" enough to create logs or I'm supposed to do something else?

Also I tried calling Produce as opposed to ProduceAsync and I get the same outcome.

One more thing. I opened an incident with Azure to investigate this situation and the only thing they can think of is that Confluent has some restrictions on local private IPs. As I mentioned before that IP Azure is using to connect to Kafka was 10.0.0.241.
Is there such limitation?

Later I will try those other operations and see if I'm successful. Thanks.

Regards,

Leo

[mhowlett]

by default the log messages should be written to stderr, however you can also set a custom log handler using the builder class.

you will get a lot of messages... it's very verbose.

[leopinheirohk]

Ok cool I got good new.

Here is the custom log:

INFOKAFKA: [thrd:app]: Selected provider PLAIN (builtin) for SASL mechanism PLAIN
INFOKAFKA: [thrd:app]: librdkafka built with OpenSSL version 0x1000212f
INFOKAFKA: [thrd:app]: Failed to open Windows certificate Root store: Access is denied...: falling back to OpenSSL default CA paths
INFOKAFKA: [thrd:app]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Enabled low-latency ops queue wake-ups
INFOKAFKA: [thrd:app]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Added new broker with NodeId -1
INFOKAFKA: [thrd:app]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
INFOKAFKA: [thrd:app]: librdkafka v1.2.1 (0x10201ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0xffff)
INFOKAFKA: [thrd::0/internal]: :0/internal: Enter main broker thread
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Enter main broker thread
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2364
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:app]: New local topic: tdemoav
INFOKAFKA: [thrd:app]: NEW tdemoav [-1] 03075B30 (at rd_kafka_topic_new0:393)
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 405ms in state CONNECT)
ERRORKAFKA: Local_AllBrokersDown: 1/1 brokers are down
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 1 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2412
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 626ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 2 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2412
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 407ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 3 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2416
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 410ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 4 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2412
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 409ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 5 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Delaying next reconnect by 1352ms
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2416
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 410ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 6 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Delaying next reconnect by 3054ms
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connecting to ipv4#40.119.203.68:9092 (sasl_ssl) with socket 2412
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Connected to ipv4#40.119.203.68:9092
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: No such file or directory)
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Purging bufq with 0 buffers
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Updating 0 buffers on connection reset
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state DOWN -> INIT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 424ms in state CONNECT)
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Selected for cluster connection: refresh unavailable topics (broker has 7 connection attempt(s))
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Received CONNECT op
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: Broadcasting state change
INFOKAFKA: [thrd:sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/b]: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: Delaying next reconnect by 5696ms
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
INFOKAFKA: [thrd:main]: Topic tdemoav metadata information unknown
INFOKAFKA: [thrd:main]: Topic tdemoav partition count is zero: should refresh metadata
INFOKAFKA: [thrd:main]: tdemoav: 1 message(s) timed out
INFOKAFKA: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
INFOKAFKA: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
INFOKAFKA: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection

As you can see there was an error saying that the handshake failed.

What I did was I added EnableSslCertificateVerification as follow:

        var config = new ProducerConfig
        {
            BootstrapServers = brokerName,
            SaslMechanism = SaslMechanism.Plain,
            SecurityProtocol = SecurityProtocol.SaslSsl,
            SaslUsername = "MYUSERNAME",
            SaslPassword = "MYPASSWORD",
            Debug = "all",
            EnableSslCertificateVerification = false
        };

And now it is working.

Would you be able to help me understand what this is happening and why this problem only occurs when I publish it to Azure? Thanks a lot.

[mhowlett]

ok, makes sense. disabling ssl cert verification is dangerous because it leaves you susceptible to man-in-the-middle attacks. what operating system is your client running on? Is it old? It seems like you don't have the root CA certificates on your machine that can be used to validate the cert sent by the server. You can download the ones distributed with curl here: https://curl.haxx.se/docs/caextract.html which should work and specify using the SslCaLocation property.

Let me know if that works. Also, any information on your operating system would be good. If this is going to be a common problem, I'll bring it up with the cloud people.

[leopinheirohk]

It works fine after downloading the certificate and passing it through SslCaLocation.

So the production environment in this case is the Azure App Service is a platform-as-a-service (PaaS) offering of Microsoft Azure.
By using this we do not have access to the server. All we define is whether it will be Windows or Linux.

I believe that anyone who will use Azure App Service to connect to Confluent Kafka will have the same issue.

[edenhill]

The problem seems to be access to the Root store:
INFOKAFKA: [thrd:app]: Failed to open Windows certificate Root store: Access is denied...: falling back to OpenSSL default CA paths

[darknasty]

It works fine after downloading the certificate and passing it through SslCaLocation.

I experienced the same issue you describe, and was able to get it to work by setting:
EnableSslCertificateVerification = false

You mention above that you downloaded the cert and got it working by downloading the cert and passing it through SslCaLocation. Would you mind sharing the config line that sets the SslCaLocation?

[leopinheirohk]

It works fine after downloading the certificate and passing it through SslCaLocation.

I experienced the same issue you describe, and was able to get it to work by setting:
EnableSslCertificateVerification = false

You mention above that you downloaded the cert and got it working by downloading the cert and passing it through SslCaLocation. Would you mind sharing the config line that sets the SslCaLocation?

    var config = new ProducerConfig
    {
        BootstrapServers = brokerName,
        SaslMechanism = SaslMechanism.Plain,
        SecurityProtocol = SecurityProtocol.SaslSsl,
        SaslUsername = "MYUSERNAME",
        SaslPassword = "MYPASSWORD",
        Debug = "all",
        SslCaLocation = "cacert.pem"
    };

cacert.pem is in the root folder of my app and I downloaded it from https://curl.haxx.se/docs/caextract.html

[darknasty]

That worked. Thanks!

[dann41]

This issue saved my day! Thank you very much

[DanConway]

We've just had a similar issue running a Kafka producer in dotnet core hosted on IIS.

Setting "Load User Profile" to true for the application pool got it working

[davidmrdavid]

Has anyone had the experience of their Azure App working up until recently and then stopped working with this same exception? Perhaps @DanConway ?

[mhowlett]

"Failed to open Windows certificate Root store: Access is denied...: falling back to OpenSSL default CA paths". On windows, CA certs won't be found via the default CA paths unless explicitly configured, so something else should happen i think. Marking this issue as enhancement because I think this problem should result in a more catastrophic failure (exception) on startup. Another client creation API would be required to enable this though.

@davidmrdavid - you mean the following log message: "ERRORKAFKA: Local_Ssl: sasl_ssl://pkc-4j8dq.southeastasia.azure.confluent.cloud:9092/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 405ms in state CONNECT)"? try the curl certs.

[davidmrdavid]

@mhowlett
I didn't have any specific log messages in mind other than the original Failed to open Windows certificate Root store: Access is denied...: falling back to OpenSSL default CA paths exception mentioned in this thread. Just trying to validate my own experience.

I do think a different exception on the client side would be helpful, perhaps one that suggests the fix outlined and nature of the error as outlined in this thread :)

[joedyndale]

@davidmrdavid I experienced this with an Azure Function App. It worked fine up to 18:00 UTC on May 12th, at which point Microsoft did a platform upgrade which affected some blades, and after that I had this issue. Using the cURL cert file solved it, but we don't want to have to maintain certs ourselves. As @DanConway mentioned, loading a user profile solved the access to the root ca store.

For the function app the relevant solution was setting the application setting "WEBSITE_LOAD_USER_PROFILE" to "1".

Microsoft has said they are "looking into diagnostics improvements to see if we can call out the situations when a User Profile may be required for SSL operations with third party libraries."

[KyleMccr]

For info, I have same issue and error messages using Confluent Cloud with dotnet3.1 on MacOS.

%7|1592856510.525|CONNECT|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: Connecting to ipv4#3.230.179.163:9092 (sasl_ssl) with socket 211
%7|1592856510.564|CONNECT|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: Connected to ipv4#3.230.179.163:9092
%7|1592856510.619|BROKERFAIL|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: Undefined error: 0)
%3|1592856510.619|FAIL|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: SSL handshake failed: s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 163ms in state CONNECT)
%3|1592856510.619|ERROR|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: SSL handshake failed: s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: (after 163ms in state CONNECT)
%7|1592856510.619|STATE|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
%3|1592856510.619|ERROR|Kyles-Laptop.local#producer-1| [thrd:sasl_ssl://pkc-4nym6.us-east-1.aws.confluent.cloud:9092/bootstr]: 1/1 brokers are down

[mhowlett]

@KyleMccr - did you resolve this?

There's some internal discussion about this (but relating to the go client). The suggestion was to install openssl with brew, to which the response was:

Turns out, on my laptop there’s only [email protected] even with brew install, no presence of openssl . I created a symlink then problem solved!

[KyleMccr]

I did the workaround that you suggested above -- download cacert.pem and configure using SslCaLocation. That worked perfectly at least for the short term.

I'll see if I can upgrade openssl and let you know if that works too.

[saherahwal]

I am facing similar issue I had to create cacert.pem and copy Public key signatures there. added that file to the root of the .net core app.
Why can't it read the cert directly from the store in Windows?

even though I see this debug log

[18:12:40 Debug][1]  MySerivice.Messaging.Kafka.EventBusConsumer
[thrd:app]: 72/73 certificate(s) successfully added from Windows Certificate Root store

I then see this error

[18:12:44 Debug][12]  MyService.Messaging.Kafka.EventBusConsumer
[thrd:sasl_ssl://brokern22p.domain.MyService.corp:9094/bootstrap]: sasl_ssl://brokern22p.domain.MyService.corp:9094/bootstrap: failed: err: Local: SSL error: (errno: No error)

18:12:44 Error][7]  MyService.Messaging.Kafka.EventBusConsumer
errorCode: Local_Ssl, reason: sasl_ssl://brokern22p.domain.MyService.corp:9094/bootstrap: SSL handshake failed: .\ssl\s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:  (after 84ms in state CONNECT)

[vemoreno]

ok, makes sense. disabling ssl cert verification is dangerous because it leaves you susceptible to man-in-the-middle attacks. what operating system is your client running on? Is it old? It seems like you don't have the root CA certificates on your machine that can be used to validate the cert sent by the server. You can download the ones distributed with curl here: https://curl.haxx.se/docs/caextract.html which should work and specify using the SslCaLocation property.

Let me know if that works. Also, any information on your operating system would be good. If this is going to be a common problem, I'll bring it up with the cloud people.

My class ProduceConfig not has such property: EnableSslCertificateVerification

Does Kafka removed the property in new versions?