kafkajs 🚀 - Connection error to kafka hosted on confluent cloud with v1.14

We run the latest KafkaJS beta continuously against a Confluent Cloud Kafka cluster hosted on AWS, and have not seen any such issues. I think it's extremely unlikely that this is an issue with KafkaJS itself, but rather some associated change you rolled out at the same time, or an environmental issue of some kind.

Nevon on 21 Oct 2020

@mdespriee @Nevon We have been getting the same error since yesterday at 3 PM CEST. We have tried all versions of KafkaJS (including beta), and we're unable to connect both from local machines and our hosting environment. We're connecting to the same broker endpoint as you (pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092).
We can connect to our cluster using Kafka CLI, but not with KafkaJS.

This is the errors we're getting with KafkaJS:

    [BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout {
      timestamp: '2020-10-21T07:57:15.628Z',
      logger: 'kafkajs',
      retryCount: 2,
      retryTime: 1006
    }

    [Connection] Connection error: Client network socket disconnected before secure TLS connection was established {
      timestamp: '2020-10-21T07:57:16.419Z',
      logger: 'kafkajs',
      broker: 'pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092',
      clientId: 'xxxxxxxxx',
      stack: 'Error: Client network socket disconnected before secure TLS connection was established\n' +
        '    at connResetException (internal/errors.js:610:14)\n' +
        '    at TLSSocket.onConnectEnd (_tls_wrap.js:1546:19)\n' +
        '    at Object.onceWrapper (events.js:421:28)\n' +
        '    at TLSSocket.emit (events.js:327:22)\n' +
        '    at endReadableNT (_stream_readable.js:1220:12)\n' +
        '    at processTicksAndRejections (internal/process/task_queues.js:84:21)'
    }

It is still failing to connect at this moment.

@mdespriee Have you managed to resolve this issue?

renklaf on 21 Oct 2020

Oh wow, you are right, and our monitoring is clearly busted because I just logged into AWS to check on our application, and it's having the same issue (without any alarms, yay!)

One line of inquiry is to figure out if this is an issue with KafkaJS that was introduced sometime after 1.12.0 or if this is a change on the confluent cloud side of things. I'm gonna do some experimentation to find out.

Nevon on 21 Oct 2020

Looking in our logs, I can see this happening at 2020-10-20T13:23:01.263Z for the first time, which corresponds to the 3 PM CEST that you're reporting, @renklaf. Given that this happened at around the same time for both of us, this clearly indicates that some kind of configuration change must have been rolled out on the confluent cloud side.

I don't have any access to Confluent Cloud support, so I don't have a way to get more information from them, but I have at least reached out in their Slack channel. I'm not really expecting any response though, as it's usually quite dead. If anyone has support access, please inquire with them and direct them to this thread if possible.

Nevon on 21 Oct 2020

@Nevon.
We're subscribing to their status and received the following yesterday @ 2.20 PM CEST:

Confluent Cloud
Performance degradation for Basic and Standard clusters in AWS eu-central-1
Incident status: Monitoring
A fix has been implemented and we are monitoring the results.
Time posted
Oct 20, 07:19 CDT
View 2 previous incident updates
You received this email because you are subscribed to Confluent Cloud's service status notifications._

I agree that they must have changed something on their side, given the fix they implemented yesterday, but I still find it odd that we are able to connect from kafka CLI and not KafkaJS.

We have already created an issue with Confluent Cloud support and also linked them this Github issue.
I'll let you know as soon as we get any updates.

renklaf on 21 Oct 2020

Added some more debug output by running with NODE_DEBUG=tls as well as a custom socket factory with tracing enabled on the socket.

debug: ┏ [Connection] Connecting
debug: ┃ [0] {
debug: ┃ [1]   "timestamp": "2020-10-21T09:02:11.239Z",
debug: ┃ [2]   "logger": "kafkajs",
debug: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
debug: ┃ [4]   "clientId": "test-admin-id",
debug: ┃ [5]   "ssl": true,
debug: ┃ [6]   "sasl": true
debug: ┗ [7] }
TLS 49146: client _init handle? true
TLS 49146: client initRead handle? true buffered? false
TLS 49146: client _start handle? true connecting? false requestOCSP? false
Sent Record
Header:
  Version = TLS 1.0 (0x301)
  Content Type = Handshake (22)
  Length = 261
    ClientHello, Length=257
      client_version=0x303 (TLS 1.2)
      Random:
        gmt_unix_time=0xEC4DD66B
        random_bytes (len=28): 0101D49BD07D6A0A417A07C681927EA9FC1E34366AC7EE60AD82195B
      session_id (len=32): 1C92B4BDF83CCFD0BFF901614333ABE6FCB870E4892116A0F7FA1C86FC486A5D
      cipher_suites (len=8)
        {0x13, 0x02} TLS_AES_256_GCM_SHA384
        {0x13, 0x03} TLS_CHACHA20_POLY1305_SHA256
        {0x13, 0x01} TLS_AES_128_GCM_SHA256
        {0x00, 0xFF} TLS_EMPTY_RENEGOTIATION_INFO_SCSV
      compression_methods (len=1)
        No Compression (0x00)
      extensions, length = 176
        extension_type=server_name(0), length=47
          0000 - 00 2d 00 00 2a 70 6b 63-2d 6c 71 38 76 37 2e   .-..*pkc-lq8v7.
          000f - 65 75 2d 63 65 6e 74 72-61 6c 2d 31 2e 61 77   eu-central-1.aw
          001e - 73 2e 63 6f 6e 66 6c 75-65 6e 74 2e 63 6c 6f   s.confluent.clo
          002d - 75 64                                          ud
        extension_type=ec_point_formats(11), length=4
          uncompressed (0)
          ansiX962_compressed_prime (1)
          ansiX962_compressed_char2 (2)
        extension_type=supported_groups(10), length=12
          ecdh_x25519 (29)
          secp256r1 (P-256) (23)
          ecdh_x448 (30)
          secp521r1 (P-521) (25)
          secp384r1 (P-384) (24)
        extension_type=session_ticket(35), length=0
        extension_type=encrypt_then_mac(22), length=0
        extension_type=extended_master_secret(23), length=0
        extension_type=signature_algorithms(13), length=30
          ecdsa_secp256r1_sha256 (0x0403)
          ecdsa_secp384r1_sha384 (0x0503)
          ecdsa_secp521r1_sha512 (0x0603)
          ed25519 (0x0807)
          ed448 (0x0808)
          rsa_pss_pss_sha256 (0x0809)
          rsa_pss_pss_sha384 (0x080a)
          rsa_pss_pss_sha512 (0x080b)
          rsa_pss_rsae_sha256 (0x0804)
          rsa_pss_rsae_sha384 (0x0805)
          rsa_pss_rsae_sha512 (0x0806)
          rsa_pkcs1_sha256 (0x0401)
          rsa_pkcs1_sha384 (0x0501)
          rsa_pkcs1_sha512 (0x0601)
        extension_type=supported_versions(43), length=3
          TLS 1.3 (772)
        extension_type=psk_key_exchange_modes(45), length=2
          psk_dhe_ke (1)
        extension_type=key_share(51), length=38
            NamedGroup: ecdh_x25519 (29)
            key_exchange:  (len=32): 71CEDCA193D9784DEA541D114F2BD03C7FB301E2CF33244E1925031435D7AC4F

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Alert (21)
  Length = 2
    Level=fatal(2), description=protocol version(70)

debug: ┏ [Connection] disconnecting...
debug: ┃ [0] {
debug: ┃ [1]   "timestamp": "2020-10-21T09:02:11.300Z",
debug: ┃ [2]   "logger": "kafkajs",
debug: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
debug: ┃ [4]   "clientId": "test-admin-id"
debug: ┗ [5] }
error: ┏ [Connection] Connection error: Client network socket disconnected before secure TLS connection was established
error: ┃ [0] {
error: ┃ [1]   "timestamp": "2020-10-21T09:02:11.301Z",
error: ┃ [2]   "logger": "kafkajs",
error: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
error: ┃ [4]   "clientId": "test-admin-id",
error: ┃ [5]   "stack": "Error: Client network socket disconnected before secure TLS connection was established\n    at connResetException (internal/errors.js:613:14)\n    at TLSSocket.onConnectEnd (_tls_wrap.js:1545:19)\n    at TLSSocket.emit (events.js:326:22)\n    at endReadableNT (_stream_readable.js:1226:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21)"
error: ┗ [6] }
debug: ┏ [Connection] disconnecting...
debug: ┃ [0] {
debug: ┃ [1]   "timestamp": "2020-10-21T09:02:11.301Z",
debug: ┃ [2]   "logger": "kafkajs",
debug: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
debug: ┃ [4]   "clientId": "test-admin-id"
debug: ┗ [5] }
debug: ┏ [Connection] disconnected
debug: ┃ [0] {
debug: ┃ [1]   "timestamp": "2020-10-21T09:02:11.301Z",
debug: ┃ [2]   "logger": "kafkajs",
debug: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
debug: ┃ [4]   "clientId": "test-admin-id"
debug: ┗ [5] }
debug: ┏ [Connection] disconnected
debug: ┃ [0] {
debug: ┃ [1]   "timestamp": "2020-10-21T09:02:11.302Z",
debug: ┃ [2]   "logger": "kafkajs",
debug: ┃ [3]   "broker": "pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092",
debug: ┃ [4]   "clientId": "test-admin-id"
debug: ┗ [5] }
TLS 49146: client onerror [Error: 4499246528:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 70
] {
  library: 'SSL routines',
  function: 'ssl3_read_bytes',
  reason: 'tlsv1 alert protocol version',
  code: 'ERR_SSL_TLSV1_ALERT_PROTOCOL_VERSION'
} had? true
^C

"alert number 70" from openssl is: "The protocol version the client attempted to negotiate is recognized, but not supported. For example, old protocol versions might be avoided for security reasons. This message is always fatal."

If I'm reading the output right, it looks like the node process is saying that it wants to use TLS 1.2, which according to Confluent Clouds' documentation is supported. https://docs.confluent.io/current/cloud/faq.html#:~:text=Security%20Controls%20whitepaper.-,What%20version%20of%20TLS%20is%20supported%20on%20Confluent%20Cloud,TLS%20version%201.2%20is%20supported.&text=Effective%20March%2015%2C%202020%2C%20connections,1.1%20are%20no%20longer%20supported.

Nevon on 21 Oct 2020

👍3

We're seeing the same issue on our infrastructure, however only our Staging cluster, production seems fine. So far. This is quite worrying. 2020-10-20T13:23:01.263Z is exactly when it started happening for us as well, in the middle of applications being up, so it definitely wasn't a dependency upgrade that caused it. Also trying to get in touch with Confluent support.

efstajas on 21 Oct 2020

@Nevon

Header:
  Version = TLS 1.0 (0x301)

Doesn't this look like kafkajs is trying to connect with TLS 1.0, and the response indicates that only 1.2 is supported…?

efstajas on 21 Oct 2020

Doesn't this look like kafkajs is trying to connect with TLS 1.0, and the response indicates that only 1.2 is supported…?

I'm not entirely sure. The output format is from openssl's trace format, but it needs to be compiled with enable-ssl-trace for it to work, so I'm looking to see if there's a pre-compiled binary with that that I can try with to see what the output looks like from there. EDIT: Now I'm compiling openssl myself. What a fun day this turned out to be.

My admittedly very basic understanding is that the initial handshake is done with TLS 1.0 but containing the supported TLS versions on the client, which the server and client then use to negotiate which protocol version to use:

ClientHello, Length=257
      client_version=0x303 (TLS 1.2)

Nevon on 21 Oct 2020

👍2

Hey there, have you tried to increase the timeout on the broker config?

new kafkajs.Kafka({
    connectionTimeout: 10_000,
    authenticationTimeout: 10_000,
    ....
});

Here on my tests it worked, a bit shaky but it works.

jaysonsantos on 21 Oct 2020

👍3

Here's something very interesting. I tried to create as bare-bones of an example as I could, to reproduce the issue, but in fact I was not able to:

const tls = require('tls')

const log = event => (...args) => {
  console.log(
    JSON.stringify(
      {
        event,
        args: args,
      },
      null,
      2
    )
  )
}

const config = {
  host: process.env.HOST,
  port: process.env.PORT,
  servername: process.env.HOST,
  ssl: true,
}

const socket = tls.connect(config, log('connect'))

socket.on('data', log('data'))
socket.on('end', log('end'))
socket.on('error', log('error'))
socket.on('timeout', log('timeout'))

socket.setKeepAlive(true, 60000)
socket.enableTrace()

This in fact does connect, going through the whole handshake process:

$ node repro.js
Sent Record
Header:
  Version = TLS 1.0 (0x301)
  Content Type = Handshake (22)
  Length = 391
    ClientHello, Length=387
      client_version=0x303 (TLS 1.2)
      Random:
        gmt_unix_time=0x29C98767
        random_bytes (len=28): 88A372C548C62F4072CEA242300EDDE4DB3BF206163D0EA4125ACF7D
      session_id (len=32): C7B5072C4F5C19CF0E9A8AB2A18EBD1A440E2EBA0F41B3E90B76A2CA5C6A230C
      cipher_suites (len=118)
        {0x13, 0x02} TLS_AES_256_GCM_SHA384
        {0x13, 0x03} TLS_CHACHA20_POLY1305_SHA256
        {0x13, 0x01} TLS_AES_128_GCM_SHA256
        {0xC0, 0x2F} TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x2B} TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x30} TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        {0xC0, 0x2C} TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        {0x00, 0x9E} TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x27} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x67} TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        {0xC0, 0x28} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
        {0x00, 0x6B} TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        {0x00, 0xA3} TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
        {0x00, 0x9F} TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        {0xCC, 0xA9} TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xA8} TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xAA} TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xC0, 0xAF} TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8
        {0xC0, 0xAD} TLS_ECDHE_ECDSA_WITH_AES_256_CCM
        {0xC0, 0xA3} TLS_DHE_RSA_WITH_AES_256_CCM_8
        {0xC0, 0x9F} TLS_DHE_RSA_WITH_AES_256_CCM
        {0xC0, 0x5D} TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384
        {0xC0, 0x61} TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
        {0xC0, 0x57} TLS_DHE_DSS_WITH_ARIA_256_GCM_SHA384
        {0xC0, 0x53} TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
        {0x00, 0xA2} TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
        {0xC0, 0xAE} TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8
        {0xC0, 0xAC} TLS_ECDHE_ECDSA_WITH_AES_128_CCM
        {0xC0, 0xA2} TLS_DHE_RSA_WITH_AES_128_CCM_8
        {0xC0, 0x9E} TLS_DHE_RSA_WITH_AES_128_CCM
        {0xC0, 0x5C} TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256
        {0xC0, 0x60} TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
        {0xC0, 0x56} TLS_DHE_DSS_WITH_ARIA_128_GCM_SHA256
        {0xC0, 0x52} TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
        {0xC0, 0x24} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        {0x00, 0x6A} TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
        {0xC0, 0x23} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x40} TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
        {0xC0, 0x0A} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        {0xC0, 0x14} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x39} TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x38} TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        {0xC0, 0x09} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        {0xC0, 0x13} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x33} TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x32} TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        {0x00, 0x9D} TLS_RSA_WITH_AES_256_GCM_SHA384
        {0xC0, 0xA1} TLS_RSA_WITH_AES_256_CCM_8
        {0xC0, 0x9D} TLS_RSA_WITH_AES_256_CCM
        {0xC0, 0x51} TLS_RSA_WITH_ARIA_256_GCM_SHA384
        {0x00, 0x9C} TLS_RSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0xA0} TLS_RSA_WITH_AES_128_CCM_8
        {0xC0, 0x9C} TLS_RSA_WITH_AES_128_CCM
        {0xC0, 0x50} TLS_RSA_WITH_ARIA_128_GCM_SHA256
        {0x00, 0x3D} TLS_RSA_WITH_AES_256_CBC_SHA256
        {0x00, 0x3C} TLS_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x35} TLS_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x2F} TLS_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0xFF} TLS_EMPTY_RENEGOTIATION_INFO_SCSV
      compression_methods (len=1)
        No Compression (0x00)
      extensions, length = 196
        extension_type=server_name(0), length=47
          0000 - 00 2d 00 00 2a 70 6b 63-2d 6c 71 38 76 37 2e   .-..*pkc-lq8v7.
          000f - 65 75 2d 63 65 6e 74 72-61 6c 2d 31 2e 61 77   eu-central-1.aw
          001e - 73 2e 63 6f 6e 66 6c 75-65 6e 74 2e 63 6c 6f   s.confluent.clo
          002d - 75 64                                          ud
        extension_type=ec_point_formats(11), length=4
          uncompressed (0)
          ansiX962_compressed_prime (1)
          ansiX962_compressed_char2 (2)
        extension_type=supported_groups(10), length=12
          ecdh_x25519 (29)
          secp256r1 (P-256) (23)
          ecdh_x448 (30)
          secp521r1 (P-521) (25)
          secp384r1 (P-384) (24)
        extension_type=session_ticket(35), length=0
        extension_type=encrypt_then_mac(22), length=0
        extension_type=extended_master_secret(23), length=0
        extension_type=signature_algorithms(13), length=48
          ecdsa_secp256r1_sha256 (0x0403)
          ecdsa_secp384r1_sha384 (0x0503)
          ecdsa_secp521r1_sha512 (0x0603)
          ed25519 (0x0807)
          ed448 (0x0808)
          rsa_pss_pss_sha256 (0x0809)
          rsa_pss_pss_sha384 (0x080a)
          rsa_pss_pss_sha512 (0x080b)
          rsa_pss_rsae_sha256 (0x0804)
          rsa_pss_rsae_sha384 (0x0805)
          rsa_pss_rsae_sha512 (0x0806)
          rsa_pkcs1_sha256 (0x0401)
          rsa_pkcs1_sha384 (0x0501)
          rsa_pkcs1_sha512 (0x0601)
          ecdsa_sha224 (0x0303)
          ecdsa_sha1 (0x0203)
          rsa_pkcs1_sha224 (0x0301)
          rsa_pkcs1_sha1 (0x0201)
          dsa_sha224 (0x0302)
          dsa_sha1 (0x0202)
          dsa_sha256 (0x0402)
          dsa_sha384 (0x0502)
          dsa_sha512 (0x0602)
        extension_type=supported_versions(43), length=5
          TLS 1.3 (772)
          TLS 1.2 (771)
        extension_type=psk_key_exchange_modes(45), length=2
          psk_dhe_ke (1)
        extension_type=key_share(51), length=38
            NamedGroup: ecdh_x25519 (29)
            key_exchange:  (len=32): DC9157ACE97684D13A3A0DA9BA276757BF72EE8AA5268E91B5D5E4285908E741

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Handshake (22)
  Length = 3023
    ServerHello, Length=81
      server_version=0x303 (TLS 1.2)
      Random:
        gmt_unix_time=0x4A7E7CD8
        random_bytes (len=28): 1973BB6700530F602F21BCFE2E99A22AB414CBA1D8C82E7D9D5C8A82
      session_id (len=32): 713551A52A69DEB7535CBEB1F1284E8197C0B2A4B0A45543CCEE3BD784DD8466
      cipher_suite {0xC0, 0x30} TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      compression_method: No Compression (0x00)
      extensions, length = 9
        extension_type=extended_master_secret(23), length=0
        extension_type=renegotiate(65281), length=1
            <EMPTY>

    Certificate, Length=2630
      certificate_list, length=2627
        ASN.1Cert, length=1447
------details-----
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:96:2c:cc:d9:4f:3f:62:e7:f5:63:f5:34:2d:a5:a1:ab:9f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Sep 14 18:45:40 2020 GMT
            Not After : Dec 13 18:45:40 2020 GMT
        Subject: CN = *.eu-central-1.aws.confluent.cloud
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:ab:6d:16:27:b7:e3:e2:3c:25:2e:d9:ac:7c:23:
                    ae:24:58:64:2e:8f:17:4a:79:b0:fa:8b:52:cd:31:
                    a9:ad:63:a6:65:40:86:ba:65:92:cf:83:5a:c9:b0:
                    4b:42:cb:59:a4:e5:eb:a5:2f:1d:28:c8:62:96:92:
                    45:d4:a7:89:63:9f:03:c9:1b:82:2d:ef:e4:a2:df:
                    41:28:e1:a2:c6:66:4c:b8:78:95:e0:fc:c5:91:36:
                    e9:7d:7c:fe:6b:0c:4d:cb:98:87:82:01:91:1f:6f:
                    f4:2b:d2:aa:e1:b0:6a:ca:c0:6e:3a:b2:2b:bc:ab:
                    0a:b4:43:8f:58:a1:5b:61:fb:ae:90:b4:f3:f7:77:
                    79:49:bf:6e:eb:57:70:8f:3c:87:a9:27:5b:eb:0b:
                    8f:ee:9f:db:53:42:3b:e8:51:96:fb:16:34:58:76:
                    38:01:52:7b:65:9a:1e:db:11:4f:fb:07:a2:dd:06:
                    47:27:39:22:b4:50:15:13:07:9d:b8:29:a1:94:6c:
                    e5:1c:48:ac:66:82:97:0c:61:31:72:d4:26:d8:90:
                    05:e6:6c:07:45:d5:89:59:56:39:ec:37:c4:14:be:
                    7e:d5:e4:2a:e6:9f:64:89:4b:26:7d:2a:36:7d:fd:
                    73:57:11:19:bd:cf:47:b9:71:ba:34:8c:b3:96:5f:
                    67:6d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                8A:40:86:49:F6:DF:F0:83:AF:2F:C4:EE:B8:53:A0:D7:1F:29:29:E2
            X509v3 Authority Key Identifier:
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access:
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name:
                DNS:*.eu-central-1.aws.confluent.cloud, DNS:*.eu-central-1.aws.glb.confluent.cloud
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : F0:95:A4:59:F2:00:D1:82:40:10:2D:2F:93:88:8E:AD:
                                4B:FE:1D:47:E3:99:E1:D0:34:A6:B0:A8:AA:8E:B2:73
                    Timestamp : Sep 14 19:45:41.061 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:66:B9:5D:4D:18:FF:7D:F5:E1:9D:CE:87:
                                7B:34:C7:75:14:09:C0:17:85:EE:B9:7B:20:A4:29:B0:
                                13:B2:DB:82:02:21:00:D5:E1:25:01:A0:D1:70:11:6C:
                                AB:AD:D5:3E:78:90:FA:F1:83:B5:7A:93:F6:54:8B:D7:
                                2D:AB:E5:18:43:2A:33
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B2:1E:05:CC:8B:A2:CD:8A:20:4E:87:66:F9:2B:B9:8A:
                                25:20:67:6B:DA:FA:70:E7:B2:49:53:2D:EF:8B:90:5E
                    Timestamp : Sep 14 19:45:41.076 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:07:E9:24:78:CD:72:87:5B:C2:10:C5:56:
                                EE:2C:61:D6:31:78:E4:43:C0:82:EF:A3:04:01:1A:E4:
                                19:0C:AD:E8:02:20:7B:3F:A0:51:B3:5C:0C:87:D8:09:
                                8B:61:80:8E:84:A9:6A:D4:CA:51:F2:93:96:03:44:2E:
                                41:E5:5E:91:A2:94
    Signature Algorithm: sha256WithRSAEncryption
         03:ae:9c:8c:f3:d6:92:4d:5d:db:1d:6e:aa:a9:3e:55:2f:47:
         db:1b:a2:d1:74:87:57:83:0f:0c:c2:6e:7a:03:f1:f5:40:83:
         a6:d5:73:96:5d:ef:0e:b0:0f:97:76:0c:8b:5c:11:65:8b:44:
         e0:f8:d6:66:72:6a:ec:fd:af:7d:d1:e2:c7:2a:10:3b:e0:06:
         f9:9e:58:c7:26:37:d7:da:24:5b:e5:41:d7:d3:9f:f1:87:3a:
         18:25:5c:88:0b:b2:77:3f:94:03:90:2a:f6:a5:aa:6d:3e:c9:
         6d:32:0e:5b:fb:01:6b:ee:40:1b:d7:ec:9b:de:97:a6:cc:44:
         c8:35:fe:77:61:57:d9:69:04:8d:14:96:71:16:14:f3:57:66:
         57:ac:2a:46:bb:f9:54:ba:4e:eb:14:0a:98:d6:7d:ff:23:85:
         4c:02:9f:c6:df:d1:a4:08:1b:66:c3:50:55:ee:0f:5b:46:57:
         c9:bf:aa:e0:d5:bd:fd:75:5f:9e:de:3c:e0:f6:3a:f4:df:f1:
         59:3a:c6:8b:d5:a7:cf:16:a1:16:32:bb:f0:39:ce:a4:44:da:
         de:8f:e0:33:c4:b6:84:7c:1e:ac:1f:43:4c:48:bf:91:3b:bc:
         8c:51:2e:54:2d:66:64:e4:7c:43:c8:a2:ef:5b:d4:25:99:60:
         23:76:4d:88
-----BEGIN CERTIFICATE-----
MIIFozCCBIugAwIBAgISA5YszNlPP2Ln9WP1NC2loaufMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA5MTQxODQ1NDBaFw0y
MDEyMTMxODQ1NDBaMC0xKzApBgNVBAMMIiouZXUtY2VudHJhbC0xLmF3cy5jb25m
bHVlbnQuY2xvdWQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrbRYn
t+PiPCUu2ax8I64kWGQujxdKebD6i1LNMamtY6ZlQIa6ZZLPg1rJsEtCy1mk5eul
Lx0oyGKWkkXUp4ljnwPJG4It7+Si30Eo4aLGZky4eJXg/MWRNul9fP5rDE3LmIeC
AZEfb/Qr0qrhsGrKwG46siu8qwq0Q49YoVth+66QtPP3d3lJv27rV3CPPIepJ1vr
C4/un9tTQjvoUZb7FjRYdjgBUntlmh7bEU/7B6LdBkcnOSK0UBUTB524KaGUbOUc
SKxmgpcMYTFy1CbYkAXmbAdF1YlZVjnsN8QUvn7V5Crmn2SJSyZ9KjZ9/XNXERm9
z0e5cbo0jLOWX2dtAgMBAAGjggKeMIICmjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0l
BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYE
FIpAhkn23/CDry/E7rhToNcfKSniMB8GA1UdIwQYMBaAFKhKamMEfd265tE5t6ZF
Ze/zqOyhMG8GCCsGAQUFBwEBBGMwYTAuBggrBgEFBQcwAYYiaHR0cDovL29jc3Au
aW50LXgzLmxldHNlbmNyeXB0Lm9yZzAvBggrBgEFBQcwAoYjaHR0cDovL2NlcnQu
aW50LXgzLmxldHNlbmNyeXB0Lm9yZy8wVQYDVR0RBE4wTIIiKi5ldS1jZW50cmFs
LTEuYXdzLmNvbmZsdWVudC5jbG91ZIImKi5ldS1jZW50cmFsLTEuYXdzLmdsYi5j
b25mbHVlbnQuY2xvdWQwTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMB
AQEwKDAmBggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggED
BgorBgEEAdZ5AgQCBIH0BIHxAO8AdgDwlaRZ8gDRgkAQLS+TiI6tS/4dR+OZ4dA0
prCoqo6ycwAAAXSOJsrFAAAEAwBHMEUCIGa5XU0Y/3314Z3Oh3s0x3UUCcAXhe65
eyCkKbATstuCAiEA1eElAaDRcBFsq63VPniQ+vGDtXqT9lSL1y2r5RhDKjMAdQCy
HgXMi6LNiiBOh2b5K7mKJSBna9r6cOeySVMt74uQXgAAAXSOJsrUAAAEAwBGMEQC
IAfpJHjNcodbwhDFVu4sYdYxeORDwILvowQBGuQZDK3oAiB7P6BRs1wMh9gJi2GA
joSpatTKUfKTlgNELkHlXpGilDANBgkqhkiG9w0BAQsFAAOCAQEAA66cjPPWkk1d
2x1uqqk+VS9H2xui0XSHV4MPDMJuegPx9UCDptVzll3vDrAPl3YMi1wRZYtE4PjW
ZnJq7P2vfdHixyoQO+AG+Z5YxyY319okW+VB19Of8Yc6GCVciAuydz+UA5Aq9qWq
bT7JbTIOW/sBa+5AG9fsm96XpsxEyDX+d2FX2WkEjRSWcRYU81dmV6wqRrv5VLpO
6xQKmNZ9/yOFTAKfxt/RpAgbZsNQVe4PW0ZXyb+q4NW9/XVfnt484PY69N/xWTrG
i9WnzxahFjK78DnOpETa3o/gM8S2hHwerB9DTEi/kTu8jFEuVC1mZOR8Q8ii71vU
JZlgI3ZNiA==
-----END CERTIFICATE-----
------------------
        ASN.1Cert, length=1174
------details-----
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:9c:d3:0c:f0:5a:e5:2e:47:b7:72:5d:37:83:b3:
                    68:63:30:ea:d7:35:26:19:25:e1:bd:be:35:f1:70:
                    92:2f:b7:b8:4b:41:05:ab:a9:9e:35:08:58:ec:b1:
                    2a:c4:68:87:0b:a3:e3:75:e4:e6:f3:a7:62:71:ba:
                    79:81:60:1f:d7:91:9a:9f:f3:d0:78:67:71:c8:69:
                    0e:95:91:cf:fe:e6:99:e9:60:3c:48:cc:7e:ca:4d:
                    77:12:24:9d:47:1b:5a:eb:b9:ec:1e:37:00:1c:9c:
                    ac:7b:a7:05:ea:ce:4a:eb:bd:41:e5:36:98:b9:cb:
                    fd:6d:3c:96:68:df:23:2a:42:90:0c:86:74:67:c8:
                    7f:a5:9a:b8:52:61:14:13:3f:65:e9:82:87:cb:db:
                    fa:0e:56:f6:86:89:f3:85:3f:97:86:af:b0:dc:1a:
                    ef:6b:0d:95:16:7d:c4:2b:a0:65:b2:99:04:36:75:
                    80:6b:ac:4a:f3:1b:90:49:78:2f:a2:96:4f:2a:20:
                    25:29:04:c6:74:c0:d0:31:cd:8f:31:38:95:16:ba:
                    a8:33:b8:43:f1:b1:1f:c3:30:7f:a2:79:31:13:3d:
                    2d:36:f8:e3:fc:f2:33:6a:b9:39:31:c5:af:c4:8d:
                    0d:1d:64:16:33:aa:fa:84:29:b6:d4:0b:c0:d8:7d:
                    c3:93
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            Authority Information Access:
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier:
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier:
                A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
    Signature Algorithm: sha256WithRSAEncryption
         dd:33:d7:11:f3:63:58:38:dd:18:15:fb:09:55:be:76:56:b9:
         70:48:a5:69:47:27:7b:c2:24:08:92:f1:5a:1f:4a:12:29:37:
         24:74:51:1c:62:68:b8:cd:95:70:67:e5:f7:a4:bc:4e:28:51:
         cd:9b:e8:ae:87:9d:ea:d8:ba:5a:a1:01:9a:dc:f0:dd:6a:1d:
         6a:d8:3e:57:23:9e:a6:1e:04:62:9a:ff:d7:05:ca:b7:1f:3f:
         c0:0a:48:bc:94:b0:b6:65:62:e0:c1:54:e5:a3:2a:ad:20:c4:
         e9:e6:bb:dc:c8:f6:b5:c3:32:a3:98:cc:77:a8:e6:79:65:07:
         2b:cb:28:fe:3a:16:52:81:ce:52:0c:2e:5f:83:e8:d5:06:33:
         fb:77:6c:ce:40:ea:32:9e:1f:92:5c:41:c1:74:6c:5b:5d:0a:
         5f:33:cc:4d:9f:ac:38:f0:2f:7b:2c:62:9d:d9:a3:91:6f:25:
         1b:2f:90:b1:19:46:3d:f6:7e:1b:a6:7a:87:b9:a3:7a:6d:18:
         fa:25:a5:91:87:15:e0:f2:16:2f:58:b0:06:2f:2c:68:26:c6:
         4b:98:cd:da:9f:0c:f9:7f:90:ed:43:4a:12:44:4e:6f:73:7a:
         28:ea:a4:aa:6e:7b:4c:7d:87:dd:e0:c9:02:44:a7:87:af:c3:
         34:5b:b4:42
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
------------------

    ServerKeyExchange, Length=296
      KeyExchangeAlgorithm=ECDHE
        named_curve: ecdh_x25519 (29)
        point (len=32): A7988B1868CE95FD2FA582C33ED33D3650DFB183BFBEB968E4B29B67516A3A14
      Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
      Signature (len=256): 7A06C3F3D5A1404C186989E184F01760ECF50A64DFE6E6F2ACBDE917CBA0790F5458042248D594A5DE071BBF8FBB068FBBCC961366D3B1ED368F39D99EE8D318D13E03C47D85FF8039A8D8A2548803474678C0D6AC6D8603926A64F4868EA9A7B38249F5D03A44D723D09DB9AE40CF0A1C4EA251B0F75F654F6EF12ECB84A68F5597CAB2B785966A677A77DEDB7CE7FE29D2820D2A5B04E381FBECABBECC47F23A6B71BBE5925AC10B5EBDE4A71602E7CC50920553058BFCAB50185C4E9F5EC79BF3340A9B393C17CE72C6AE6A10C16BC4CC53090E0E4BC6F37EEA038B88F5AA9A476AB89E2F4C8713B0C53D3A961AB755EEA2CBA710725927BCFA0DACA15555

    ServerHelloDone, Length=0

Sent Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Handshake (22)
  Length = 37
    ClientKeyExchange, Length=33
      KeyExchangeAlgorithm=ECDHE
        ecdh_Yc (len=32): C7BF70510A4B793A9AF3DDCAB26376E6100A215598A4C4AE03CB1D302335134C

Sent Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ChangeCipherSpec (20)
  Length = 1
    change_cipher_spec (1)

Sent Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Handshake (22)
  Length = 40
    Finished, Length=12
      verify_data (len=12): 8070A10F3F4E2C5BE06C0733

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = ChangeCipherSpec (20)
  Length = 1
Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Handshake (22)
  Length = 40
    Finished, Length=12
      verify_data (len=12): DCB8B49688A7D97985D8D0F6

{
  "event": "connect",
  "args": []
}
^C

So I compared the request from the client between KafkaJS and my simple script, and other than the client random section, timestamps, key exchange and the session ids, which are expected to be different, they are the same:

diff simple.txt kafka.txt
9,11c9,11
<         gmt_unix_time=0xB8992233
<         random_bytes (len=28): 28C932CE7E72C054EC4F9291BEE2EB4D961BC5C7FE0F93DE72BA1649
<       session_id (len=32): B4434780721C280A5CFDE90E63BF2E0FD9C0B55644FDFF9172ACB1AACDA874FE
---
>         gmt_unix_time=0xC4E8CC57
>         random_bytes (len=28): A5232352B152AB4BE3D2003BC7645137531262147544F98E5CAA10C9
>       session_id (len=32): 2193FB4032C7846160FACA1554E129DC47DA772A0C2216679543B58A0D9F0394
124c124
<             key_exchange:  (len=32): 01A07328A168436ABF39678076ABA688AF9CBB530C35BC677F910D8B41515267
\ No newline at end of file
---
>             key_exchange:  (len=32): 91FBBC85FB8ED7FC3CC2301627FDE3652AEF4841A6D816B87A7F584C3FBA9706
\ No newline at end of file

Nevon on 21 Oct 2020

I can confirm it connects (mostly) fine with increased connectionTimeout of 10000 millis. It appears to drop out and in sometimes but _it connects._ Strange.

efstajas on 21 Oct 2020

What the heck...

I can indeed see that the TLS handshake takes kind of forever, and the default connectionTimeout is 1 second, so it makes sense that we'd time out, but the thing is that we handle this one level above the socket itself. We literally just use a setTimeout and throw an error ourselves if it hasn't connected by then.

https://github.com/tulios/kafkajs/blob/f46f29077a2a11e1ce381e5330b3531276b979f1/src/network/connection.js#L180-L200

That shouldn't result in some error from openssl, since as far as the socket is concerned the only timeout is the default handshakeTimeout, which defaults to 2 minutes.

EDIT: Oh, of course, it makes sense now.

What's happening is that we enforce the timeout after 1 second, which closes the socket from the client end. That in turn causes the socket to emit an error because, well... The client network socket disconnected before secure TLS connection was established, just like the error says.

So what's happening is that Confluent Cloud did something to cause establishing a TCP connection to take more than 1 second. Everyone who's using the default KafkaJS connection settings will time out the connection after 1 second and will see this error. This is not really something to fix in KafkaJS, but for users (including ourselves), the solution in the short term is to increase the connection timeout to some value that works, and then for Confluent Cloud to figure out why establishing a connection takes more than 1 second.

Nevon on 21 Oct 2020

Thanks for investigating @Nevon! I'll update our support case at Confluent and link to your comment.

renklaf on 21 Oct 2020

I'm going to close this issue, since this appears to not be an issue with KafkaJS, but feel free to keep adding updates in case there is some information from Confluent regarding this issue.

Nevon on 21 Oct 2020

👍3

@Nevon We've received the following on our support ticket at Confluent Cloud:

_I reached out to our cloud engineering team to understand what has changed on cluster pkc-lq8v7 yesterday that could impact clients and I've been informed that a connection rate throttling for TCP connections has been added to the cluster: this might affect clients with very short TLS handshake timeouts.
For example some kafkajs clients might have a default TLS handshake timeout of 1 second, increasing this timeout (for example to 10s), should solve the problem._

If this is Confluent Cloud's default from now on, perhaps an increase in KafkaJS's default timeouts could make sense.

renklaf on 21 Oct 2020

Thanks all for this very thorough analysis.
I got the same message as you from confluent support ("Some kafkajs clients might have a default TLS handshake timeout of 1 second, increasing this timeout for example to 10s, should solve the problem").

Is there a way to configure this in kafakjs ? I don't see any param like this in tls.createSecureContext

mdespriee on 22 Oct 2020

@mdespriee you can take a look at our documentation website, the parameter is:
https://kafka.js.org/docs/configuration#connection-timeout

tulios on 22 Oct 2020

Hey guys
I got to talk to a Confluent personnel and one thing he's mentioned is that Confluent has server throttling that kicks in when there are "too many" requests.

His exact email:

I have not installed the application locally yet, but I notice that you are using the kafkajs library. We have seen a similar issue with this library with another client. It appears to be related to Confluent Cloud server throttling, which kicks in due to too many connection requests in a period of time. The quick fix is to adjust the TLS handshake timeout for that library to 10-15 seconds. We are not sure why this library makes so many requests - it may not use a keepalive? - but other customers have resolved the issue by following these steps.
More information can be found at https://github.com/tulios/kafkajs/issues/931

(This actually led me to this ticket)

anonimitoraf on 13 Nov 2020

We are not sure why this library makes so many requests - it may not use a keepalive? - but other customers have resolved the issue by following these steps.

That triggered me a bit. :)

I checked briefly, it seems KafkaJS creates one set of connections for each consumer(), producer(), or admin() result, and this does come down to a lot of connections when your cluster is big or you're having many consumers. I actually think this might also explain one thing I saw with smaller AWS MSK clusters, which seemed to get flooded when I tested them with just "a bit of traffic".

ankon on 16 Nov 2020

Hey guys
I got to talk to a Confluent personnel and one thing he's mentioned is that Confluent has server throttling that kicks in when there are "too many" requests.

Well... our forwarder of event to Kafka is on AWS lambda.
When the broker has latencies, this induces more parallelism. That is, opening more connections. If at some point we reach the throttling limit, you get connection timeouts, which means the concurrency on AWS lambda will skyrocket, amplifying the problem by a factor 1000 or so. That's what happened to us.

If the endpoint of the broker cannot absorb a peak, then you have to have a system to absorb it (with internal buffers or something).

We started a migration towards kafka REST proxy instead of talking directly to the broker through kafkajs.

mdespriee on 16 Nov 2020

Kafkajs: Connection error to kafka hosted on confluent cloud with v1.14

Most helpful comment

All 21 comments

Related issues