Druid: SSL handshake failed - Kafka Index Service

[MaxOrelus]

[2020-02-15 12:45:32,705] INFO [SocketServer brokerId=0] Failed authentication with /54.147.244.29 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I have the extension:

druid_extensions_loadList=["druid-kafka-indexing-service", "druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "druid-azure-extensions", "postgresql-metadata-storage"]

That's the error. After going through all the documentation and issues in github I'm not able to connect to a Kafka instance that is using TLS/SSL. These are the settings I have so far:

{
  "bootstrap.servers": "kafka001.domain.com:9093",
  "ssl.truststore.password": "password",
  "ssl.keystore.password": "password",
  "ssl.key.password": "password"
}

The full blown spec looks like:

{
  "type": "kafka",
  "ioConfig": {
    "type": "kafka",
    "consumerProperties": {
      "bootstrap.servers": "kafka001.domain.com:9093",
      "ssl.truststore.password": "password",
      "ssl.keystore.password": "password",
      "ssl.key.password": "password"
    },
    "topic": "my-topic"
  },
  "tuningConfig": {
    "type": "kafka"
  },
  "dataSchema": {
    "dataSource": "new-data-source",
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "HOUR",
      "queryGranularity": "HOUR"
    }
  }
}

The website is lacking information on how to setup up Kafka index services when you have TLS/SSL. I don't want to take up much of any one's time, but I'm just trying to figure out how to connect to my Kafka broker via druid. Is there something I'm missing here or is there any other steps I have to do to get this working?

Warm regards,
M

[jp707049]

Hello,

Try using following parameters. Let me know if this works.

bootstrap.servers: "hostname:port",  
security.protocol: "SASL_SSL",  
isolation.level: "read_committed",  
sasl.mechanism: "PLAIN",  
ssl.enabled.protocols: "TLSv1.2",  
sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='username_here' password='password_here';",
ssl.endpoint.identification.algorithm: ""

Thank you,
Jeet

[MaxOrelus]

Thanks, but I wasn't using SASL_SSL, just the basic SSL. I was able to get it working with the following:

{
  "bootstrap.servers": "<SERVER>:9093",
  "security.protocol": "SSL",
  "ssl.truststore.location": "/tmp/ssl/server.truststore.jks",
  "ssl.truststore.password": "<PASSWORD>",
  "ssl.truststore.type": "JKS",
  "ssl.key.password": "<PASSWORD>",
  "ssl.keystore.location": "/tmp/ssl/server.keystore.jks",
  "ssl.keystore.password": "<PASSWORD>,
  "ssl.keystore.type": "JKS",
  "ssl.endpoint.identification.algorithm": ""
}

[junneyang]

Hello,

Try using following parameters. Let me know if this works.

bootstrap.servers: "hostname:port",  
security.protocol: "SASL_SSL",  
isolation.level: "read_committed",  
sasl.mechanism: "PLAIN",  
ssl.enabled.protocols: "TLSv1.2",  
sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='username_here' password='password_here';",
ssl.endpoint.identification.algorithm: ""

Thank you,
Jeet

it works for me, very thanks~