Cp-docker-images: java.security.cert.CertificateException: No subject alternative names present

[therealrawal]

Kafka version: 2.0.0
Confluent version: 5.0.0
Image name: confluentinc/cp-kafka:5.0.0
Documentation: https://docs.confluent.io/current/kafka/encryption.html#

It's a 3 node kafka running on kubernetes using helm. I'm trying to enable SSL between fafka brokers by following the documentation on confluentinc website (Linked above).

I generate the ca cert and key on my local machine and mount it on each pod then the following script runs prior to starting kafka.

#!/bin/bash
cd /etc/kafka/secrets

keytool -keystore server.keystore.jks -alias localhost -validity 3650 -genkey -storepass <PASSWORD> -keypass <PASSWORD>-dname CN=kafka.namespace.svc.cluster.local -ext SAN="DNS:${HOSTNAME},IP:${POD_IP},DNS:kafka.namespace.svc.cluster.local"

keytool -import -noprompt -alias CARoot -file ca-cert -keystore client.truststore.jks -storepass <PASSWORD>

keytool -import -noprompt -alias CARoot -file ca-cert -keystore server.truststore.jks -storepass <PASSWORD>

keytool -certreq -alias localhost  -file cert-file -keystore server.keystore.jks -storepass <PASSWORD>

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:<PASSWORD>

keytool -import -noprompt -alias CARoot -file ca-cert -keystore server.keystore.jks -storepass <PASSWORD>

keytool -import -noprompt -alias localhost -file cert-signed -keystore server.keystore.jks -storepass <PASSWORD>

exec /etc/confluent/docker/run

I get the following error on kafka-0.

[DATETIME] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[DATETIME] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker <KAFKA_0_IP_ADDRESS>:9093 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
        at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
        at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
        ... 9 more
Caused by: java.security.cert.CertificateException: No subject alternative names present
        at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:145)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
        ... 18 more

I checked the crt in server.keystore.jks it has the correct SANs. I'm not sure where i should be looking for fix this.

[mirwais27]

Have same issue with plain kafka-client 2.0.0 running on spring boot 2.1.
Workaround: Switch kafka-client back to version 1.1.0. Then it works.

[therealrawal]

Is there any other workaround? It would be nice to use 2.0.0 or greater.

[ccrivelli]

Same here with Kafka 2.0.0.
I re-created and signed all the certificates 3 times.
Last time I included all the possible combinations of hostnames (including Lower case and Upper case).

Like:

SubjectAlternativeName [
  DNSName: hostname.domain
  DNSName: hostname
  DNSName: HOSTNAME.domain
  DNSName: HOSTNAME
  IPAddress: IP
]

I tested it both with NiFi "PublishKafka_2_0" Processor and "kafka-console-consumer.sh" getting the same error:

e.g.
# bin/kafka-console-consumer.sh --bootstrap-server hostname.domain:6668 --topic test --consumer.config conf/consumer-ssl.properties

ERROR:

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching HOSTNAME.domain found.

or

ERROR:

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching hostname.domain found.

Any suggestions?

[roshanphilips]

From kafka 2.0 onwards, host name verification of servers is enabled by default and the errors were logged because, the kafka hostname didnt match the certificate CN. If your hostname and certificate doesnt match, then you can disable the hostname verification by setting the property ssl.endpoint.identification.algorithm to empty string

ssl.endpoint.identification.algorithm=

[afayes]

@roshanphilips Thanks that worked! Was thinking it would be far more complex to fix!