Strimzi-kafka-operator: no subject alternative DNS name matching strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cumulus-k8s found.

[j-chao]

Hi,

I'm running into an issue where the connection to the nodes is failing authentication due to SSL handshake failure.
The logs show that it is an issue with the DNS name matching, does anyone have any solutions or workarounds for this?

2019-03-28 17:59:15,758 ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cumulus-k8s/10.233.69.142:9091) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [Controller-0-to-broker-0-send-thread]
2019-03-28 17:59:15,758 WARN [RequestSendThread controllerId=0] Controller 0's connection to broker strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cumulus-k8s:9091 (id: 0 rack: ) was unsuccessful (kafka.controller.RequestSendThread) [Controller-0-to-broker-0-send-thread]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528)
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1197)
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1165)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:447)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:312)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:265)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:129)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:532)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:521)
        at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73)
        at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
        at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:970)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:967)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:401)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:483)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:339)
        ... 9 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cumulus-k8s found.
        at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
        at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
        at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1626)
        ... 18 more

I am running strimzi v0.11.1.

Thank you!

[scholzj]

For most Kubernetes clusters the default DNS domain is cluster.local ... so the services are named like strimzi-kafka-0.strimzi-kafka-brokers.strimzi.svc.cluster.local. That is why Strimzi generates the certificates expecting this address. Your cluster seems to have the DNS domain cumulus-k8s whcih is causing this problem.

You should be able to change the behaviour by setting the environment variable KUBERNETES_SERVICE_DNS_DOMAIN in the Strimzi Cluster Operator deployment to cumulus-k8s (Just add this to the environment variables in the Deployment).

To be honest, right now out of the box I'm not sure whether it would regenerate the certificates automatically or only with new cluster setup. You would need to try it.

[j-chao]

@scholzj
Ah, that fixed it!
Thanks a lot!