Strimzi-kafka-operator: Error in operator when running on k8s

[arunkjn]

Hi ,

I am running v0.4 of operator in k8s cluster. It is getting the below error and restarting -

DefaultChannelPipeline:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:292)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
... 16 more
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1408)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1316)
... 20 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching kubernetes.default.svc found.
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:214)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:96)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.AbstractTrustManagerWrapper.checkAdditionalTrust(SSLContextImpl.java:1061)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1000)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)
... 28 more

[tombentley]

@scholzj can you help?

[scholzj]

I updated the master so that it is possible to set a environment variable KUBERNETES_TRUST_CERTIFICATES to true and disable to validation of certificates and hostname. This was done on master and will be released with 0.5.0.

[arunkjn]

thanks @scholzj. Do you guys have any timeline for releasing 0.5?

[scholzj]

@arunkjn Hopefully next week.

[AlbertoOS]

Hello @scholzj, how can I set this KUBERNETES_TRUST_CERTIFICATES config for the kafka cluster?
I've tried to edit some resources but wasn't able to keep this setting.
For example I've tried editing a statefulset of Kafka kind to have this setting as a environment variable but it gets overwritten after a while and I'm not getting why right now.

[ppatierno]

The env var should be set in the YAML file with the cluster operator deployment. This env var is for the fabric8 library used by cluster operator to interact with Kubernetes.
So it's not an env var to set for the Kafka cluster but for the cluster operator.

[davecaplinger]

@scholzj, @ppatierno - I have a similar issue - I added KUBERNETES_TRUST_CERTIFICATES to the cluster operator yaml (install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml) and was able to successfully start examples/kafka/kafka-ephemeral.yaml, but the cluster-entity-operator pod that results appears to be in CrashLoopBackOff because its containers do not have the KUBERNETES_TRUST_CERTIFICATES environment variable. I can't see where I can change this; do you have any hints?

[scholzj]

@davecaplinger That is a good point. Could you please open a separate issue for this since this one is already closed? Thanks.

[davecaplinger]

No problem, #993 opened.

[scholzj]

Thanks a lot.