Strimzi-kafka-operator: [Question] Standalone topic operator with AWS MSK (TLS error)

[elisiano]

I stood up a standalone topic operator following the docs (latest 0.17.0) and I'm trying to have it communicate with an MSK cluster.

The MSK cluster is configured to allow only TLS connections to the brokers.

If I just set STRIMZI_TLS_ENABLED to true, the startup scripts fail because it expects the certificates in the tls-sidecar path, but those directories do not exist (there is no side car running, nor I could find instructions on how to do so, but I assume that the sidecar gets deployed when using the topic operator within the cluster operator).

Looking at the startup scripts it seems that the logic to create a keystore is only executed if the truststore and keystore variables are not used.

The upstream AWS MSK docs suggest to use the default truststore since the certificates used in the MSK clusters have public certs.

So I tried setting STRIMZI_TRUSTSTORE_LOCATION to /usr/lib/jvm/jre/lib/security/cacerts which, after a series of symlink, resolve to the correct file (I tried listing the content with keytool with an empty password and the list of CAs is pretty big).

this is the startup logs (and failure):

➭ kubectl logs strimzi-topic-operator-msk-0-575b9d9df7-jbpkh
+ shift
+ . /opt/strimzi/bin/dynamic_resources.sh
++ get_heap_size
+++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
++ CONTAINER_MEMORY_IN_BYTES=524288000
++ DEFAULT_MEMORY_CEILING=32505856
++ '[' 524288000 -lt 32505856 ']'
+ MAX_HEAP=
+ '[' -n '' ']'
+ export MALLOC_ARENA_MAX=2
+ MALLOC_ARENA_MAX=2
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom'
++ get_gc_opts
++ '[' '' == true ']'
++ echo ''
+ JAVA_OPTS=' -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom '
+ exec /usr/bin/tini -w -e 143 -- java -Dvertx.cacheDirBase=/tmp -Djava.security.egd=file:/dev/./urandom -classpath lib/io.strimzi.topic-operator-0.17.0.jar:lib/io.prometheus.simpleclient_common-0.7.0.jar:lib/com.github.luben.zstd-jni-1.4.3-1.jar:lib/io.netty.netty-handler-4.1.45.Final.jar:lib/com.101tec.zkclient-0.11.jar:lib/io.netty.netty-codec-http-4.1.45.Final.jar:lib/com.squareup.okio.okio-1.15.0.jar:lib/io.netty.netty-buffer-4.1.45.Final.jar:lib/org.yaml.snakeyaml-1.24.jar:lib/io.fabric8.openshift-client-4.6.4.jar:lib/io.netty.netty-common-4.1.45.Final.jar:lib/org.apache.logging.log4j.log4j-api-2.13.0.jar:lib/org.xerial.snappy.snappy-java-1.1.7.3.jar:lib/org.hdrhistogram.HdrHistogram-2.1.11.jar:lib/io.prometheus.simpleclient-0.7.0.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.10.2.jar:lib/io.netty.netty-codec-4.1.45.Final.jar:lib/io.micrometer.micrometer-core-1.3.1.jar:lib/io.strimzi.certificate-manager-0.17.0.jar:lib/jakarta.activation.jakarta.activation-api-1.2.1.jar:lib/io.vertx.vertx-core-3.8.5.jar:lib/io.netty.netty-codec-dns-4.1.45.Final.jar:lib/io.fabric8.kubernetes-model-4.6.4.jar:lib/io.netty.netty-codec-socks-4.1.45.Final.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/org.apache.zookeeper.zookeeper-jute-3.5.6.jar:lib/io.netty.netty-resolver-4.1.45.Final.jar:lib/io.netty.netty-handler-proxy-4.1.45.Final.jar:lib/com.squareup.okhttp3.logging-interceptor-3.12.6.jar:lib/io.strimzi.api-0.17.0.jar:lib/io.netty.netty-transport-native-unix-common-4.1.45.Final.jar:lib/org.apache.zookeeper.zookeeper-3.5.6.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.vertx.vertx-micrometer-metrics-3.8.5.jar:lib/com.fasterxml.jackson.core.jackson-core-2.10.2.jar:lib/io.netty.netty-transport-4.1.45.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.45.Final.jar:lib/jakarta.xml.bind.jakarta.xml.bind-api-2.3.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.13.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.10.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/org.lz4.lz4-java-1.6.0.jar:lib/io.fabric8.kubernetes-client-4.6.4.jar:lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.10.2.jar:lib/com.squareup.okhttp3.okhttp-3.12.6.jar:lib/io.netty.netty-codec-http2-4.1.45.Final.jar:lib/io.strimzi.operator-common-0.17.0.jar:lib/org.apache.logging.log4j.log4j-core-2.13.0.jar:lib/io.fabric8.kubernetes-model-common-4.6.4.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.10.2.jar:lib/io.netty.netty-resolver-dns-4.1.45.Final.jar:lib/org.slf4j.slf4j-api-1.7.25.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.strimzi.crd-annotations-0.17.0.jar:lib/org.apache.kafka.kafka-clients-2.4.0.jar:lib/io.micrometer.micrometer-registry-prometheus-1.3.1.jar io.strimzi.operator.topic.Main
2020-03-29 04:48:52 INFO  Main:30 - TopicOperator 0.17.0 is starting
2020-03-29 04:48:52 DEBUG Config:465 - Trying to configure client from Kubernetes config...
2020-03-29 04:48:52 DEBUG Config:491 - Did not find Kubernetes config at: [/home/strimzi/.kube/config]. Ignoring.
2020-03-29 04:48:52 DEBUG Config:395 - Trying to configure client from service account...
2020-03-29 04:48:52 DEBUG Config:400 - Found service account host and port: 100.64.0.1:443
2020-03-29 04:48:52 DEBUG Config:406 - Found service account ca cert at: [/var/run/secrets/kubernetes.io/serviceaccount/ca.crt].
2020-03-29 04:48:52 DEBUG Config:414 - Found service account token at: [/var/run/secrets/kubernetes.io/serviceaccount/token].
2020-03-29 04:48:52 DEBUG Config:605 - Trying to configure client namespace from Kubernetes service account namespace path...
2020-03-29 04:48:52 DEBUG Config:610 - Found service account namespace at: [/var/run/secrets/kubernetes.io/serviceaccount/namespace].
2020-03-29 04:48:53 DEBUG InternalLoggerFactory:45 - Using SLF4J as the default logging framework
2020-03-29 04:48:53 DEBUG ResourceLeakDetector:130 - -Dio.netty.leakDetection.level: simple
2020-03-29 04:48:53 DEBUG ResourceLeakDetector:131 - -Dio.netty.leakDetection.targetRecords: 4
2020-03-29 04:48:53 DEBUG InternalThreadLocalMap:54 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2020-03-29 04:48:53 DEBUG InternalThreadLocalMap:57 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2020-03-29 04:48:53 DEBUG MultithreadEventLoopGroup:44 - -Dio.netty.eventLoopThreads: 2
2020-03-29 04:48:53 DEBUG NioEventLoop:106 - -Dio.netty.noKeySetOptimization: false
2020-03-29 04:48:53 DEBUG NioEventLoop:107 - -Dio.netty.selectorAutoRebuildThreshold: 512
2020-03-29 04:48:53 DEBUG PlatformDependent0:396 - -Dio.netty.noUnsafe: false
2020-03-29 04:48:53 DEBUG PlatformDependent0:852 - Java version: 8
2020-03-29 04:48:53 DEBUG PlatformDependent0:121 - sun.misc.Unsafe.theUnsafe: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:145 - sun.misc.Unsafe.copyMemory: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:183 - java.nio.Buffer.address: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:244 - direct buffer constructor: available
2020-03-29 04:48:53 DEBUG PlatformDependent0:314 - java.nio.Bits.unaligned: available, true
2020-03-29 04:48:53 DEBUG PlatformDependent0:379 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2020-03-29 04:48:53 DEBUG PlatformDependent0:386 - java.nio.DirectByteBuffer.<init>(long, int): available
2020-03-29 04:48:53 DEBUG PlatformDependent:1030 - sun.misc.Unsafe: available
2020-03-29 04:48:53 DEBUG PlatformDependent:1149 - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2020-03-29 04:48:53 DEBUG PlatformDependent:1228 - -Dio.netty.bitMode: 64 (sun.arch.data.model)
2020-03-29 04:48:53 DEBUG PlatformDependent:174 - -Dio.netty.maxDirectMemory: 127729664 bytes
2020-03-29 04:48:53 DEBUG PlatformDependent:181 - -Dio.netty.uninitializedArrayAllocationThreshold: -1
2020-03-29 04:48:53 DEBUG CleanerJava6:92 - java.nio.ByteBuffer.cleaner(): available
2020-03-29 04:48:53 DEBUG PlatformDependent:201 - -Dio.netty.noPreferDirect: false
2020-03-29 04:48:53 DEBUG PlatformDependent:891 - org.jctools-core.MpscChunkedArrayQueue: available
2020-03-29 04:48:53 DEBUG DefaultDnsServerAddressStreamProvider:82 - Default DNS servers: [/100.64.0.10:53] (sun.net.dns.ResolverConfiguration)
2020-03-29 04:48:53 DEBUG NetUtil:139 - -Djava.net.preferIPv4Stack: false
2020-03-29 04:48:53 DEBUG NetUtil:140 - -Djava.net.preferIPv6Addresses: false
2020-03-29 04:48:53 DEBUG NetUtil:224 - Loopback interface: lo (lo, 127.0.0.1)
2020-03-29 04:48:53 DEBUG NetUtil:271 - /proc/sys/net/core/somaxconn: 128
2020-03-29 04:48:53 INFO  Session:70 - Using config:
    STRIMZI_TRUSTSTORE_LOCATION: /usr/lib/jvm/jre/lib/security/cacerts
    STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=cxp-msk-0
    STRIMZI_KAFKA_BOOTSTRAP_SERVERS: b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094,b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094,b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094
    STRIMZI_NAMESPACE: cxp
    STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 20000
    STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 900000
    STRIMZI_ZOOKEEPER_CONNECT: z-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181,z-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181,z-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:2181
    STRIMZI_TLS_ENABLED: true
    STRIMZI_KEYSTORE_PASSWORD:
    STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
    STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
    STRIMZI_KEYSTORE_LOCATION:
    TC_ZK_CONNECTION_TIMEOUT_MS: 20000
    STRIMZI_TRUSTSTORE_PASSWORD:
    STRIMZI_REASSIGN_THROTTLE: 9223372036854775807

2020-03-29 04:48:53 DEBUG InternalLoggerFactory:61 - Using SLF4J as the default logging framework
2020-03-29 04:48:53 INFO  Session:141 - Starting
2020-03-29 04:48:53 INFO  AdminClientConfig:347 - AdminClientConfig values:
    bootstrap.servers = [b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094, b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094, b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094]
    client.dns.lookup = default
    client.id =
    connections.max.idle.ms = 300000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 120000
    retries = 5
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = GSSAPI
    security.protocol = SSL
    security.providers = null
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = HTTPS
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location =
    ssl.keystore.password = [hidden]
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = /usr/lib/jvm/jre/lib/security/cacerts
    ssl.truststore.password = [hidden]
    ssl.truststore.type = JKS

2020-03-29 04:48:54 DEBUG AdminMetadataManager:248 - [AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [b-2.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -3 rack: null), b-1.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -2 rack: null), b-3.cxp-k8s-usw2-dev.wfq4wf.c3.kafka.us-west-2.amazonaws.com:9094 (id: -1 rack: null)], partitions = [], controller = null).
2020-03-29 04:48:54 ERROR Main:55 - Error deploying Session
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:451) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:59) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at io.strimzi.operator.topic.Session.start(Session.java:157) ~[io.strimzi.topic-operator-0.17.0.jar:0.17.0]
    at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.8.5.jar:3.8.5]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.45.Final.jar:4.1.45.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.45.Final.jar:4.1.45.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore  of type JKS
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    ... 13 more
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore  of type JKS
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    ... 13 more
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore  of type JKS
    at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    ... 13 more
Caused by: java.io.IOException: Is a directory
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_242]
    at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46) ~[?:1.8.0_242]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_242]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_242]
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_242]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[?:1.8.0_242]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[?:1.8.0_242]
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[?:1.8.0_242]
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_242]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265) ~[?:1.8.0_242]
    at java.security.DigestInputStream.read(DigestInputStream.java:124) ~[?:1.8.0_242]
    at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[?:1.8.0_242]
    at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:658) ~[?:1.8.0_242]
    at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[?:1.8.0_242]
    at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_242]
    at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[?:1.8.0_242]
    at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_242]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:69) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:426) ~[org.apache.kafka.kafka-clients-2.4.0.jar:?]
    ... 13 more

If I disable TLS, I see zookeeper being able to list the topics but the K8sTopicWatcher does not go past this:

2020-03-29 04:33:12 DEBUG K8sTopicWatcher:40 - Ignoring initial event for KafkaTopic cxp-strimzi-test during initial reconcile

Any help would be greatly appreciated.

[scholzj]

I think the Topic Operator currently expects that you will either have no encryption or TLS encryption and authentication. I do not think it can do anything in between. So you need to specify the following options:

• STRIMZI_TRUSTSTORE_LOCATION
• STRIMZI_TRUSTSTORE_PASSWORD
• STRIMZI_KEYSTORE_LOCATION
• STRIMZI_KEYSTORE_PASSWORD

@ppatierno I think you were looking into it recently - can you confirm it?

[ppatierno]

Yes exactly. Currently, the topic operator doesn't allow you to specify just the truststore for the TLS encryption. It needs the keystore as well for TLS client authentication, this is why the above error not able to load a Keystore which is actually, in your case, an empty path (not specifying the STRIMZI_KEYSTORE_LOCATION). Anyway, I can see this as a potential use case because for the work I am doing on the user operator, you will be able to enable TLS encryption (so specifying truststore) but not necessarily TLS client authentication (through the truststore). I would allow this in the current topic operator as well. Wdyt @scholzj ?

[elisiano]

so would it work if I have an init container doing the setup of an empty keystore and use that? Or does it actually expect to do TLS client auth?

[scholzj]

I'm not sure what exactly would _empty keystore_ be. But at the end you would need to try it out since I do not think anyone from us ever tried this. So we have no idea how it would or wouldn't work. You could also ask on our Slack. I think there are some users using it with MSK. But I'm not sure if they use different config or something because they never reported anything.

[trinadsz]

@ppatierno - We have similar requirement. We want to enable TLS encryption but authentication is sasl plaintext. We want sasl plaintext authentication and TLS encryption. What looked promising for authentication is below property adminClientProps.setProperty(AdminClientConfig.SECURITY_PROTOCOL_CONFIG, "SSL");
If we change it to adminClientProps.setProperty(AdminClientConfig.SECURITY_PROTOCOL_CONFIG, "SSL_PLAINTEXT");
hopefully, this will enable authentication. However, we are trying to find out how to disable TLS client authentication. Do you have pointers to what we can try ? Thanks

[ppatierno]

@trinadsz the possible values for the security protocol are PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL but anyway it's not possible in the current topic operator.
Right now the only thing you can do is disable TLS at all with STRIMZI_TLS_ENABLED = false and passing a STRIMZI_KAFKA_BOOTSTRAP_SERVERS to Kafka plain bootstrap (i.e. 9092) without any TLS or authentication.
Allowing TLS but not TLS client authentication needs changes. Allowing TLS and using SASL plaintext authentication need more changes.
It's not on our roadmap right now but any kind of contribution is very welcome.

[trinadsz]

@ppatierno - Thank you for your prompt reply. Appreciate it.

[elisiano]

Just for whoever stumbles upon this, I was able to make this work without modifications (ie: with MSK using TLS but not client cert auth).
I patched the deployment (we use kustomize) to add an init container that creates the trust/key store (we don't have any private key so we point both variables to the same store, but it could be separate I guess, I haven't tested that).
Both the init container and the topic operator container use the same image and they both mount an emptyDir volume to have access to the same files.

all the init container does is:

STORE=/certs/topic-operator-truststore.p12 # point this to the shared volume
[ -f $STORE ] && rm -f $STORE
keytool -importkeystore -srckeystore /usr/lib/jvm/jre/lib/security/cacerts -srcstorepass changeit -srcstoretype JKS -destkeystore $STORE -deststoretype PKCS12 -deststorepass changeit

Then in the main container I pass the vars:

         - name: STRIMZI_TLS_ENABLED
           value: "true"
         - name: STRIMZI_TRUSTSTORE_LOCATION
           value: /certs/topic-operator-truststore.p12
         - name: STRIMZI_KEYSTORE_LOCATION
           value: /certs/topic-operator-truststore.p12
         - name: STRIMZI_TRUSTSTORE_PASSWORD
           value: changeit
         - name: STRIMZI_KEYSTORE_PASSWORD
           value: changeit

Given they are separate containers, I couldn't create the stores with a random password without sharing it somehow.
This could be improved if somehow the startup scripts would include some logic to accommodate this scenario (so that random passwords could be used), but for as long as i'm concerned this issue can be closed.

[scholzj]

Thanks for the update. Glad you made it work at the end.

[dsouzat]

we were able to get SASL_PLAINTEXT work for our project.