Strimzi-kafka-operator: [Question] ... No CA found. error while creating kafka cluster with ingress and own CA certificate

[yogeshbidari]

I have used my own CA certificate to configure the kafka with ingress.

After spinning up the cluster the pods are going to "CrashLoopBackOff" with below error.

Detected Zookeeper ID 1
Preparing truststore
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore is complete
Looking for the right CA
No CA found. Thus exiting.

I have created the cluster-ca-cert secret with file having certificates and cluster-ca secret with key file.
The certificates are in below order

1. CA certificate
2. Intermediate certificate
3. Root CA certificate

Please guide me here on setting up ingress with own CA certificate.

[scholzj]

Can you share your Kafka CR and the secret YAMLs you created? Also, what version do you use?

Also, what exactly do you mean _setting up Ingress with own CA_? If you just want to have your own certificate for the listener, then maybe the listener certificates would be easier: https://strimzi.io/docs/operators/latest/full/using.html#proc-installing-certs-per-listener-str

[yogeshbidari]

Please find the kafka CR below:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: default
spec:
  clusterCa:
    generateCertificateAuthority: false
  kafka:
    version: 2.5.0
    replicas: 3
    jmxOptions: {}
    readinessProbe:
       initialDelaySeconds: 15
       timeoutSeconds: 5
    livenessProbe:
       initialDelaySeconds: 15
       timeoutSeconds: 5
    listeners:
      plain: {}
      external:
        type: ingress
        tls: true
        class: nginx
        configuration:
          bootstrap:
            host: bootstrap.ybjust.online
          brokers:
          - broker: 0
            host: broker-0.ybjust.online
          - broker: 1
            host: broker-1.ybjust.online
          - broker: 2
            host: broker-2.ybjust.online
    metrics:
      lowercaseOutputName: true
      rules:
        - pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
          name: kafka_server_$1_$2
          type: GAUGE
          labels:
            clientId: "$3"
            topic: "$4"
            partition: "$5"
        - pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
          name: kafka_server_$1_$2
          type: GAUGE
          labels:
            clientId: "$3"
            broker: "$4:$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
          name: kafka_$1_$2_$3_percent
          type: GAUGE
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
          name: kafka_$1_$2_$3_percent
          type: GAUGE
        - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
          name: kafka_$1_$2_$3_percent
          type: GAUGE
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
          name: kafka_$1_$2_$3_total
          type: COUNTER
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
          name: kafka_$1_$2_$3
          type: GAUGE
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
          labels:
            "$4": "$5"
            "$6": "$7"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            "$6": "$7"
            quantile: "0.$8"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
          labels:
            "$4": "$5"
        - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            "$4": "$5"
            quantile: "0.$6"
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
          name: kafka_$1_$2_$3_count
          type: COUNTER
        - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
          name: kafka_$1_$2_$3
          type: GAUGE
          labels:
            quantile: "0.$4"
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.4"
      ssl.endpoint.identification.algorithm: ""
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
      class: standard
  zookeeper:
      replicas: 3
      readinessProbe:
        initialDelaySeconds: 15
        timeoutSeconds: 5
      livenessProbe:
        initialDelaySeconds: 15
        timeoutSeconds: 5
      storage:
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: standard
      metrics:
        lowercaseOutputName: true
        rules:
          - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
            name: "zookeeper_$2"
          - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
            name: "zookeeper_$3"
            labels:
              replicaId: "$2"
          - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
            name: "zookeeper_$4"
            labels:
              replicaId: "$2"
              memberType: "$3"
          - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
            name: "zookeeper_$4_$5"
            labels:
              replicaId: "$2"
              memberType: "$3"
          - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)"
            name: "zookeeper_$2"
          - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+), name1=(InMemoryDataTree)><>(\\w+)"
            name: "zookeeper_$2_$3"
  entityOperator:
      topicOperator: {}
      userOperator: {}
  kafkaExporter:
      topicRegex: ".*"
      groupRegex: ".*"

Strimzi kafka version: 0.19.0

I am using ingress to expose kafka cluster. It is working fine if I use default generated certificates.
When I tried to use my own CA certificates by creating cluster-ca and cluster-ca-cert secrets it failing with above error.

  clusterCa:
    generateCertificateAuthority: false

[scholzj]

Can you also share the secrets you created? (feel free to mask the values, but the structure and metadata are important!)

You also didn't answered this:

Also, what exactly do you mean setting up Ingress with own CA? If you just want to have your own certificate for the listener, then maybe the listener certificates would be easier: https://strimzi.io/docs/operators/latest/full/using.html#proc-installing-certs-per-listener-str

[Dryr]

Hello, I am experiencing a similar issue with strimzi 0.20

I am trying to setup TLS for an external listener.
I installed my own certificate by following this section of the documentation.

When creating the new Kafka cluster, I specified the following option:
generateCertificateAuthority: false

Here's the yaml used for creating the kafka cluster:

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster-tls
spec:
  clusterCa:
    generateCertificateAuthority: false
  kafka:
    replicas: 1
    listeners:
      internal:
        port: 9093
        type: internal
        tls: false
      external:
        port: 9094
        type: nodeport
        tls: true
        overrides:
          bootstrap:
            nodePort: 32200
    storage:
      type: ephemeral
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

After applying this yaml file, a new zookeeper pod is created but crashes immediatly.
This is the log of the zookeeper container:

Detected Zookeeper ID 1
Preparing truststore
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/zookeeper/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore is complete
Looking for the right CA
No CA found. Thus exiting.

I checked the code that throws this error.

The error comes from this function:

function find_ca {
    for ca in "$1"/*; do
        if openssl verify -CAfile "$ca" "$2" &> /dev/null; then
            echo "$ca"
        fi
    done
}

It seems that the code checks whether there is a chain between a certificate contained in the folder _"/opt/kafka/cluster-ca-certs"_ and the certificate _"/opt/kafka/zookeeper-node-certs/$HOSTNAME.crt"_

The folder _"/opt/kafka/cluster-ca-certs"_ is a mount point of the secret _"my-cluster-tls-clients-ca-cert"_, which contains the certificate I manually installed.
However, the certificate _"/opt/kafka/zookeeper-node-certs/$HOSTNAME.crt"_ is from the secret _"my-cluster-tls-zookeeper-nodes"_, which has been generated automatically by the strimzi operator.
These certificates can not be part of the same chain. That would explain why the openssl verify never succeeds.
I have already checked inside the container, the certificates files are present and match the certificates contained in the associated secrets.

[scholzj]

@Dryr

• /opt/kafka/cluster-ca-certs is my-cluster-tls-cluster-ca-cert, not my-cluster-tls-clients-ca-cert
• my-cluster-tls-zookeeper-nodes is generated and signed by my-cluster-tls-cluster-ca-cert and my-cluster-tls-cluster-ca you were supposed to provide. So I do not see any reason why they should not match the chain.

[Dryr]

My bad, I copied the wrong name, I meant my-cluster-tls-cluster-ca-cert.
I tried to use the same openssl verify command and I got this error

[kafka@my-cluster-tls-zookeeper-0 kafka]$ openssl verify -CAfile /opt/kafka/cluster-ca-certs/ca.crt /opt/kafka/zookeeper-node-certs/my-cluster-tls-zookeeper-0.crt 
/opt/kafka/zookeeper-node-certs/my-cluster-tls-zookeeper-0.crt: O = io.strimzi, CN = my-cluster-tls-zookeeper

error 20 at 0 depth lookup:unable to get local issuer certificate

If I remember correctly it means that my certificate is not a root certificate. I'll check this and keep you in touch if I get it to work.

[Dryr]

I still get the same errors. But I found something intriguing

This time I tried with only the root certificate first.
After that I added the whole certificate chain into the secret my-cluster-tls-cluster-ca-cert (4 certificates in total). I also created a keystore that I named "ca.p12" in which I imported all the certificates, and also added it into the secret my-cluster-tls-cluster-ca-cert.

The secret contained the following files:

• ca.crt
• int1.crt
• int2.crt
• root.crt
• ca.p12
• ca.password

I tried again the command

[kafka@my-cluster-tls-zookeeper-0 kafka]$ openssl verify -CAfile "/opt/kafka/cluster-ca-certs/root.crt" "/opt/kafka/cluster-ca-certs/int1.crt" "/opt/kafka/cluster-ca-certs/int2.crt" "/opt/kafka/cluster-ca-certs/ca.crt" "/opt/kafka/zookeeper-node-certs/$HOSTNAME.crt"
/opt/kafka/cluster-ca-certs/int1.crt: OK
/opt/kafka/cluster-ca-certs/int2.crt: OK
/opt/kafka/cluster-ca-certs/ca.crt: OK
/opt/kafka/zookeeper-node-certs/my-cluster-tls-zookeeper-0.crt: O = io.strimzi, CN = my-cluster-tls-zookeeper
error 20 at 0 depth lookup:unable to get local issuer certificate

Same error as in my previous post

I checked the content of the zookeeper certificate

[kafka@my-cluster-tls-zookeeper-0 kafka]$ openssl x509 -in "/opt/kafka/zookeeper-node-certs/$HOSTNAME.crt" -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=FR, O=***, OU=***, CN=<same cn as the ca.crt>
        Validity
            Not Before: Nov 27 09:51:25 2020 GMT
            Not After : Nov 27 09:51:25 2021 GMT
        Subject: O=io.strimzi, CN=my-cluster-tls-zookeeper
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:*.my-cluster-tls-zookeeper-client.my-kafka-project.svc, DNS:*.my-cluster-tls-zookeeper-client.my-kafka-project.svc.cluster.local, DNS:my-cluster-tls-zookeeper-0.my-cluster-tls-zookeeper-nodes.my-kafka-project.svc.cluster.local, DNS:my-cluster-tls-zookeeper-0.my-cluster-tls-zookeeper-nodes.my-kafka-project.svc, DNS:*.my-cluster-tls-zookeeper-nodes.my-kafka-project.svc, DNS:my-cluster-tls-zookeeper-client.my-kafka-project.svc, DNS:my-cluster-tls-zookeeper-client.my-kafka-project.svc.cluster.local, DNS:my-cluster-tls-zookeeper-client, DNS:my-cluster-tls-zookeeper-client.my-kafka-project, DNS:*.my-cluster-tls-zookeeper-nodes.my-kafka-project.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
         ...

I can see the CN of my certificate "ca.crt" in the Issuer of the zookeeper certificate.
As you said above the zookeeper certificate really is generated and signed by my ca.crt, but apparently the openssl verify command disagrees

I'm confused now, I did not think that was possible

[scholzj]

The secret contained the following files:

• ca.crt
• int1.crt
• int2.crt
• root.crt
• ca.p12
• ca.password

Well, that is why I'm asking from the beginning for the secrets but none of you provided them. It is hard to help when you are not listening. You are definitely not expected to have anything other than the ca.* files in the secret. The ca.crt should contain the whole chain. So maybe that is what is causing the problem?

I understand that you probably cannot share the actual certificates. But you can have a look here (I think I shared it earlier with @yogeshbidari on Slack, but I didn't hared it here yet): https://github.com/scholzj/strimzi-custom-ca-test ... this repo generates multilevel CA and uses it with Strimzi. You can probably dig in there to see how it creates the secrets and the ca.crt file. That seems to work fine for me. So that might help you to identify the differences between this setup and your setup.

[Dryr]

Thanks for the link to the test git project, I'm looking into it.

I also tried putting the whole chain inside the ca.crt, while being careful of putting everything in the right order (according to the strimzi documentation), but I'm getting the same error.
We've been using the same chain for a while to secure a few https websites, so I think the chain is valid.
There must be something wrong with the way I'm adding it to the ...-cluster-ca-cert secret, but for the life of me I don't see it.

Here are the resulting yaml files of the secrets I created:

apiVersion: v1
data:
  ca.crt: ...
kind: Secret
metadata:
  creationTimestamp: "2020-11-27T14:06:42Z"
  labels:
    strimzi.io/cluster: my-cluster-tls
    strimzi.io/kind: Kafka
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:ca.crt: {}
      f:metadata:
        f:labels:
          .: {}
          f:strimzi.io/cluster: {}
          f:strimzi.io/kind: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-27T14:06:42Z"
  name: my-cluster-tls-cluster-ca-cert
  namespace: my-kafka-project
  resourceVersion: "15832848"
  selfLink: /api/v1/namespaces/my-kafka-project/secrets/my-cluster-tls-cluster-ca-cert
  uid: ...
type: Opaque

apiVersion: v1
data:
  ca.key: ...
kind: Secret
metadata:
  creationTimestamp: "2020-11-27T14:06:42Z"
  labels:
    strimzi.io/cluster: my-cluster-tls
    strimzi.io/kind: Kafka
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:ca.key: {}
      f:metadata:
        f:labels:
          .: {}
          f:strimzi.io/cluster: {}
          f:strimzi.io/kind: {}
      f:type: {}
    manager: kubectl
    operation: Update
    time: "2020-11-27T14:06:42Z"
  name: my-cluster-tls-cluster-ca
  namespace: my-kafka-project
  resourceVersion: "15832845"
  selfLink: /api/v1/namespaces/my-kafka-project/secrets/my-cluster-tls-cluster-ca
  uid: ...
type: Opaque

[scholzj]

The secrets look good to me I guess. The content of the file matters as well - in case of the chain in the ca.crt it is also the order in which they are stored there. So make sure it is the same as in my repo.

[tusharbhasme]

We are going through the very same issue. I have right now generated my own certs following the steps shown in the github repo shared above. After generating my own certs and keys, I simply updated the k8s secrets with them (by modifying load.sh in the repo and manually too). The cluster-operator detected some change but then kept giving "Failed to acquire lock within 10000ms". After waiting for some time, I just restarted the cluster-operator, but the cert and key changes do not seem to have been propagated to zookeeper and worker nodes. My client with old certs is still able to talk to the cluster configured with new certs.
some more warns and errors from logs:

2020-11-30 19:25:46 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2020-11-30 19:25:56 WARN  AbstractOperator:247 - Reconciliation #8(timer) Kafka(kafka/my-cluster): Failed to acquire lock lock::kafka::Kafka::my-cluster within 10000ms.
2020-11-30 19:26:47 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
2020-11-30 19:26:48 WARN  ZookeeperScaler:156 - Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
2020-11-30 19:26:48 WARN  KafkaAssemblyOperator:1630 - {}: Failed to verify Zookeeper configuration
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more
2020-11-30 19:26:48 ERROR AbstractOperator:175 - Reconciliation #6(timer) Kafka(kafka/my-cluster): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more
2020-11-30 19:26:48 WARN  AbstractOperator:330 - Reconciliation #6(timer) Kafka(kafka/my-cluster): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more
2020-11-30 19:27:46 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2020-11-30 19:27:46 INFO  AbstractOperator:173 - Reconciliation #9(timer) Kafka(kafka/my-cluster): Kafka my-cluster should be created or updated
2020-11-30 19:29:46 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2020-11-30 19:29:56 WARN  AbstractOperator:247 - Reconciliation #10(timer) Kafka(kafka/my-cluster): Failed to acquire lock lock::kafka::Kafka::my-cluster within 10000ms.
2020-11-30 19:31:46 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka...
2020-11-30 19:31:56 WARN  AbstractOperator:247 - Reconciliation #11(timer) Kafka(kafka/my-cluster): Failed to acquire lock lock::kafka::Kafka::my-cluster within 10000ms.
2020-11-30 19:32:47 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
2020-11-30 19:32:47 WARN  ZookeeperScaler:156 - Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
2020-11-30 19:32:47 WARN  KafkaAssemblyOperator:1630 - {}: Failed to verify Zookeeper configuration
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more
2020-11-30 19:32:48 ERROR AbstractOperator:175 - Reconciliation #9(timer) Kafka(kafka/my-cluster): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more
2020-11-30 19:32:48 WARN  AbstractOperator:330 - Reconciliation #9(timer) Kafka(kafka/my-cluster): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.ZookeeperScalingException: Failed to connect to Zookeeper my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181. Connection was not ready in 300000 ms.
    at io.strimzi.operator.cluster.operator.resource.ZookeeperScaler.lambda$connect$5(ZookeeperScaler.java:157) ~[io.strimzi.cluster-operator-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:195) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:125) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:121) ~[io.strimzi.operator-common-0.19.0.jar:0.19.0]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.50.Final.jar:4.1.50.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for ZooKeeperAdmin connection to my-cluster-zookeeper-0.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-1.my-cluster-zookeeper-nodes.kafka.svc:2181,my-cluster-zookeeper-2.my-cluster-zookeeper-nodes.kafka.svc:2181 to be connected
    ... 11 more

[tusharbhasme]

OK, i see some errors in the zookeeper pods:

2020-11-30 19:40:04,749 INFO SSL handler added for channel: [id: 0xee0dd260, L:/172.31.7.243:2181 - R:/172.31.42.167:57654] (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
2020-11-30 19:40:04,764 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
2020-11-30 19:40:04,764 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
    at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:668)
    at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:623)
    at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:441)
    at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:420)
    at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
    at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1329)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
    ... 17 more
2020-11-30 19:40:06,425 INFO Processing ruok command from /127.0.0.1:50532 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]

[scholzj]

I guess the errors suggest that the Cluster Operator has a wrong certificate. So I guess you can check it manually if it matches the CA. You do not describe any steps for reproducing nor share complete logs or the secrets. so other than that, you can just check the repository I shared above to see if that corresponds to what you did.

[tusharbhasme]

After trying and retrying, verifying and re-verifying certs, recreating my test cluster, I just noticed that the truststore in the zookeeper pods are not updating with the ca cert, even though the ca cert itself is getting updated from the updated secrets. Any reason why the truststore is not getting updated? The zookeeper pod logs are only printing what I already shared above.

[tusharbhasme]

Any flag which we can enable to debug further?

[scholzj]

@tusharbhasme What exactly does it mean not updating with the CA cert? Do you have this problem after changing the CA certificate? None of the logs show any change to the CA certificate / key. I'm sorry, but if you do not explain what exactly are you doing, what your configuration is nobody will be able to help you. Did you compared the certificates from the secret whether they are signed with your CA as I asked you about? Do they match or not?

[tusharbhasme]

@scholzj Sorry for being abstract. This is what I was trying to do:
We already have a cluster running with Strimzi generated certs. We now need to use the certs signed by our CA, and update the same to Strimzi. I was following https://strimzi.io/docs/0.8.1/#installing-your-own-ca-certificates-str till now which seems be missing few critical information like labeling the new secrets.

I then realized the doc was for a very old version and got to know updated steps from https://strimzi.io/docs/operators/latest/using.html#certificate-authorities-str. I then created a cluster using my certs where it just created zookeeper pods and no kafka pods. When I redeployed the cluster with certs from the tool, it worked! So I thought must be some certificate issue!

Even though the steps mention (which I noted just today) that cluster should not yet be deployed, it was always working with the certs generated by ur tool, ie, with cluster already running, and then deleting and recreating k8s secrets with cert info, without restarting anything. However, same did not work with certs signed by our CA and I thought something must be wrong in the certs. At this point, when I looked into zookeeper pods, I noticed that /opt/kafka/cluster-ca-certs/ca.crt was having the correct info as specified in the cluster-ca-cert in k8s secret. However, the truststore still had only the old ca cert (this is what I meant by truststore not being updated with ca cert.)

However, when I tried again today with our certs with a totally new deployment, it started working. Same steps did not work yesterday as mentioned above.

So now, my question would be, isn't there any way Strimzi could auto update the cluster with new certs? Seems it is already doing some work of copying the new certs from k8s secrets to the pods.

[Zeka13]

Just worked out this for me, sharing my configuration which is started kafka for me sucessfully

kafka-cert.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: My-kafka-cert
spec:
  isCA: true
  subject:
    organizations:
      - My
  duration: 26280h # 3 years
  commonName: My Kafka
  issuerRef:
    kind: Issuer
    name: selfsigned-issuer
  secretName: My-kafka-cert

kafka-persistent.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my
spec:
  kafka:
    clusterCa:
      generateCertificateAuthority: false
    tolerations:
      - key: "role"
        operator: "Equal"
        value: "kafka"
        effect: "NoSchedule"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: agentpool
                  operator: In
                  values:
                    - kafka
    version: 2.5.0
    replicas: 3
    listeners:
#      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 3000Gi
        deleteClaim: false
  zookeeper:
    tolerations:
      - key: "role"
        operator: "Equal"
        value: "zookeeper"
        effect: "NoSchedule"
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: agentpool
                  operator: In
                  values:
                    - zookeeper
    replicas: 3
    storage:
      type: persistent-claim
      size: 1Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafkaExporter:
    groupRegex: ".*"
    topicRegex: ".*"

Installation process:

kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
kubectl apply --namespace kafka -f ../resource-manifests/clusters/kafka/kafka-cert.yaml
sleep 5
kubectl -n kafka get secret my-kafka-cert -o=go-template='{{index .data "tls.key"}}' | base64 -d > /tmp/kafka-key.crt
kubectl -n kafka get secret my-kafka-cert -o=go-template='{{index .data "ca.crt"}}' | base64 -d > /tmp/kafka-ca.crt
kubectl -n kafka create secret generic my-cluster-ca-cert --from-file=ca.crt=/tmp/kafka-ca.crt
kubectl -n kafka create secret generic my-cluster-ca --from-file=ca.key=/tmp/kafka-key.crt
kubectl -n kafka label secret my-cluster-ca-cert strimzi.io/kind=Kafka strimzi.io/cluster=my
kubectl -n kafka label secret my-cluster-ca strimzi.io/kind=Kafka strimzi.io/cluster=my
kubectl apply --namespace kafka -f ../resource-manifests/clusters/kafka/kafka-persistent.yaml

[scholzj]

@tusharbhasme You are not really expected to move from Strimzi CA to custom CA in the middle (at least in a sense that there are no tests covering this etc.). That can definitely cause issues. And in general, if you use Strimzi 0.8, there have been a lot of bugfixes and improvements since.

[tusharbhasme]

@scholzj I was referring that link as it was the one Google pointed me to.. I never bothered to check the version it was for until I started running into issues :-(. We are actually on 19.0

[scholzj]

Ok. I'm not aware of any bugs in 0.19.0 with regards to that. But the part about the change in the middle not being tested still applies there.

[knacktim]

@Zeka13 I don't believe your chart is actually using your imported CA. The clusterCa property should be at the same level as kafka

spec:
  clusterCA:
    generateCertificateAuthority: false
  kafka:
     ...

Reference: