Strimzi-kafka-operator: can reinstall kafka cluster without renew certificat?

[totee19]

Hello,

My openshift 3.11 platform crashed cause enough capacity on disk and /var/lib/docker was full.
I recovered the openshift platform and set log limit. cf https://access.redhat.com/solutions/2334181

So now I meet a problem with strimzi.
I have 2 strimzi

• first strimzi is ephemeral
• second strimzi in persistent storage

on both projects I can't recovered it.
I tried stop and start the platform but it's not working.

And the pod cluster-operator can't reconcile my kafka-cluster (cf log files)

cluster-ephemeral.log
cluster-persistent.log

My question is :
It 's possible reinstall my kafka-cluster without renew all secrets and certificats ?
I would deploy my kafka-cluster like first install but keep my old secret and certificats and kafkauser with kafka-topic.
It's possible to do that?

Best regards,
Tie

[scholzj]

TBH, Judging from the exception, it looks like the certificate secrets are empty or missing? In which case the certificates are probably already lost.

[totee19]

secret and certificats exist on both namespace
I can read it and export it.

[scholzj]

So can you dump them and share them? Feel free to mask the contents, but I want to see the keys etc. If you don't share the content, you should also check if the valid certificates are still inside by decoding them. Because the exceptions when reading the certificates have to come from somewhere.

[totee19]

Here is bit of my secret :

NAME                                                       TYPE                                  DATA      AGE
admin                                                      Opaque                                1         137d
builder-dockercfg-s7gzq                                    kubernetes.io/dockercfg               1         29d
builder-token-rz6zp                                        kubernetes.io/service-account-token   4         29d
builder-token-x4d2l                                        kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-clients-ca                        Opaque                                1         29d
cluster-kafka-persistent-clients-ca-cert                   Opaque                                3         29d
cluster-kafka-persistent-cluster-ca                        Opaque                                1         29d
cluster-kafka-persistent-cluster-ca-cert                   Opaque                                3         29d
cluster-kafka-persistent-cluster-operator-certs            Opaque                                4         29d
cluster-kafka-persistent-entity-operator-certs             Opaque                                4         29d
cluster-kafka-persistent-entity-operator-dockercfg-rd9rh   kubernetes.io/dockercfg               1         29d
cluster-kafka-persistent-entity-operator-token-fjrvg       kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-entity-operator-token-rf5t6       kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-kafka-brokers                     Opaque                                20        29d
cluster-kafka-persistent-kafka-dockercfg-8bxxx             kubernetes.io/dockercfg               1         29d
cluster-kafka-persistent-kafka-token-4kqjc                 kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-kafka-token-szgvq                 kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-zookeeper-dockercfg-8426h         kubernetes.io/dockercfg               1         29d
cluster-kafka-persistent-zookeeper-nodes                   Opaque                                0         29d
cluster-kafka-persistent-zookeeper-token-w7ll4             kubernetes.io/service-account-token   4         29d
cluster-kafka-persistent-zookeeper-token-wdzt5             kubernetes.io/service-account-token   4         29d

Do you think the problem come from of the secret name keep like ?

• cluster-kafka-persistent-zookeeper-token-wdzt5
• cluster-kafka-persistent-zookeeper-token-w7ll4

[scholzj]

I think this is your problem:

cluster-kafka-persistent-zookeeper-nodes Opaque 0 29d

Can you try to delete this secret and the Strimzi cluster operator pod (just the pod, not the deployment) to have it restarted?

[totee19]

I don't know what it's doing
but it's that:

2020-10-23 20:15:50 INFO  AbstractOperator:173 - Reconciliation #3(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:16:00 INFO  AbstractOperator:318 - Reconciliation #3(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled
2020-10-23 20:17:50 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-uat...
2020-10-23 20:17:50 INFO  AbstractOperator:173 - Reconciliation #4(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:18:00 INFO  AbstractOperator:318 - Reconciliation #4(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled
2020-10-23 20:19:50 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-uat...
2020-10-23 20:19:50 INFO  AbstractOperator:173 - Reconciliation #5(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:19:59 INFO  AbstractOperator:318 - Reconciliation #5(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled
2020-10-23 20:21:50 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-uat...
2020-10-23 20:21:50 INFO  AbstractOperator:173 - Reconciliation #6(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:21:59 INFO  AbstractOperator:318 - Reconciliation #6(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled
2020-10-23 20:23:50 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-uat...
2020-10-23 20:23:50 INFO  AbstractOperator:173 - Reconciliation #7(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:23:59 INFO  AbstractOperator:318 - Reconciliation #7(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled
2020-10-23 20:25:50 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-uat...
2020-10-23 20:25:50 INFO  AbstractOperator:173 - Reconciliation #8(timer) Kafka(kafka-uat/cluster-kafka-persistent): Kafka cluster-kafka-persistent should be created or updated
2020-10-23 20:26:00 INFO  AbstractOperator:318 - Reconciliation #8(timer) Kafka(kafka-uat/cluster-kafka-persistent): reconciled

[scholzj]

Well, that looks good ... no errors. So is the cluster running now?

[totee19]

Thank you very much for your help!!!
It's working fine

[totee19]

Have you an explain why all secret cluster-kafka-zookeeper-nodes altered for each namespace ?

[scholzj]

I don't. I will have a look at the NPE caused by the empty secret. But I do not know why it was empty.

[totee19]

Ok thank you.
Happy to contribute for improve Strimzi.