Strimzi-kafka-operator: [Question] Is possible to start two user and topic operators (standalone) watching for different namespaces feeding same Kafka cluster?

[cristianburca]

Hello.

We're using Strimzi connected with AWS MSK and the integration went perfectly.

But we have a situation: we'll work with multiple namespaces in k8s from where will create topics and users feeding the same Kafka server. And we see that topics conflicts each other and get deleted even STRIMZI_RESOURCE_LABELS is different.
Do you have any suggestion? Is this a limitation?

The setup is as follows:

Helm charts are based on the suggested setup for standalone operators:
https://github.com/strimzi/strimzi-kafka-operator/tree/master/install

Topic operator helm chart values (secrets references, k8s resources and some other env variables are cut) :

name: strimzi-topic-operator
namespace: strimzi-operator

topicOperator:
  - topic: strimzi-msk-namespace1
    image: quay.io/strimzi/operator
    tag: 0.20.0
    env:
      STRIMZI_RESOURCE_LABELS: "strimzi.io/cluster=strimzi-msk-namespace1"
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS: "same AWS MSK kafka cluster"
      STRIMZI_ZOOKEEPER_CONNECT: "same AWS MSK kafka cluster"
      STRIMZI_TLS_ENABLED: "true"
      STRIMZI_NAMESPACE: "namespace1"
  - topic: strimzi-msk-namespace2
     image: quay.io/strimzi/operator
     tag: 0.20.0
     env:
      STRIMZI_RESOURCE_LABELS: "strimzi.io/cluster=strimzi-msk-namespace2"
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS: "same AWS MSK kafka cluster"
      STRIMZI_ZOOKEEPER_CONNECT: "same AWS MSK kafka cluster"
      STRIMZI_TLS_ENABLED: "true"
      STRIMZI_NAMESPACE: "namespace2"

This helm chart values will create the following resources:

• A deployment in namespace strimzi-operator (2 pods)

NAME                                                         READY   STATUS    RESTARTS   AGE
strimzi-topic-operator-strimzi-msk-namespace1-74cd9cc78-c29st     1/1     Running   0          34m
strimzi-topic-operator-strimzi-msk-namespace2-84dd5d5447-2zhff   1/1     Running   0          34m

• A role in both helm chart release namespace (strimzi-operator) and also in the namespaces to watch (namespace1, namespace2)
• A roleBinding in both helm chart release namespace and also in the namespaces to watch (namespace1, namespace2)

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    topic: strimzi-topic-operator-strimzi-msk-namespace1
  name: strimzi-topic-operator-rolebinding-strimzi-msk-namespace1
  namespace: namespace1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: strimzi-topic-operator-role-strimzi-msk-namespace1
subjects:
- kind: ServiceAccount
  name: strimzi-topic-operator-strimzi-msk-namespace1
  namespace: strimzi-operator

• A serviceAccount in helm chart release namespace
• A namespaced CDR

User operator helm chart values (secrets references, k8s resources and some other env variables are cut):

name: strimzi-user-operator
namespace: strimzi-operator

userOperator:
  - user: strimzi-msk-namespace1
    image: quay.io/strimzi/operator
    tag: 0.20.0
    env:
      STRIMZI_LABELS: "strimzi.io/cluster=strimzi-msk-namespace1"
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS: "same AWS MSK kafka cluster"
      STRIMZI_ZOOKEEPER_CONNECT: "same AWS MSK kafka cluster"
      STRIMZI_TLS_ENABLED: "true"
      STRIMZI_NAMESPACE: "namespace1"
  - user: strimzi-msk-namespace2
    image: quay.io/strimzi/operator
    tag: 0.20.0
    env:
      STRIMZI_LABELS: "strimzi.io/cluster=strimzi-msk-namespace2"
      STRIMZI_KAFKA_BOOTSTRAP_SERVERS: "same AWS MSK kafka cluster"
      STRIMZI_ZOOKEEPER_CONNECT: "same AWS MSK kafka cluster"
      STRIMZI_TLS_ENABLED: "true"
      STRIMZI_NAMESPACE: "namespace2"

This helm chart will create the following resources:

• A deployment in namespace strimzi-operator (2 pods)

NAME                                                         READY   STATUS    RESTARTS   AGE
strimzi-user-operator-strimzi-msk-namespace1-6d65dbd67-xxlb6      1/1     Running   0          34m
strimzi-user-operator-strimzi-msk-namespace2-7c7df7dc68-9gpbl    1/1     Running   0          34m

• A role in both helm chart release namespace (strimzi-operator) and also in the namespaces to watch (namespace1, namespace2)
• A roleBinding in both helm chart release namespace and also in the namespaces to watch (namespace1, namespace2)

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    user: strimzi-user-operator-strimzi-msk-namespace1
  name: strimzi-user-operator-rolebinding-strimzi-msk-namespace1
  namespace: namespace1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: strimzi-user-operator-role-strimzi-msk-namespace1
subjects:
- kind: ServiceAccount
  name: strimzi-user-operator-strimzi-msk-namespace1
  namespace: strimzi-operator

• A serviceAccount in helm chart release namespace
• A namespaced CDR

Strimzi:

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: my-topic-namespace1
  labels:
    strimzi.io/cluster: "strimzi-msk-namespace1"
  namespace: namespace1
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: my-topic-namespace2
  labels:
    strimzi.io/cluster: "strimzi-msk-namespace2"
  namespace: namespace2
spec:
  partitions: 3
  replicas: 1
---
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: my-user-namespace1
  namespace: namespace1
  labels:
    strimzi.io/cluster: "strimzi-msk-namespace1"
spec:
  authentication:
    type: tls
---
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: my-user-namespace2
  namespace: namespace2
  labels:
    strimzi.io/cluster: "strimzi-msk-namespace2"
spec:
  authentication:
    type: tls

Some logs of the pod

2020-11-25 15:19:27 INFO  ZkTopicsWatcher:96 - Created topics: [my-topic-namespace1]
2020-11-25 15:19:27 INFO  TopicOperator:684 - 1521|/brokers/topics -my-topic-namespace1|15948178: All three topics are identical
2020-11-25 15:19:27 INFO  TopicOperator:536 - 1522|/brokers/topics +my-topic-namespace1|15948178: Reconciling topic my-topic-namespace1, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:nonnull
2020-11-25 15:19:27 INFO  TopicOperator:684 - 1522|/brokers/topics +my-topic-namespace1|15948178: All three topics are identical
2020-11-25 15:19:27 INFO  ZkTopicsWatcher:80 - Deleted topics: [my-topic-namespace1]
2020-11-25 15:19:27 INFO  TopicOperator:536 - 1523|/brokers/topics -my-topic-namespace1|15948178: Reconciling topic my-topic-namespace1, k8sTopic:nonnull, kafkaTopic:null, privateTopic:nonnull
2020-11-25 15:19:27 INFO  K8sTopicWatcher:43 - 1524|kube =my-topic-namespace1|15948359: event MODIFIED on resource my-topic-namespace1 generation=2, labels={strimzi.io/cluster=strimzi-msk-namespace1}
2020-11-25 15:19:28 INFO  K8sTopicWatcher:43 - 1525|kube -my-topic-namespace1|15948364: event DELETED on resource my-topic-namespace1 generation=2, labels={strimzi.io/cluster=strimzi-msk-namespace1}

[scholzj]

I'm afraid the operators do not work like that. You can have only one of them installed and connected to a single MSK cluster. When you have more of them, the following would be happening:
1) You create KafkaUser in namespace1
2) The operator in namespace1 creates the KafkaUser in Kafka (ACLs, SCRAM password, quotas etc.)
3) Operator in namespace2 will see that these values are in Kafka. But he will not see the corresponding KafkaUser resource. So it decides to delete the ALCs, SCRAM passwords, quotas etc. because according to namespace2, they should not be there.
4) The operator in namespace1 will later see they are not in Kafka, but it has the KafkaUSer. So it will create it again.
5) And so on ...

So they will basically play back and forth creating and deleting the user. Similarly with the topics. So this is not possible.

[cristianburca]

Thanks @scholzj. Got it. Thanks for your quick reply.

I see this functionality to watch multiple namespaces is already supported for cluster operator (strimzi managed).
Could be a different thing, but do you suggest a workaround when comes to standalone install for topic operator within external kafka cluster?

[scholzj]

This is very different between Cluster Operator and the Topic / User Operators. The Cluster operator operates within an environment which has namespace support (Kubernetes). But the User / Topic Operator basically operates in Kafka which has no namespace concept. So it is not completely trivial to implement this for them, because how do you deal with "my-user` being created in multiple namespaces with different ACLs? The operator would need to decide which one is the right one and kick the rest. It is something we hope to get to, but it has not yet been implemented.

[cristianburca]

Thanks again. Let me know if you want me to close the issue or move it to some enhancements backlog.

[scholzj]

I think we can close it.