Strimzi-kafka-operator: the number of partitions and replicas of topic was changed by topic operator

[foxyriver]

Describe the bug
1、create kafka topic with 3 partitions and 3 replicas
2、use kafka client to production and consumption
3、the number of partitions and replicas of topic was changed to 1

To Reproduce
Occasionally present

Expected behavior
the number of partitions and replicas should not be changed by topic operator

Environment (please complete the following information):

• Strimzi version: [0.11.1]
• Installation method: [YAML files]
• Kubernetes cluster: [Kubernetes 1.12]

[foxyriver]

[root@10 ~]# kubectl describe kafkatopic -n bmt testbyss1
Name: testbyss1
Namespace: bmt
Labels: strimzi.io/cluster=liuhaotest
Annotations:
API Version: kafka.strimzi.io/v1alpha1
Kind: KafkaTopic
Metadata:
Creation Timestamp: 2019-04-23T03:20:55Z
Generation: 1
Resource Version: 14939267
Self Link: /apis/kafka.strimzi.io/v1alpha1/namespaces/bmt/kafkatopics/testbyss1
UID: ce2b3cef-6576-11e9-98d0-fa163e3db4a9
Spec:
Config:
Segment . Bytes: 524288
Partitions: 3
Replicas: 3
Events:
[root@10 ~]# kubectl describe kafkatopic -n bmt testbyss1
Name: testbyss1
Namespace: bmt
Labels: strimzi.io/cluster=liuhaotest
Annotations:
API Version: kafka.strimzi.io/v1alpha1
Kind: KafkaTopic
Metadata:
Creation Timestamp: 2019-04-23T03:20:55Z
Generation: 1
Resource Version: 14940951
Self Link: /apis/kafka.strimzi.io/v1alpha1/namespaces/bmt/kafkatopics/testbyss1
UID: ce2b3cef-6576-11e9-98d0-fa163e3db4a9
Spec:
Config:
Message . Format . Version: 2.1-IV2
Retention . Bytes: 524288
Partitions: 1
Replicas: 1
Topic Name: testbyss1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Info io.strimzi.operator.topic.TopicOperator KafkaTopic is incompatible with the topic metadata. The topic metadata will be treated as canonical.

[scholzj]

I do not think the TO would change it to 1 on its own. Kafka even doesn't allow to decrease the number of partitions. So I would expect that it was already created with 1. A theory about how this could have happened:

• The default settings in Kafka is that when consumer / producer connects to a topic which doesn't exist it will be auto-created by Kafka with the default settings (by default 1 partition, 1 replica).
• The creation of topics by the Cluster Operator is asynchronous. So when you do kubectl apply or kubectl create, it only created the resource in the ETCD database. Afterwards it notifies the Topic Operator which creates the topic.
• But since the previous step is asynchronous, it can have some delay. If you didn't disabled your Kafka brokers from auto creating the topics, it might happen that yous consumer / producer connects before the topic was created by the Topic Operator and triggers its creation in Kafka with the default settings. The Topic Operator would then basically just update the CR to the value form Kafka.

Is it possible that something like this happened in your case? This is pretty much a race condition which the operator cannot easily handle.

To avoid this, the best is to disable the topic autocreation in Kafka:

# ...
spec:
  kafka:
    # ...
    config:
      auto.create.topics.enable: "false"
    # ...

[foxyriver]

@scholzj i agree with you. but it is difficult to reproduce this condition. i describe my test :

1. using consumer to connect to a topic which doesn't exist. then i found that kafka creates default topic(1 partition,1 replica), and topic operator creates kafkatopic resource also(1 partition,1 replica).
2. using kubectl apply to create kafkatopic(3 partition,3replica) which has same name with the topic created in first step. then i found that topic operator merged this kafkatopic and kafkatopic created in first step, finally the kafkatopic with 3 partition and 1 replica(replica can not be expanded).

so when kafka has created default topic and topic operator has not created kafkatopic in the first step,then we execute the second step immediately, may reproduce this condition. am i right?

[scholzj]

Right, exactly. And because we have only little control on how Kubernetes handles things, the only real solution is to disable the auto-creation.

Even if the TO was able to change the replicas, the change in partition could can for example result in badly partitioned messages which are sent before this happens etc.

[foxyriver]

thanks for your answer :)

[foxyriver]

config auto.create.topics.enable:false resolved my issue