Strimzi-kafka-operator: MirrorMaker 2 creates topics but doesn't mirror the data

[eazhilan-nagarajan]

Hi there,

Hope you're all doing good. In the past I came with questions and got answers and hopeful this time as well.

Let me explain the issue I face!
What is the issue?
I have setup MirrorMaker2 between two clusters, in the target cluster the topics from the source cluster are created, like (source-cluster.topic-1) but the data are not mirrored.

Some more details of how I ended up here:
Initially I had setup the MM2 between two Kafka clusters and everything was working fine until I deleted the target cluster as I had to make some changes to persistent storage for the target cluster. When I recreated the target cluster with the same name, I ended up with the above mentioned issue.

Just to experiment, I also tried creating the target cluster with a different name and now everything worked fine.

So to summarize, when I recreate the target cluster with the same old name, though the topics are created, the data is not mirrored. Any suggestions or possible fixes ?

I don't have anything much crazy going on from the logs. Just the below one I see but not sure how does this affect.

MirrroMaker Logs:
2020-07-24 10:34:39,251 ERROR Plugin class loader for connector: org.apache.kafka.connect.mirror.MirrorCheckpointConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@57459491 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [pool-3-thread-2]
2020-07-24 10:34:37,060 ERROR Plugin class loader for connector: 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@57459491 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [pool-3-thread-1]

Thanks,
Eazhilan

[scholzj]

The error is just some Kafka bug which should not affect any functionality.

TBH, I'm not sure what is the expected behaviour here since by deleting the target cluster with lot of the data. Which messages are you missing? The old messages from before you deleted the target cluster? Or the new messages received on the source cluster after deleting and recreating the target cluster?

[eazhilan-nagarajan]

Hey, thanks for replying.

I'm sorry for confusing.

Target cluster name (originally): my-target-cluster-1

I deleted the target cluster, changed the storage type and recreated the target cluster with the same name (my-target-cluster-1). I expect the data in the target cluster to be lost as I've also deleted the storage but the issue is when I recreate the target cluster with the same name as the original one (my-target-cluster-1), the data is not getting mirrored to the target cluster though the topics are getting created.

Just for experiment, I changed the target cluster name (my-target-cluster-2) and now the data is getting mirrored.

I know it sounds silly but I have tried these out multiple times and every time it's the same behavior. I'm also sure of the configs as I got consistent behavior every time.

The below table gives a view of the topics from the target cluster. If you see my-source-cluster.topic-1 topic, no data is mirrored into it though there is data being populated to it's counterpart in the source cluster. Also the other MM related topics are getting filled with data.

Topic Name | OffsetID
-- | --
my-source-cluster.topic-1 | 0
my-source-cluster.connect-cluster-status | 0
my-source-cluster.connect-cluster-offsets | 0
my-source-cluster.connect-cluster-configs | 0
my-source-cluster.checkpoints.internal | 944
mirrormaker2-cluster-status | 17
mirrormaker2-cluster-offsets | 1309

Note sure if this is good information for you to help me :)

[scholzj]

the data is not getting mirrored to the target cluster though the topics are getting created.

Does the cluster have enabled Topic Operator? If yes, when deleting the target cluster, did you also delete the KafkaTopic resources? Because if you use Topic Operator and if you don't delete the KafkaTopics, the first one to recreate the topics will be the Topic Operator. So the topics might not have been created by the Mirror Maker.

But TBH I have no idea what the problem here might be.

[eazhilan-nagarajan]

the data is not getting mirrored to the target cluster though the topics are getting created.

Does the cluster have enabled Topic Operator? If yes, when deleting the target cluster, did you also delete the KafkaTopic resources? Because if you use Topic Operator and if you don't delete the KafkaTopics, the first one to recreate the topics will be the Topic Operator. So the topics might not have been created by the Mirror Maker.

But TBH I have no idea what the problem here might be.

Yes, the Topic Operator is enabled in this cluster but initially a few times I wasn't paying attention to the KafkaTopics and the topic operator created the topics as soon I spun up the cluster. In later attempts I deleted all the KafkaTopics before I spun up the cluster so then immediately after the cluster spun up no topics were created rather they were created only after I started the MM.

Let me give this one more try with a clear head and come back.

I really appreciate that you try to help. Many thanks!

[scholzj]

@ajborley Any idea what could be the problem here?

[eazhilan-nagarajan]

@scholzj, I tried creating the target Kafka cluster with the below listed names to see if it's working.

Target Cluster Name | Are Mirror Topics Created | Is Data Mirrored
-- | -- | --
my-target-cluster-1 | Yes | No
my-target-cluster-1-0 | Yes | No
my-target-cluster-4 | Yes | Yes

I had to use '4' as I have already used '2' and '3' in the names. To me, it seems like the cluster name pattern is matched which stops mirroring the data but I'm really not sure where would it be matched from. May be from some topic from the source cluster as that's the only thing I didn't delete.

:confused: :worried:

[ajborley]

@eazhilan-nagarajan

This may be due to data in the mm2-offset-syncs.my-target-cluster-1.internal topic on your source cluster that is created and used by the MM2 MirrorSourceConnector. That topic is used to store the offset position that has been reached by the consumer in the MirrorSourceConnector for each mirrored topic-partition. It also stores the mapping of the offset position on the source cluster to the equivalent offset position on the target cluster.

If you did not delete this topic on the source cluster when you recreated the target cluster, the MM2 connectors will start up, read the mm2-offset-syncs.my-target-cluster-1.internal topic and think that you have mirrored records up to the stored offset for each mirrored topic-partition (topic creation is handled separately to record mirroring).

If there is no new data coming into the source topic, then no data will get mirrored, as the MM2 MirrorSourceConnector thinks it has already mirrored the data. If there is new data coming into the topic, the new data should be mirrored, but the historic data will not.

When you created my-target-cluster-4, a new empty mm2-offset-syncs.my-target-cluster-4.internal topic was created on the source cluster, so all the data was mirrored.

I don't yet understand why data wasn't mirrored when you created my-target-cluster-1-0, but it may be worth removing the spec.mirrors from your KafkaMirrorMaker2 CR to stop the connectors, deleting the mm2-offset-syncs.my-target-cluster-1-0.internal topic on the source cluster, and then add the spec.mirrors back again, so that the connectors start up again. This should create a new empty mm2-offset-syncs.my-target-cluster-1-0.internal topic on the source cluster, which will mean all the historic data on the source topics should be mirrored by the MM2 MirrorSourceConnector.

Another option is to simply rename the target cluster alias in your KafkaMirrorMaker2 CR spec.clusters list (and copy the new alias name to spec.connectCluster) - this will give MM2 a new alias name for the target cluster, so a new empty mm2-offset-syncs.<target-cluster-alias>.internal topic will be created on the source cluster,

[eazhilan-nagarajan]

Hi @ajborley ,

Thanks for the detailed explanation of how the internals work. I think your suggestion of "removing the spec.mirrors, delete the
mm2-offset-syncs.my-target-cluster-1.internal topic and add the spec.mirrors again" trick worked.

Now my target cluster runs with the same old name (my-target-cluster-1) and the KMM2 is able to mirror the data but TBH, I'm not sure if it's working just because of removing and adding the spec.mirrors alone. Why I think so is, even in my previous attempts I have deleted the mm2-offset-syncs.my-target-cluster-1.internal topic but never touched the spec.mirrors.

Seems like I got an answer this time as well :relaxed:

Thank you @scholzj and @ajborley for the time and help.