Clickhouse: No data in replicated table

[simPod]

I have two replicas. Data are inserted into table in replica A. Table counterpart in replica B is empty though. Data are not replicated between replicas. I tried recreating table in replica B but it did not trigger the sync.

Replication for other tables is working just fine.

What can I do to debug the issue? show create table gives same result for both replicas.

ClickHouse version 20.6

[den-crane]

check (at both nodes) : select * from system.replication_queue

wrong ZK path? different shards?

[simPod]

I don't have multiple shards.

replication queue does not contain any records for the target table

path is the same (reported by show create table)

🤔

[den-crane]

Show grep tablename clickhouseserver.log

[simPod]

I tried grepping cat /var/log/clickhouse-server/clickhouse-server.log in both replicas but for this particular table contains no logs (it does have some warnings for larger tables though).

[den-crane]

It could be anything. How did you copy/pasted / attached zk path when you created a table?

For example these two strings are different.
Latin: /ccccc/
Сyrrylic: /ссссс/

Check zookeeper select * from system.zookeeper where path = '/..zk_path/replicas
Check select * from system.replicas where table = ...

[simPod]

select * from system.zookeeper where path = '/..zk_path/replicas
Gives result on both replicas.

But on the B, there's replicas last added replica: A (version: 1)
For fully replicated tables it's replicas last added replica: B (version: 4)

system.replicas says the total_replicas is 1. Seems like those tables are disconnected on both replicas. Is there a way to get in in sync somehow?

How did you copy/pasted / attached zk path when you created a table?

I did it a long time ago but It was simple CREATE TABLE ON CLUSTER I believe

[den-crane]

Check at both servers that zhash is identical

select replica_name, total_replicas, active_replicas, zookeeper_path, 
           cityHash64(zookeeper_path) zhash, hex(zookeeper_path) 
from system.replicas
where table = '....'

[simPod]

Zookeeper paths differ in a small typo :/ Thank you very much for helping me debug this. I have setup a new monitoring query to catch this in the future.