Clickhouse: I wonder how storage policy do on disk failure.

[achimbab]

My ClickHouse configurations are like below.

• Node1

  • two disks
  • /data1
  • /data2

• Node2

  • two disks
  • /data1
  • /data2

Data is replicated between Node1 and Node2 by ReplicatedMergeTree.
And each ReplicatedMergeTree uses the below storage_policy.
So ReplicatedMergeTree splits data into two disks; /data1 and /data2.

Settings

<storage_configuration>                                                          
    <disks>                                                                      
        <jbod1>                                                                  
            <path>./jbod1/</path>                                                
            <keep_free_space_bytes>1024</keep_free_space_bytes>                  
        </jbod1>                                                                 
        <jbod2>                                                                  
            <path>./jbod2/</path>                                                
            <keep_free_space_bytes>1024</keep_free_space_bytes>                  
        </jbod2>                                                                 
    </disks>                                                                     

    <policies>                                                                   
        <jbods>                                                                  
            <volumes>                                                            
                <main>                                                           
                    <disk>jbod1</disk>                                           
                    <disk>jbod2</disk>                                           
                </main>                                                          
            </volumes>                                                           
        </jbods>                                                                 
    </policies>                                                                  
</storage_configuration>                                                         

What happen If /data1 on Node1 is broken?
Does Node1 lose the data on /data1?
Or Does Node1 copy data from Node2?

Thank you in advance.

[alexey-milovidov]

If data1 is broken, you should replace the disk (create filesystem and mount at the same location) and restart clickhouse-server. ClickHouse will download missing parts from replica.

[alexey-milovidov]

You can also remove disk1 from volumes configuration and restart clickhouse-server. ClickHouse will download missing parts from replica and place them on disk2.

[alexey-milovidov]

BTW, different replicas can have different volumes and different placement of data parts (but the set of data parts as a whole will be synchronized).

[achimbab]

@alexey-milovidov
Thank you for your explanation.