This ARTICLE describes how to move all data in ClickHouse from one node to another node.
Scope
FortiSIEM.
Solution
In ClickHouse FortiSIEM management, it may be required to move data from one node to another to remove data management on that node or before deleting the node from the cluster.
The next steps are based on the Rebalancing Shards process, where ClickHouse Partitions are moved one by one, but embedded in the script to automate this data migration:
Context of use:
Move data from one shard configured with only one replica to another shard.
Destination Shard has enough space to store data that is in the source shard.
Move the Super data node to the worker data node.
Move the Worker data node to another worker data node.
Disactivate 'Data' on the source node: From the GUI at Admin -> Settings -> ClickHouse Cluster, identify the source node from where data needs to be moved, untick the 'Data' column for that node, and note the Shard Number and click 'Test' and 'Deploy'.
Install the script: Copy the attached move_all_partitions.zip file and transfer it using scp on the source data node in /tmp directory, then from the node CLI as root:
chmod +x /tmp/move_all_partitions.sh
cd /tmp
Prepare the command with the source IP, the Shard number previously noted, and the destination IP.
Run the script to move data: From source node CLI as root:
nohup sh move_all_partitions.sh source_node_ip source_shard_number destination_node_ip | tee -a /tmp/move_result.txt &
Check the output:
tail -f /tmp/move_result.txt
This can take some time to transfer the amount of data.
Remove the node from the ClickHouse Cluster: under Admin -> Settings -> ClickHouse Cluster, select the '-' where the shard can be removed and select 'Test' and 'Deploy'.
