Technical Tip: How to remove a mounted disk from a ClickHouse server

To delete a mounted disk in ClickHouse if it is no longer usable, follow these steps:

Mounted disk

Step 1: Create a backup of the fstab file and remove the corresponding entry from /etc/fstab:

cat /etc/fstab | grep data-clickhouse  (Verify Mount point)

sed -i.bak '/data-clickhouse\./d' /etc/fstab (Backup create for fatab file)

sed -i.bak '/data-clickhouse/d' /etc/fstab  (Remove data-clickhouse line from file)

cat /etc/fstab | grep /data-clickhouse 

Step 2: Unmount the disk from the server.

# umount /data-clickhouse-hot-1 <----- Mount point can get using df -h command.

Step 3: Remove the ClickHouse directory after unmounting from the filesystem. 

rm -rf /data-clickhouse-hot-1 <----- This could be warm or hot, depending on whether it needs to be replaced.

Step 4: Log in to FortiSIEM and remove the node from the GUI by navigating to Admin -> License -> Nodes and remove the Hot Tier as shown in the screenshot below.

Note:

This operation may lead to some warning in /opt/clickhouse/log/clickhouse-server.err.log log file:

<Warning> fsiem.events_replicated (ReplicatedMergeTreePartCheckThread): Found parts with the same min block and with the same max block as the missing part 18250-20250205_371_375_1 on replica 1. Hoping that it will eventually appear as a result of a merge.
<Error> fsiem.events_replicated (ReplicatedMergeTreePartCheckThread): No replica has part covering 18250-20250205_353_378_2 and a merge is impossible: we didn't find a smaller part with the same max block.
<Warning> fsiem.events_replicated (a5a85f1a-6ebf-4cf1-b82b-686f928798cc): Cannot commit empty part 18250-20250205_353_378_2 with error DB::Exception: Part 18250-20220620_353_378_2 (state Outdated) already exists, but it will be deleted soon

These warnings are not affecting the system but if this is frequent run the below command to stop these:

clickhouse-client --query "SELECT replica_path || '/queue/' || node_name FROM system.replication_queue JOIN system.replicas USING (database, table) WHERE last_exception LIKE '%No active replica has part%'" | while read i; do /opt/zookeeper/bin/zkCli.sh deleteall $i; done

clickhouse-client --query "SYSTEM RESTART REPLICAS"