Troubleshooting Tip: Removing stuck ClickHouse Keeper queue entries for NO_REPLICA_HAS_PART errors

Description

This article describes how to remove stuck ClickHouse Keeper queue entries when ClickHouse replication is blocked by NO_REPLICA_HAS_PART errors.

Scope

FortiSIEM.

Solution

Review the ClickHouse error logs and identify queue entries associated with NO_REPLICA_HAS_PART errors. Example:

Error MergeTreeBackgroundExecutor: Exception while executing background task {99f41910-0d5f-40f4-92d7-29dbf03e86e6::365-20260531_14602_14731_22}: Code: 234. DB::Exception: No active replica has part 365-20260531_14602_14731_22 or covering part (cannot execute queue-0000002455: MERGE_PARTS with virtual parts [365-20260531_14602_14731_22]). (NO_REPLICA_HAS_PART)
Error MergeTreeBackgroundExecutor: Exception while executing background task {2ff2b774-f282-4e73-9912-6dfb5c5bcbe5::18250-20260623_0_7696_1532}: Code: 234. DB::Exception: No active replica has part 18250-20260623_0_7696_1532 or covering part (cannot execute queue-0000001115: MERGE_PARTS with virtual parts [18250-20260623_0_7696_1532]).

Note down the queue numbers from the logs - 0000002455 and 0000001115.

1. Connect to ClickHouse Keeper.
   
   

/opt/zookeeper/bin/zkCli.sh -server localhost:2181

2. Verify the queue exists.
   
   

ls /clickhouse/tables/1/fsiem.events/replicas/1/queue

3. Delete the stuck queue entry.
   
   

delete /clickhouse/tables/1/fsiem.events/replicas/1/queue/queue-0000002450

4. Confirm removal.
   
   

ls /clickhouse/tables/1/fsiem.events/replicas/1/queue

Output would be blank, such as '[]'.

5. Repeat for other ClickHouse tables, check and remove the affected queue entries from tables such as:
   
   

fsiem.events
fsiem.mv_events
fsiem.summary

Use the queue IDs identified from the ClickHouse logs.