Troubleshooting Tip: How to resolve Clickhouse tables in read-only mode due to Part xxxx intersects previous part yyyy in FortiSIEM

Description

This article describes how to resolve Clickhouse tables in read-only mode due to Part xxxx intersects Part yyyy in FortiSIEM.

Scope

FortiSIEM.

Solution

The following errors are observed in /opt/clickhouse/log/clickhouse-server.log:

tail -f /opt/clickhouse/log/clickhouse-server.log

Code: 49. DB::Exception: Part 20250718_13529_13535_1 intersects previous part 20250718_13500_13533_6. It is a bug or a result of manual intervention in the ZooKeeper data. (LOGICAL_ERROR) (version 22.6.1.1985 (official build))
zookeeper_exception:

In the GUI, navigate to Admin -> Health -> Cloud Health to see which tables are in read_only mode by hovering over the Critical status on a given worker.

The contents of the screenshot above indicate that fsiem.events_replicated and mv_events tables are in read-only mode.

The following approach can be followed from the respective Worker Data Node to resolve the errors.

clickhouse-client -q "select * from system.replicas"

Note down the Replica number which will be used in the next steps.

To recover the mv_t_events table from read-only mode, enter the following commands:

clickhouse-client -q "DETACH TABLE fsiem.mv_t_events"

clickhouse-client -q "SYSTEM DROP REPLICA '1' FROM ZKPATH '/clickhouse/tables/1/fsiem.mv_events'"

clickhouse-client -q "ATTACH TABLE fsiem.mv_t_events"

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.mv_t_events"

clickhouse-client -q "SYSTEM SYNC REPLICA fsiem.mv_t_events"

To recover events_replicated table from read only mode, enter the following commands:

clickhouse-client -q "DETACH TABLE fsiem.events_replicated"

clickhouse-client -q "SYSTEM DROP REPLICA '1' FROM ZKPATH '/clickhouse/tables/1/fsiem.events'"

clickhouse-client -q "ATTACH TABLE fsiem.events_replicated"

clickhouse-client -q "SYSTEM RESTORE REPLICA fsiem.events_replicated"

clickhouse-client -q "SYSTEM SYNC REPLICA fsiem.events_replicated"

Use the following command to check if tables are no longer in read-only mode:

clickhouse-client -q "select * from system.replicas"

If the tables are still in read-only mode, check for any DR nodes (Supervisor/Worker) added as a Keeper/Data Node in the Clickhouse cluster.

If multiple Keeper/Data nodes are added in the Clickhouse Cluster, try removing the respective Keeper/Data node from the cluster in Admin -> Settings -> Database -> Clickhouse config -> Remove DR nodes -> Test & Deploy.

Then, check again if the tables are removed from read-only mode.