Technical Tip: Handling 'High Inter-shard Storage Gap between ClickHouse Workers' and 'High Intra-shard Storage Gap between ClickHouse Workers' Alerts in FortiSIEM

FortiSIEM may trigger one or both of the following alerts in environments using ClickHouse as the event database:

• PH_RULE_CLICKHOUSE_INTER_SHARD_STORAGE_GAP_HIGH indicates that storage usage between ClickHouse shards has exceeded the configured threshold.

• PH_RULE_CLICKHOUSE_INTRA_SHARD_STORAGE_GAP_HIGH indicates that replicas within the same shard have become unbalanced in terms of stored data.

These alerts are raised when there is a significant difference in disk usage between shards or replicas, which can result in degraded query performance, uneven resource utilization, and increased latency in data processing.

Cause:

A storage gap occurs when one shard or replica stores significantly more data than the others.
Common reasons include:

• Uneven log ingestion.

• Delayed merges or replication lag.

• Node performance or disk differences.

• Temporary imbalance after maintenance or restart.

Solution:

To resolve the High Inter-shard Storage gap/High Intra-shard Storage Gap between ClickHouse Workers alert, follow these steps:

Run the following command on the node that has the out-of-order issue:

# /opt/phoenix/bin/clickhouse-rebalance-partitions

This redistributes partitions evenly across shards and replicas. After rebalancing has been performed, monitor the alerts for a few hours to verify alerts no longer trigger.