|
FortiSIEM may trigger one or both of the following alerts in environments using ClickHouse as the event database:
-
PH_RULE_CLICKHOUSE_INTER_SHARD_STORAGE_GAP_HIGH indicates that storage usage between ClickHouse shards has exceeded the configured threshold.
-
PH_RULE_CLICKHOUSE_INTRA_SHARD_STORAGE_GAP_HIGH indicates that replicas within the same shard have become unbalanced in terms of stored data.
These alerts are raised when there is a significant difference in disk usage between shards or replicas, which can result in degraded query performance, uneven resource utilization, and increased latency in data processing.
Cause:
A storage gap occurs when one shard or replica stores significantly more data than the others.
Common reasons include:
-
Uneven log ingestion.
-
Delayed merges or replication lag.
-
Node performance or disk differences.
-
Temporary imbalance after maintenance or restart.
Solution:
To resolve the High Inter-shard Storage gap/High Intra-shard Storage Gap between ClickHouse Workers alert, follow these steps:
Run the following command on the node that has the out-of-order issue:
# /opt/phoenix/bin/clickhouse-rebalance-partitions
This redistributes partitions evenly across shards and replicas. After rebalancing has been performed, monitor the alerts for a few hours to verify alerts no longer trigger.
|
