ClickHouse Partitions Order: ClickHouse data partitions are not in order

A possible solution is in the FortiSIEM 7.4.0 Release Notes under Key Enhancements > ClickHouse Enhancements.
https://docs.fortinet.com/document/fortisiem/7.4.0/release-notes/46107/whats-new-in-7-4-0#Key
=======================

1. In ClickHouse, events in Hot, Warm, Cold and Archive partitions must maintain time order. This means Archives events must have earlier timestamp than Cold events, Cold events must have earlier timestamp than Warm events, etc. This condition may be violated when a new (Replica) Worker is added to a Shard, and there is another Worker in the shard with data. This situation may result in performance degradation and more importantly, newer data may be incorrectly archived or purged.

   In this release, FortiSIEM detects this situation and

   • A GUI system error (ClickHouse Partitions Order: ClickHouse data partitions are not in order) is generated.

   • A log: ClickHouse data partitions are not in order (PH_CLICKHOUSE_DATA_PARTITIONS_OUT_OF_ORDER) is generated.

   When this happens, you need to run this command on the node with out-of-order issue:

   /opt/phoenix/bin/clickhouse-rebalance-partitions

   The GUI system error (ClickHouse Partitions Order: ClickHouse data partitions are not in order) should go away.

=======================

Hi,

Your issue is that ClickHouse partitions are out of order. This means the data (Hot, Warm, Cold, Archive) isn’t aligned chronologically. To address this properly, we need to escalate through TAC. Please open a TAC ticket, and the team will investigate and remediate this for you.

Thank you, people for your support.

Hello @aebadi and all

Can this command perform a rebalance between shards even when no errors are encountered? For example, consider a scenario where one hot shard is at 85% and another is at 40%.