ClickHouse hot tier full not completed data send to warm tier

[wfgaitan]

Hi Guys.

Hope yo can help me.  I have a fortisiem deployment with one supervisor, one worker and one collector, with a hot tier disk of 200GB and a warm tier disk of 300GB. Both disks are attached to the supervisor and worker respectively.
The clickhouse deployment:

Two node clickhouse keeper cluster: 1 supervisor, 1 worker

Two node clickhouse cluster: 1 shard > Replica 1 : Supervisor (Data and Query)

                                                               Replica 2 : Worker (Data)

Yesterday hot tier disk on worker got full (100%) and no more events have been stored and the queue is almost 1100 (1.1GB).
On the supervisor hot tier disk is 82% (163GB). In it´s Warm Tier is just 47GB.

On the worker hot tier disk is 100%(200GB) .In it´s warm Tier is just 1% (2.2GB)

Why did the event moving did not work properly? is there any mean to trigger that moving manually?

Thanks.

FortiSIEM 

[jamesmid]

---

@wfgaitan wrote:

Hi Guys.

Hope yo can help me.  I have a fortisiem deployment with one supervisor, one worker and one collector, with a hot tier disk of 200GB and a warm tier disk of 300GB. Both disks are attached to the supervisor and worker respectively.
The clickhouse deployment:

Two node clickhouse keeper cluster: 1 supervisor, 1 worker
Two node clickhouse cluster: 1 shard > Replica 1 : Supervisor (Data and Query)

                                                               Replica 2 : Worker (Data)

Yesterday hot tier disk on worker got full (100%) and no more events have been stored and the queue is almost 1100 (1.1GB).
On the supervisor hot tier disk is 82% (163GB). In it´s Warm Tier is just 47GB.

On the worker hot tier disk is 100%(200GB) .In it´s warm Tier is just 1% (2.2GB)
Why did the event moving did not work properly? is there any mean to trigger that moving manually?

Thanks.

FortiSIEM sherman oaks locksmith

---

The event moving likely didn't occur because ClickHouse's data management settings or thresholds weren't properly configured to trigger automatic movement from the hot tier to the warm tier when the hot tier reached capacity.

To manually trigger the data movement, you can use the ALTER TABLE ... MOVE PARTITION TO DISK command to move specific partitions from the hot tier (likely stored on fast storage like SSDs) to the warm tier (typically larger, slower disks).

[adem_netsys]

Hi guys @wfgaitan @jamesmid 

Have you been able to resolve this, Super disk avaliable is decreasing but no increase in warm. Should we move it manually, is it right to do this, have you implemented a new solution?