Storage Policy: ClickHouse lowest storage tier free less than 0.20

[CKD]

Hello, we got below warning in our FortiSIEM, it's because our disk usage is currently 81% in Admin > Health > Supervisor > Disk



Furthermore, we continiously received below correlation which later I defined exception to 8%

But even I gave the exception + cleared error, error keeps coming (correlation stopped being triggered)


we have hot and warm disks in our environment
I want to find out how to prevent this error from being triggered. We use clickhouse as visible in pictures.
so, once disk reaches 90% clickhouse space-based retention moves logs from hot disk to warm disk
(if no other disk available they are purged)


/opt/phoenix/config/phoenix_config.txt

I'm considering to change the config file above from 20 to 8 to prevent alarm from being triggered (8% because Just in case if space-based action retention doesn't move logs and then our hot disk reaches to 92% It would be better for correlation to get triggered)

Do you have any suggestion such as:
- How to prevent this alarm from being triggered
- Should I change this config 
- How to move logs manually from hot disk to warm disk

or anything else which is out of the box

Best Regards
Ceyhun Kıvanç Demir

[adem_netsys]

Hi @CKD,

I'm in the same situation, have you done anything about it?

Thanks

[CKD]

As I have said, one way comes to my mind is modifying the below file
#vi /opt/phoenix/config/phoenix_config.txt

then between lines, find section
"online_low_space_warning_threshold_GB=20"
You may try to change 20 to 8 for example
(which means alarm will be triggered only if 8% space left in hot-disk, however when 10% disk space is left, clickhouse should already move logs to warm disk, Therefore, theoretically this alarm should never showup again as long as FortiSIEM action comes in play at 10% disk space left and moves logs to warm disk which will create 20% empty space in hot-disk (according to official fortisiem doc atleast).

If this alarm keeps coming it means ClickHouse couldn't start action of moving from hot to warm at 10% and now empty space is less than 8% than there might be something wrong with action taking) 

once you saved the change on config file, you should
#killall -9 phDataManage
#killall -9 phDataPurger 

which will restart these services. Then error should stop coming I guess....

I couldn't try it on real FortiSIEM, so I'm setting up my test FortiSIEM first,
I will try it there. If I ll get any news I ll inform u

I'm just a community member. so, keep in mind that, always take snapshot and backup before you do any change.  Things I say might not be 100% accurate,
I wouldn't want to be guilty of a worse situation, so responsibility is yours mate

ty
Ceyhun Kıvanç Demir