Skip to main content
CKD
CKD
New Member
October 14, 2024
Question

Storage Policy: ClickHouse lowest storage tier free less than 0.20

  • October 14, 2024
  • 1 reply
  • 1280 views

Hello, we got below warning in our FortiSIEM, it's because our disk usage is currently 81% in Admin > Health > Supervisor > Disk

FSM_Error.jpeg

Furthermore, we continiously received below correlation which later I defined exception to 8%
Correlation.jpeg
But even I gave the exception + cleared error, error keeps coming (correlation stopped being triggered)
red_error.jpeg

we have hot and warm disks in our environment
I want to find out how to prevent this error from being triggered. We use clickhouse as visible in pictures.
so, once disk reaches 90% clickhouse space-based retention moves logs from hot disk to warm disk
(if no other disk available they are purged)

phoenix_config.jpeg
/opt/phoenix/config/phoenix_config.txt

I'm considering to change the config file above from 20 to 8 to prevent alarm from being triggered (8% because Just in case if space-based action retention doesn't move logs and then our hot disk reaches to 92% It would be better for correlation to get triggered)

Do you have any suggestion such as:
- How to prevent this alarm from being triggered
- Should I change this config 
- How to move logs manually from hot disk to warm disk

or anything else which is out of the box

Best Regards
Ceyhun Kıvanç Demir

    1 reply

    adem_netsys
    adem_netsys
    Explorer III
    October 25, 2024

    Hi @CKD,

     

    I'm in the same situation, have you done anything about it?

     

    Thanks

     

    CKD
    CKDAuthor
    New Member
    October 25, 2024

    As I have said, one way comes to my mind is modifying the below file
    #vi /opt/phoenix/config/phoenix_config.txt

    then between lines, find section
    "online_low_space_warning_threshold_GB=20"
    You may try to change 20 to 8 for example
    (which means alarm will be triggered only if 8% space left in hot-disk, however when 10% disk space is left, clickhouse should already move logs to warm disk, Therefore, theoretically this alarm should never showup again as long as FortiSIEM action comes in play at 10% disk space left and moves logs to warm disk which will create 20% empty space in hot-disk (according to official fortisiem doc atleast).

    If this alarm keeps coming it means ClickHouse couldn't start action of moving from hot to warm at 10% and now empty space is less than 8% than there might be something wrong with action taking) 

    once you saved the change on config file, you should
    #killall -9 phDataManager
    #killall -9 phDataPurger 

    which will restart these services. Then error should stop coming I guess....

    I couldn't try it on real FortiSIEM, so I'm setting up my test FortiSIEM first,
    I will try it there. If I ll get any news I ll inform u

    I'm just a community member. so, keep in mind that, always take snapshot and backup before you do any change.  Things I say might not be 100% accurate,
    I wouldn't want to be guilty of a worse situation, so responsibility is yours mate

    ty
    Ceyhun Kıvanç Demir

    Terms & ConditionsAccessibility statement