Understanding /dev/mapper/rl-root storage FortiSIEM

[gwaihir]

Hello guys, please guide me to understand this little thing.

This is a supervisor all in one deployment with Click house DB (FSM 7.1.X running on Rocky Linux)

As you can see, my /sda partition has 75G, (at first it was 25 GB)

The /dev/mapper/rl-root inside /dev/sda is eating quickly the storage that I assign to it. Why is this happening? is supposed that logs are forwarded to Click house DB in /dev/sde partition tand this should be the more used storage, also /opt is running out of storage. 

This is the second time that I resize the /dev/mapper/rl-root partition. 

Thank you!

[AEK]

Hi @gwaihir 

You can use du -sh /* command to try find what is consuming your storage.

[gwaihir]

Hi @AEK thank you for your reply.

[root@XXX ~]# du -sh /opt/* 2>/dev/null |  sort -n
0       /opt/db-backup
0       /opt/glassfish
0       /opt/Java
0       /opt/openjdk
0       /opt/vmware
1.1G    /opt/glassfish5
1.2G    /opt/jsreport
13M     /opt/zookeeper.tar.gz
31M     /opt/rpm
40M     /opt/fortiinsight-ai
40M     /opt/zookeeper
48M     /opt/exporter
63M     /opt/archive
64G     /opt/phoenix
89M     /opt/selenium
399M    /opt/clickhouse
417M    /opt/node-rest-service
727M    /opt/charting

[root@XXX ~]# du -sh /* 2>/dev/null |  sort -n
0       /bin
0       /lib
0       /lib64
0       /media
0       /mnt
0       /pbin
0       /proc
0       /query
0       /querywkr
0       /sbin
0       /srv
0       /sys
2.4G    /run
3.3G    /fsmopt.tar.gz
4.0K    /platform-info
4.0K    /svn.tar.gz
6.1G    /usr
8.7M    /root
12K     /redis.log
24K     /home
27M     /etc
32G     /var
53G     /data-clickhouse-hot-1
68G     /opt
165M    /data
253M    /tmp
332K    /svn
432K    /dev
648M    /boot
726M    /cmdb

But, really I want to understand why this is happening, if /dev/mapper/rl-root is linked to /opt and eat storage quickly, seems that I'm missing something abot initial sizing.

UPDATE: (inside /opt/phoenix/cache), can this be safely removed?

587M    /opt/phoenix/cache/_MEIfjDpil
587M    /opt/phoenix/cache/_MEIFsVx3B
587M    /opt/phoenix/cache/_MEIg0X2zL
587M    /opt/phoenix/cache/_MEIgHX101
587M    /opt/phoenix/cache/_MEIgjsjuq
587M    /opt/phoenix/cache/_MEIIuHILc
587M    /opt/phoenix/cache/_MEIj6QqhT
587M    /opt/phoenix/cache/_MEIJRt7Wd
587M    /opt/phoenix/cache/_MEIjxxg3D
587M    /opt/phoenix/cache/_MEILNdpfo
587M    /opt/phoenix/cache/_MEIMDCga2
587M    /opt/phoenix/cache/_MEIMFFd1S
587M    /opt/phoenix/cache/_MEImhGWaQ
587M    /opt/phoenix/cache/_MEIn4b2fh
587M    /opt/phoenix/cache/_MEINe9eRD
587M    /opt/phoenix/cache/_MEInpqHtW
587M    /opt/phoenix/cache/_MEIoM0HCw
587M    /opt/phoenix/cache/_MEIp2q0dp
....
a lot 
...

Thank you!

[Richie_C]

Hi @gwaihir 

This does not look like normal behavior. My recommendation would be to raise a TAC case for this. The following link provides some useful information about the output required by TAC:

https://community.fortinet.com/t5/FortiSIEM/Technical-Tip-How-to-retrieve-logs-from-FortiSIEM-VA-and...

I hope it helps.

[gwaihir]

Hello @Richie_C thank you for your reply

I deleted all those files from ../cache/ and everything is going well.

Regards.

[adem_netsys]

Hi @gwaihir,

I have the same problem, I get a /archive fullness warning in super. I can add a disk but it might happen again, I have a new environment. How did you clear the cache?

[gwaihir]

Hi @adem_netsys I deleted all files inside /opt/phoenix/cache with the prefix _M*  and also found that some kernel crash were saved on /crash directory /var 

I think this problem was solved because there are no more space disk alerts.

Regards.