Troubleshooting Tip: Resolving 100% Disk Utilization in the /dev/mapper/rl-root Directory for EventDB on NFS Storage Deployment

If the NFS server loses connection, FortiSIEM will temporarily use the local root directory (/) to store EventDB data. This quickly fills up the root partition, leading to services going down. Follow the steps below to troubleshoot and resolve this issue:

To resolve this issue, follow these steps:

     1. Identify The Cause.

First, check the current mounted directories, especially /data, using the df command:

df -h

Example Output (Worker with lost NFS connection and 100% root directory):

Filesystem           Size  Used Avail Use% Mounted on
devtmpfs              12G     0   12G   0% /dev
tmpfs                 12G     0   12G   0% /dev/shm
tmpfs                 12G   33M   12G   1% /run
tmpfs                 12G     0   12G   0% /sys/fs/cgroup
/dev/mapper/rl-root   22G   22G   20K 100% /
/dev/sdb2             69G   11G   59G  16% /opt
/dev/sda1           1014M  818M  197M  81% /boot
tmpfs                2.4G     0  2.4G   0% /run/user/500
tmpfs                2.4G     0  2.4G   0% /run/user/0

In this case, the root partition (/dev/mapper/rl-root) has reached 100% utilization, causing the services to go down.

     2. Identify the Largest Directories in the Root Partition.

Run the following command to determine which files occupy the most space in the root directory:

sudo du --max-depth=4 -xh /data | sort -hr | head -n 20

Example Output (truncated):

11G/data/eventdb/CUSTOMER_1/default
11G/data/eventdb/CUSTOMER_1
11G/data/eventdb
11G/data
5.1G/data/eventdb/CUSTOMER_1/default/19943
4.9G/data/eventdb/CUSTOMER_1/default/19942
…

Although /data is present in the ‘du’ output, it is not properly mounted as NFS, as indicated by the absence of /data in the ‘df -h’ output.

     3. Bind-Mount the Root Directory to a Temporary Mount Point.

To safely move data, bind-mount the root directory (/) to /mnt:

mount -o bind / /mnt

     4. Transfer Files to the NFS /data Directory.

Move the data from the root partition to the NFS storage using the 'rsync' command:

rsync -avzh --remove-source-files --progress /mnt/data/eventdb/CUSTOMER_1/ /data/eventdb/CUSTOMER_1

     5. Verify Disk Space Utilization.

Check the disk space again to ensure that the root partition’s utilization has decreased:

df -h

     6. Unmount the Temporary Mount Point.

If the disk space usage is no longer at 100%, unmount the temporary /mnt directory:

umount /mnt/

     7. Monitor the Status of the Services.

Ensure that the services have resumed by checking the FortiSIEM status:

phstatus​