Troubleshooting Tip: fgfmd 100% utilization and inode usage 100% in FortiManager 7.4.7

For every object created in FortiManager (like address objects, policies, etc.) is assigned an inode as a unique ID for internal reference. This helps FortiManager to track object dependencies and relationships. The inode is not something that is usually seen in the GUI, but it's more relevant to API, database and CLI queries.

The problem in FortiManager 7.4.7 appears that due to a system issue, when managing large number of devices, system may exaust inodes to 100%. The issue is related only to version 7.4.7. For example:

FortiManager-1 # diag system print df -i
Filesystem Inodes Used Available Use% Mounted on
rootfs 6159801 28040 6131761 0% /
none 6159801 134 6159667 0% /dev
none 6170158 6 6170152 0% /dev/shm
none 6170158 56 6170102 0% /tmp
/dev/sda1 65536 38 65498 0% /data
/dev/mdvg/mdlv 2048000 2048000 0 100% /var
/dev/mdvg/mdlv 2048000 2048000 0 100% /drive0
/dev/mdvg/mdlv 2048000 2048000 0 100% /Storage
/dev/loop0 2560 11 2549 0% /var/dm/tcl-root

FortiManager-1 # get system performance

--- partial output ---

Hard Disk:
Total: 523,532,296 KB
Used: 282,042,420 KB 53.9%
Inode-Total: 2,048,000
Inode-Used: 2,048,000 100.0%
IOStat: tps r_tps w_tps r_kB/s w_kB/s queue wait_ms svc_ms %util sampling_sec
         0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.02

As a result when inodes are exhausted the FortiManager will stop responding and no installs or scripts can be run properly. 

The temporary workaround that may be applied in this case will be to reboot the device, in order to clear the system files that are keeping the inode usage high.

To diagnose this problem, use the following commands: 

diagnose system print df -i

get system performance