Troubleshooting Tip: How to troubleshoot and fix flash memory exhaustion issues on FGT30E and FGT50E clusters

[ftrapani]

Description	This article describes how to identify and fix flash memory exhaustion issues on 30D, 30E and 50E clusters.
Scope	FortiOS 6.2.6 to 6.2.10.
Solution	Due to the increasing number and size of FortiGuard Databases, some low-end devices, namely FGT30D, FGT30D rugged, FGT50E, and FGT51E, could run into flash memory exhaustion. This could lead to several unwanted issues, namely: - Partial or total configuration loss in case of a power outage or hard reboot. - FortiGuard databases or IPS engine upgrade failures. - Firmware upgrade failures. To address this issue, since FortiOS 6.2.11 GA, some database size adjustments have been made, to reduce flash memory occupancy and avoid unexpected scenarios. Thus, to fix this issue, it is strongly recommended to upgrade the cluster to 6.2.12 GA, keeping in mind that 6.2.11 is vulnerable to SSL VPN Buffer-Heap vulnerability (see https://www.fortiguard.com/psirt/FG-IR-22-398). Before performing the upgrade, freeing a portion of flash memory is mandatory to complete the upgrade successfully. NOTE: Since FortiGuard databases are synced between primary and secondary nodes, it is very important to follow the flash cleanup and upgrade operations very carefully. *******Issue Identification******* If the cluster is experiencing one of the issues reported, please check the output of these CLI commands: # fnsysctl df -h Filesystem                 Size       Used  Available Use% Mounted on rootfs                  1011.9M      68.6M     943.3M   7% / tmpfs                   1011.9M      68.6M     943.3M   7% / none                       1.6G     118.2M       1.4G   7% /tmp none                       1.6G     468.0K       1.6G   0% /dev/shm none                       1.6G      23.0M       1.5G   1% /dev/cmdb /dev/mtd6                 18.0M      17.7M     244.0K  99% /data -----> flash partition. /dev/mtd7                 30.0M      15.9M      14.0M  53% /data2 Or, alternatively, # diagnose sys flash list Partition  Image                                     TotalSize(KB)  Used(KB)  Use%  Active 1          FGT50E-6.00-FW-build0272-190716                   18432     14668   80%  No    2          FGT50E-6.02-FW-build1142-200819                   18432     18188   99%  Yes   3          ETDB-1.00000                                      30720     16372   53%  No If the Usage value is in the range of 98%-99%, flash memory exhaustion is in place. Check also the output of: # fnsysctl ls -la /data/etc <output_omitted> ----rw-rw-    1 0        0       Tue Feb  8 01:54:34 2022          2956003 geoip_db.gz This command will be useful for the upgrade procedure ******Upgrade Procedure to fix flash memory issue******* 1) On Primary FortiGate, set an HA priority higher than the one of the Secondary Unit (default is 100), and only then enable HA override on the cluster. - Primary (example): # conf system ha     set priority 200 end - Secondary (example): # conf system ha     set priority 100 end After priority configuration, on both nodes (primary first): # conf system ha     set override enable end 2) Temporarily disable the scheduled FortiGuard updates from System -> FortiGuard. 3) On the primary device, run the command: diagnose geoip delete-geoip-db. This command will delete the FortiGuard GeoIP Database (geoip_db.gz), freeing about 15% of memory without traffic impact. Note: The device will be forced to reboot but, due to overriding, will be elected as primary again. This will prevent the secondary to become primary and resynchronize the GeoIP Database. After the reboot, check the output of fnsysctl ls -la /data/etc  to verify that the entry geoip_db.gz has been deleted. Then, check the output of diagnose sys flash list, the used space for the active partition should have decreased to 85-86%. Check also the HA sync status with get sys ha status: the cluster will be displayed in-sync, even if the secondary has still the GeoIP Database stored in memory: HA Health Status: OK Model: FortiGate-50E Mode: HA A-P Group: 146 Debug: 0 Cluster Uptime: 0 days 21:42:53 Cluster state change time: 2022-03-12 11:40:51 <output_omitted> FGT50E3U15002795 (updated 5 seconds ago): in-sync FFGT50E3U15000151 (updated 4 seconds ago): in-sync 4) If all the previous checks were successful, issue diagnose geoip delete-geoip-db on the secondary unit and, after reboot, perform all the previous memory and HA sync checks to ensure that also the secondary unit has enough memory to perform the upgrade and that the cluster is fully operational. 5) Perform the uninterruptible upgrade following the recommended upgrade path. Remember to check the free space before each upgrade step. 6) Once the upgrade is completed, re-enable the Scheduled FortiGuard updates and run the command execute update-now on Primary Unit to resync all the Databases. 7) At the end of the procedure, the used memory shown by diagnose sys flash list will be in the range of 89% - 92%. This is an acceptable value that will prevent the cluster from configuration loss or upgrade failures.