FortiGate 101 - Sustained High Memory Use

[ilikebbc]

Simple question.  Should I be concerned that one FortiGate 101F consistently runs at 60% - 75% memory used, while a like 101F with a significantly larger network and hardware load uses 25% - 35% of memory on a daily average.

Both 101s share the same OS version, internal hardware configuration, and basic firewall and SD-WAN configurations.  They differ in that the high-memory use 101 has three 4- port FortiSwitches serviced by FortiLink, roughly 14,000 entries in the routing table, and roughly 500 sessions.  The 101 using less memory hosts 10 48-port FortiSwitches, a 21,000 entry routing table, and roughly 4500 sessions.

Neither 101 exhibits anomalous behavior at the moment.  Still, I wonder if I should expect trouble in the future.

Thanks.

[sferoz · Accepted solution]

Since the specific unit is GEN-1 with 4GB mem, would recommend memory optimization based on the config, tweaks can be applied to reduce overall memory usage.

Some KB references for memory optimization:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Free-up-memory-to-avoid-conserve-mode/ta-p...

[sferoz]

Hi ilikebbc,

We can certainly look at the logs during high memory on FortiGate 101F when it's consistently running at 60% - 75% and investigate on the usage.

Kindly collect the below logs during high memory along with the config file and share it to sferoz@fortinet.com.
#fnsysctl date
#get hardware status
#get sys ha stat
#diagnose autoupdate versions
#get sys stat
#get sys per status (run this command 5 times in interval of 1 minute)
#diag sys top-mem 30 (run this command for 10 times)
#diagnose hardware sysinfo memory
#diagnose hardware sysinfo shm
#diagnose hardware sysinfo slab
#get system performance firewall statistic
#diag sys session stat
#diag sys top 1 50 (run this command for 60 seconds)
#diagnose debug crashlog read
#diagnose ips session status
#diagnose ips packet status
#diagnose ips memory status
#diagnose wad memory track
#diagnose wad memory report

#diag sys top-fd 20

#sh full firewall policy | grep -c "inspection-mode proxy"

Thanks,
Shameem

[ilikebbc]

Thanks, Shameem.  Lemme see what I can gather without exposing proprietary information.
Incidentally, should 60% sustained memory usage be a concern?  What are the FortiNet recommended thresholds?

[Yurisk]

Show us at least output of get hardware stat command - it will say how much RAM your 101F has, of the 100F models, there were 2 revisions - 1st one produced roughly until 2022, 2nd revision afterwards. The 1st revision had just 4 Gb of memory, which will make memory usage on 75% logical and possible even on idle. Also what is the FortiOS versions you use?

[ilikebbc]

Yurisk/Shameem, here are some details to mull.

XXXXXXXX # get hardware status
Model name: FortiGate-101F
ASIC version: SOC4
CPU: ARMv8
Number of CPUs: 8
RAM: 3614 MB
EMMC: 3662 MB(MLC) /dev/mmcblk0
Hard disk: 457862 MB /dev/sda
USB Flash: not available
Network Card chipset: FortiASIC NP6XLITE Adapter (rev.)
Hardware Revision: Rev1

XXXXXXXX # get system status
Version: FortiGate-101F v7.2.9,build1688,240813 (GA.M)
Security Level: 1
Firmware Signature: certified
Virus-DB: 1.00000(2018-04-09 18:07)
Extended DB: 1.00000(2018-04-09 18:07)
AV AI/ML Model: 0.00000(2001-01-01 00:00)
IPS-DB: 33.00012(2025-05-22 00:35)
IPS-ETDB: 0.00000(2001-01-01 00:00)
APP-DB: 33.00012(2025-05-22 00:34)
FMWP-DB: 25.00050(2025-05-21 14:32)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
IPS Malicious URL Database: 5.00422(2025-05-23 09:39)
IoT-Detect: 0.00000(2022-08-17 17:31)
Serial-Number: FG101FTK12345678
BIOS version: 05000020
System Part-Number: P24605-04
Log hard disk: Available
Hostname: XXXXXXXX
Private Encryption: Disable
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 1688
Release Version Information: GA
System time: Fri May 23 12:32:49 2025
Last reboot reason: power cycle


I see one glaring reason for the high memory usage.  This 101 has only 4GB of RAM.  We strive to keep our hardware and software configurations consistent.  I assumed it had 8GB as do the other 101s.

But, I still need to know what thresholds FortiNet recommends for CPU and memory alarm monitoring.  I believe the CPU threshold of 50% (sustained values, not spikes) is a default figure.  As for memory, I think default alarm values are 85% (Warning) and 95% (Critical).  For the purpose of external SNMP monitors, I wonder if we shouldn't be alerted well before the 85% figure.  We've seen instances when the GUI becomes completely unresponsive and the CLI intolerably sluggish with lower memory use numbers.

Currently, we monitor (via SNMP) CPU, Memory, and Low-Memory use per the default thresholds above.  Should I adjust those thresholds?  Are there other key parameters indicating an imminent freeze-up or sharp performance decline that we should monitor via SNMP?

Thanks.

[sferoz · Accepted solution]

Since the specific unit is GEN-1 with 4GB mem, would recommend memory optimization based on the config, tweaks can be applied to reduce overall memory usage.

Some KB references for memory optimization:
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Free-up-memory-to-avoid-conserve-mode/ta-p...

[Yurisk · Accepted solution]

For 100F with 4 Gb RAM it is normal for memory to be used 60-70% all the time, including when idle. It is not a problem by itself until this usage climbs up. You can enable (if this FGT has access to some mail server) Automation stitch on low memory/conserve mode to be sent mail each time (but it is kinda late). For CPU there is no recommended usage limits, as long as it is less than 90% load, it will function. memory, on the other hand, once passes the pre-defined thresholds, will cause FGT to enter Conserve Mode, in which no new connection will pass through the firewall, and existing ones will suffer slowness/irresponsiveness. 

You can read mor eon Conserve Mode here https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-conserve-mode-is-triggered/ta-p/198580 

So, the memory usage of 75% is not an indication of some problems - just watch the RAM usage with SNMP, and specifically watch for trends - if, e.g. it goes to 80% then back to 75% - no problem, but if it only increases steadily, you may have some process (WAD) leaking memory to look into. 

The only other suggestion would be to stay on 7.2.x train as long as possible, newer FortiOS -> more features -> More RAM usage.

[ilikebbc]

Yurisk/Shameem, thanks for the generous feedback.  I've been reviewing the info with my colleagues for possible further action.

I'd like to begin monitoring the FortiGate's entry into Conserve Mode.  I've searched the FortiGate MIB (downloaded from the FortiGate) and can only find Conserve Mode thresholds.  I need a simple Active/Inactive response.  Can you help?  Thanks.