Description
This article describes a mitigation for lower-end model FortiGate with 2GB of RAM to avoid conserve mode due to increased ipshelper memory use during FortiGuard update.
Scope
FortiGate v7.0 and later.
Solution
The following output is taken from FortiGate 60F during FortiGuard IPS signature update:
get system performance status
CPU states: 13% user 2% system 0% nice 85% idle 0% iowait 0% irq 0% softirq
CPU0 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 100% user 0% system 0% nice 0% idle 0% iowait 0% irq 0% softirq
CPU3 states: 6% user 21% system 0% nice 73% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
Memory: 1957612k total, 1690808k used (86.4%), 119700k free (6.1%), 147104k freeable (7.5%)
Average network usage: 662 / 723 kbps in 1 minute, 862 / 809 kbps in 10 minutes, 4436 / 4434 kbps in 30 minutes
Maximal network usage: 1817 / 1809 kbps in 1 minute, 31313 / 31307 kbps in 10 minutes, 39774 / 39771 kbps in 30 minutes
Average sessions: 2716 sessions in 1 minute, 1147 sessions in 10 minutes, 559 sessions in 30 minutes
Maximal sessions: 3058 sessions in 1 minute, 3058 sessions in 10 minutes, 3058 sessions in 30 minutes
Average session setup rate: 10 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 2 sessions per second in last 30 minutes
Maximal session setup rate: 15 sessions per second in last 1 minute, 18 sessions per second in last 10 minutes, 20 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 1 sessions in last 30 minutes
Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 3 days, 21 hours, 38 minutes
diag sys top-mem 30
ipshelper (192): 273509kB
node (191): 82268kB
ipsengine (16379): 73474kB
ipsengine (16378): 72402kB
ipsengine (16377): 71354kB
wad (15843): 50977kB
scanunitd (24178): 39431kB
wad (15845): 18091kB
reportd (190): 14976kB
cid (240): 14769kB
cw_acd (218): 14515kB
forticldd (181): 13230kB
cmdbsvr (142): 13164kB
miglogd (189): 11914kB
forticron (180): 10393kB
csfd (236): 8041kB
miglogd (325): 7793kB
dnsproxy (243): 7744kB
fgfmd (217): 7618kB
newcli (16408): 7369kB
wad (15834): 7049kB
nsm (151): 6827kB
initXXXXXXXXXXX (1): 6640kB
urlfilter (333): 5875kB
imi (314): 5744kB
bgpd (156): 5429kB
wad (15840): 5058kB
pdmd (160): 4663kB
ospfd (154): 4649kB
pimd (158): 4645kB
Top-30 memory used: 869611kB
When these diagnostics were taken, the ipshelper process was using approximately 14% of the total memory of this device. Although the elevated memory use typically only lasts for a few seconds, this can still cause significant network disruption. In severe cases where memory use is already high, the device may become unreachable and not recover until after a power cycle.
Workaround:
Most of the memory use in ipshelper during a FortiGuard update is associated with unpacking IPS signatures for hardware acceleration and does not occur if hardware acceleration is disabled for IPS pre-match.
While disabling this hardware acceleration as below can increase CPU usage for IPS processing, it is recommended to disable it for most 2GB models. This is made the default for FortiGate and FortiWiFi 4xF/6xF families as of FortiOS v.7.6.1, see Release Notes
config ips global
set cp-accel-mode none
end
A previous version of this article incorrectly stated the increased memory use was fixed in v7.2.11, v7.4.6, and v7.6.1.
While there are some memory optimizations in these versions, a significant memory spike in ipshelper during IPS signature updates is still expected, and devices with 2GB memory use should continue to have the workaround applied.
More info on np-accel-mode and cp-accel-mode:
Technical Tip: IPSA offloads flow-based advanced pattern matching
Technical Tip: Nturbo functions within FortiOS
IPSA offloads flow-based pattern matching
NTurbo and IPSA
Related articles:
Technical Tip: FortiGate is entering into Conserve Mode during FortiGuard Updates
Technical Tip: Reduce memory usage by reducing the number of spawned daemons
