| During IPS signature update, insufficient memory may trigger ipsengine and ipshelper to enter a locked state with persistent high CPU use in iowait. The high CPU usage issue 1025114 is addressed and fixed in the following firmware versions: The fix prevents high CPU usage and IPS processes from entering a 'D' state triggered by high memory use. It does not prevent high memory use during FortiGuard updates. To address the memory use issue, disable IPS signature hardware acceleration using the workaround below, and see related articles for memory use mitigation steps.
The following output is taken from a firewall during high CPU usage triggered by insufficient memory during the FortiGuard update. get system performance status CPU states: 1% user 0% system 0% nice 54% idle 45% iowait 0% irq 0% softirq CPU0 states: 5% user 3% system 0% nice 2% idle 89% iowait 0% irq 1% softirq CPU1 states: 0% user 0% system 0% nice 53% idle 47% iowait 0% irq 0% softirq CPU2 states: 0% user 0% system 0% nice 1% idle 99% iowait 0% irq 0% softirq CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU5 states: 0% user 0% system 0% nice 52% idle 48% iowait 0% irq 0% softirq CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU7 states: 6% user 0% system 0% nice 18% idle 76% iowait 0% irq 0% softirq Memory: 1957612k total, 1690808k used (86.4%), 119700k free (6.1%), 147104k freeable (7.5%) Average network usage: 662 / 723 kbps in 1 minute, 862 / 809 kbps in 10 minutes, 4436 / 4434 kbps in 30 minutes Maximal network usage: 1817 / 1809 kbps in 1 minute, 31313 / 31307 kbps in 10 minutes, 39774 / 39771 kbps in 30 minutes Average sessions: 2716 sessions in 1 minute, 1147 sessions in 10 minutes, 559 sessions in 30 minutes Maximal sessions: 3058 sessions in 1 minute, 3058 sessions in 10 minutes, 3058 sessions in 30 minutes Average session setup rate: 10 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 2 sessions per second in last 30 minutes Maximal session setup rate: 15 sessions per second in last 1 minute, 18 sessions per second in last 10 minutes, 20 sessions per second in last 30 minutes Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 1 sessions in last 30 minutes Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 3 days, 21 hours, 38 minutes diagnose sys top 1 30 Run Time: 3 days, 21 hours and 37 minutes 0U, 0N, 1S, 75I, 24WA, 0HI, 0SI, 0ST; 1911T, 127F | newcli | 24183 | R | 7.1 | 0.6 | 1 | | sshd | 16407 | S | 7.1 | 0.5 | 1 | | ipshelper | 192 | D < | 0 | 14.9 | 0 | | ipsengine | 16379 | D < | 0 | 7.4 | 5 | | ipsengine | 16378 | D < | 0 | 7.4 | 0 | | ipsengine | 16377 | D < | 0 | 7.3 | 0 | | node | 191 | S | 0 | 4.2 | 4 | | wad | 15843 | S | 0 | 3.6 | 4 | | scanunitd | 24178 | S < | 0 | 2.9 | 4 | | miglogd | 189 | D | 0 | 1.8 | 0 | | cw_acd | 218 | S | 0 | 1.6 | 0 | | cmdbsvr | 142 | S | 0 | 1.6 | 0 | | forticron | 180 | S | 0 | 1.5 | 3 | | reportd | 190 | S | 0 | 1.5 | 1 | | wad | 15845 | D | 0 | 1.4 | 0 | | wad | 15834 | S | 0 | 1.4 | 3 | | forticldd | 181 | S | 0 | 1.3 | 2 | | csfd | 236 | S | 0 | 1.2 | 2 | | fgfmd | 217 | S | 0 | 1.2 | 7 | | initXXXXXXXXXXX | 1 | S | 0 | 1.1 | 0 | | httpsd | 175 | S | 0 | 1.1 | 4 | | newcli | 16408 | S | 0 | 1.1 | 1 | | miglogd | 325 | D | 0 | 1.1 | 0 | | dnsproxy | 243 | S | 0 | 1 | 7 | | cid | 240 | S | 0 | 1 | 6 | | extenderd | 235 | S | 0 | 0.9 | 1 | | fcnacd | 187 | S | 0 | 0.8 | 6 | | autod | 237 | S | 0 | 0.8 | 2 | | updated | 197 | S | 0 | 0.8 | 5 | | urlfilter | 333 | S < | 0 | 0.8 | 7 | Workaround: Where possible, prevent ipsengine processes from becoming locked by upgrading to an unaffected FortiOS version. Whether upgrading or not, it is recommended to consider all available memory optimizations for devices having 2GB of memory. At a minimum, it is recommended to 'set cp-accel-mode none' in the IPS configuration, since otherwise, these devices have the known limitation that ipshelper contributes significantly to memory use during FortiGuard update.
config ips global set cp-accel-mode none end This configuration is a trade-off that significantly reduces memory usage during updates at the expense of background CPU usage and is a significant stability improvement for 2GB models in most environments. It is recommended to monitor firewall CPU use after making this change to verify it remains at acceptable levels. Related articles: Troubleshooting Tip: Conserve mode due to ipshelper in lower end models
Technical Tip: FortiGate is entering into Conserve Mode during FortiGuard Updates Technical Tip: Reduce memory usage by reducing the number of spawned daemons |