| Solution |
Symptoms:
The following are some general symptoms that are known to be related to this issue:
- High iowait CPU usage is observed on the FortiGate (this can be observed with get system performance stat and diagnose sys mpstat in the CLI).
- This type of CPU usage is reported when a process is spending idle time waiting to access data stored on disk/flash storage (i.e. it is not necessarily performing a task, rather it is waiting for the data necessary to proceed with its task).
Example:
CPU states: 1% user 0% system 0% nice 35% idle 64% iowait 0% irq 0% softirq
CPU0 states: 3% user 3% system 0% nice 14% idle 79% iowait 0% irq 1% softirq
CPU1 states: 2% user 0% system 0% nice 87% idle 11% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 25% idle 75% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 42% idle 58% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 1% user 1% system 0% nice 1% idle 97% iowait 0% irq 0% softirq
CPU6 states: 2% user 0% system 0% nice 2% idle 96% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 1% idle 99% iowait 0% irq 0% softirq
- Processes on the FortiGate are shown to be stuck in the D state (this can be observed with diagnose sys top).
- The D state indicates 'uninterruptible sleep', though it is frequently associated with a process waiting for another system call to complete (such as a disk I/O operation).
- Any process could potentially enter the D state for this issue, but processes like miglogd, node, httpsd, ipshelper, and ipsengine have been observed to do so according to past troubleshooting samples.
Example:
Run Time: 66 days, 17 hours and 24 minutes
2U, 0N, 1S, 52I, 45WA, 0HI, 0SI, 0ST; 1917T, 91F
flcfgd 237 S 20.1 0.9 7
hasync 203 D < 3.9 1.7 4
[mmcqd/0] 102 SW 2.4 0.0 0
node 14327 D 1.4 4.4 7
ipshelper 200 S < 0.9 2.1 0
[kworker/1:0] 13827 SW 0.9 0.0 1
[kworker/0:2] 6391 SW 0.9 0.0 0
ipsengine 329 D < 0.4 2.5 7
ipsengine 330 D < 0.4 2.4 6
fortilinkd 235 S 0.4 0.7 0
newcli 7423 S 0.4 0.6 2
newcli 7464 R 0.4 0.6 1
cw_acd 232 S 0.0 6.7 4
miglogd 301 S 0.0 6.3 6
cmdbsvr 137 S 0.0 2.8 1
ipsengine 331 D < 0.0 2.4 5
- The FortiGate is reporting low amounts of 'free' memory (can be observed with get system performance stat | grep Memory and diagnose hardware sysinfo memory).
- Memory usage on the FortiGate is divided between used memory (i.e. in active usage by processes), free memory (unused memory available for allocation), and freeable memory (any memory that is potentially being used but can be freed for more important uses, such as Cached data).
- In this case, low free memory usage.
Additionally, some users have reported additional issues that are derived from the above symptoms:
- High latency/disruptions to end-user traffic when IPS/Application Control inspection is taking place (while ipsengine processes are showing as being in the D state).
- The FortiGate web GUI might be intermittently inaccessible or sluggish (while the httpsd and/or node processes are in D state).
Explanation:
When the FortiGate's free/available memory is low, it's been observed that important data can become paged out to disk (i.e. there is not enough room in memory for everything being asked for, so the disk is used as an overflow/swap space to hold this excess data).
The problem is that this data is still being used by the process that needs it, but now the data must be accessed from the substantially slower disk storage, rather than system memory. While the process is waiting to access its data, it will be placed in the D state to indicate that it is waiting for the system to process the data.
At the same time, iowait CPU usage will increase to reflect the fact that processes are spending significant time waiting for I/O operations to complete before they can resume their work.
Ultimately, the root issue here is that of free memory. If the FortiGate's available free memory becomes too low then it can trigger this memory paging-to-disk behavior (which is necessary for the system to avoid crashing/freezing due to lack of memory), and that can lead to the symptoms described above.
Resolution:
To be clear, the issues described above are triggered when the FortiGate has insufficient free memory. Resolving that issue would require either increasing system memory resources (option for VMs but not hardware FortiGates) or reducing memory usage to allow for more free buffer space.
With that being said, improvements are being made to better handle this situation. Notably, the IPS Engine will be updated as part of FortiOS 7.0.16, 7.2.11, 7.4.6, and 7.6.1 so that critical data is much less likely to be paged out from memory. This will help to prevent the IPS Engine from entering the D state during memory-constrained scenarios and should reduce/remove impacts to user traffic when the IPS Engine is scanning traffic (IPS, App Control, flow-based inspection policies, etc.).
Besides that, the general suggestion here is to try and reduce memory usage on smaller FortiGate models where possible. Refer to the following KB articles for further guidance on this matter:
|
