| Solution |
- Run the following CLI command. Example output is shown below.
get system performance status
CPU states: 8% user 3% system 0% nice 87% idle 2% iowait 0% irq 0% softirq
CPU0 states: 8% user 3% system 0% nice 87% idle 2% iowait 0% irq 0% softirq
Memory: 2005244k total, 816796k used (40.7%), 1030464k free (51.4%), 157984k freeable (7.9%)
Average network usage: 120 / 18 kbps in 1 minute, 259 / 38 kbps in 10 minutes, 194 / 29 kbps in 30 minutes
Maximal network usage: 804 / 146 kbps in 1 minute, 804 / 146 kbps in 10 minutes, 804 / 146 kbps in 30 minutes
Average sessions: 73 sessions in 1 minute, 59 sessions in 10 minutes, 44 sessions in 30 minutes
Maximal sessions: 105 sessions in 1 minute, 105 sessions in 10 minutes, 105 sessions in 30 minutes
Average session setup rate: 2 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 3 sessions per second in last 30 minutes
Maximal session setup rate: 23 sessions per second in last 1 minute, 23 sessions per second in last 10 minutes, 23 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 0 hours, 0 minutes
Run the command above a few times and compare patterns of CPU usage, throughput, and the sessions' setup rates.
- Look at CPU States:
- 8% user -> CPU used in user space e.g. by an application process.
- 3% system -> CPU used in kernel space or by a kernel function.
- 0% nice -> CPU used by processes with a 'nice' value.
- 87% idle -> CPU in idle state - the bigger the percentage value, the less loaded this CPU core is.
- 2% iowait -> CPU waiting for IO operations - May indicate a faulty memory if too high.
- 0% irq -> CPU busy with hardware interrupts, rarely high for Fortigate.
- 0% softirq -> CPU busy with Software interrupts. Commonly high with high traffic loads and/or traffic not offloaded to NP.
- Look at the bandwidth and session setup rate:
Maximal network usage: 804 / 146 kbps in 1 minute, 804 / 146 kbps in 10 minutes, 804 / 146 kbps in 30 minutes.
Average session setup rate: 2 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 3 sessions per second in last 30 minutes.
Check the highest maximum bandwidth measured and session setup rate, and compare it to the device datasheet. If values are too high, investigate if it is expected for the environment.
- If CPU usage is high in a user space use, run 'diag sys top 1 45' in the CLI to find CPU usage per process instance.
- If CPU is high in a kernel space, run the CPU profiler to identify the function being called the most:
diag sys profile cpumask X<- Where X is the CPU core with the highest CPU usage in the system space.
diag sys profile start
Wait 20-30 seconds.
diag sys profile stop
diag sys profile show order
- If the CPU is mostly busy with softIRQ:
Check and compare number of offloaded sessions:
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Compare with the total sessions:
Average sessions: 73 sessions in 1 minute, 59 sessions in 10 minutes, 44 sessions in 30 minutes
Maximal sessions: 105 sessions in 1 minute, 105 sessions in 10 minutes, 105 sessions in 30 minutes
Most of the sessions should be offloaded. If they are not, refer to this section of the FortiGate documentation.
Run the command 'diagnose hardware sysinfo interrupts' multiple times.
Attach all of the outputs to the support ticket.
|
