| Solution |
- Run the following CLI command. Example output is shown below.
get system performance status
CPU states: 8% user 3% system 0% nice 87% idle 2% iowait 0% irq 0% softirq
CPU0 states: 8% user 3% system 0% nice 87% idle 2% iowait 0% irq 0% softirq
Memory: 2005244k total, 816796k used (40.7%), 1030464k free (51.4%), 157984k freeable (7.9%)
Average network usage: 120 / 18 kbps in 1 minute, 259 / 38 kbps in 10 minutes, 194 / 29 kbps in 30 minutes
Maximal network usage: 804 / 146 kbps in 1 minute, 804 / 146 kbps in 10 minutes, 804 / 146 kbps in 30 minutes
Average sessions: 73 sessions in 1 minute, 59 sessions in 10 minutes, 44 sessions in 30 minutes
Maximal sessions: 105 sessions in 1 minute, 105 sessions in 10 minutes, 105 sessions in 30 minutes
Average session setup rate: 2 sessions per second in last 1 minute, 4 sessions per second in last 10 minutes, 3 sessions per second in last 30 minutes
Maximal session setup rate: 23 sessions per second in last 1 minute, 23 sessions per second in last 10 minutes, 23 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 0 hours, 0 minutes
Run the command above a few times and compare patterns of CPU usage, throughput, and the sessions' setup rates.
- Look at CPU States:
- 8% user -> CPU used in user space e.g. by an application process.
- 3% system -> CPU used in kernel space or by a kernel function.
- 0% nice -> CPU used by processes with a 'nice' value.
- 87% idle -> CPU in idle state - the bigger the percentage value, the less loaded this CPU core is.
- 2% iowait -> CPU waiting for IO operations - May indicate a faulty memory if too high.
- 0% irq -> CPU busy with hardware interrupts, rarely high for Fortigate.
- 0% softirq -> CPU busy with Software interrupts. Commonly high with high traffic loads and/or traffic not offloaded to NP.
- Look at the bandwidth and session setup rate: Maximal network usage: 804 / 146 kbps in 1 minute, 804 / 146 kbps in 10 minutes, 804 / 146 kbps in 30 minutes. Average session setup rate: 2 sessions per second in the last 1 minute, 4 sessions per second in the last 10 minutes, and 3 sessions per second in the last 30 minutes. Check the highest maximum bandwidth measured and session setup rate, and compare it to the device datasheet. If values are too high, investigate if it is expected for the environment.
- If CPU usage is high in a user space use, run 'diag sys top 1 45' in the CLI to find CPU usage per process instance.
Example screenshot:
In this particular case, eap_proxy (process) use 99.9% of CPU. The commands below will provide more CPU information related to the user process. In this case, 1130 is the process ID of eap_proxy:
diagnose sys process dump 1130
diagnose sys process pstack 1130
diagnose sys process trace 1130
- If the CPU is high in a kernel space, run the CPU profiler to identify the function being called the most:
diagnose sys profile cpumask X <----- Where X is the CPU core with the highest CPU usage in the system space.
diagnose sys profile start
Wait 20-30 seconds:
diagnose sys profile stop
diagnose sys profile show order
- If the CPU is mostly busy with softIRQ: Check and compare number of offloaded sessions:
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Compare with the total sessions:
Average sessions: 73 sessions in 1 minute, 59 sessions in 10 minutes, 44 sessions in 30 minutes
Maximal sessions: 105 sessions in 1 minute, 105 sessions in 10 minutes, 105 sessions in 30 minutes
Most of the sessions should be offloaded.
Run the command 'diagnose hardware sysinfo interrupts' multiple times.
Add the command 'diagnose sys profile report' on Teraterm or Auto Script for intermittent issues.
Attach all of the outputs to the support ticket.
Related articles:
Troubleshooting Tip: Best use for the 'diagnose sys profile report' command
Troubleshooting Tip: How high CPU usage should be investigated
Troubleshooting Tip: FortiGate CPU Profiling
|
