| There is a known issue #1055740 that can cause high CPU usage when an admin is logged into the web GUI and a USB disk with many files is connected to the FortiGate. Symptoms of this issue are high CPU usage in system space when the admin has logged in. To check where the high CPU usage is, use the commands 'get system performance status' or 'diagnose system mpstat'. For example: FGT # get system performance status
CPU states: 0% user 25% system 0% nice 75% idle 0% iowait 0% irq 0% softirq CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU1 states: 3% user 0% system 0% nice 97% idle 0% iowait 0% irq 0% softirq CPU2 states: 0% user 99% system 0% nice 1% idle 0% iowait 0% irq 0% softirq CPU3 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq CPU4 states: 0% user 100% system 0% nice 0% idle 0% iowait 0% irq 0% softirq CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq CPU7 states: 0% user 1% system 0% nice 99% idle 0% iowait 0% irq 0% softirq Memory: 1963828k total, 1470896k used (74.9%), 332996k free (17.0%), 159936k freeable (8.1%) Average network usage: 8920 / 9019 kbps in 1 minute, 10671 / 11190 kbps in 10 minutes, 10431 / 10523 kbps in 30 minutes Maximal network usage: 12764 / 12833 kbps in 1 minute, 52921 / 69513 kbps in 10 minutes, 112727 / 112712 kbps in 30 minutes Average sessions: 3108 sessions in 1 minute, 3791 sessions in 10 minutes, 3716 sessions in 30 minutes Maximal sessions: 3195 sessions in 1 minute, 4484 sessions in 10 minutes, 4773 sessions in 30 minutes Average session setup rate: 11 sessions per second in last 1 minute, 15 sessions per second in last 10 minutes, 18 sessions per second in last 30 minutes Maximal session setup rate: 39 sessions per second in last 1 minute, 132 sessions per second in last 10 minutes, 322 sessions per second in last 30 minutes Average NPU sessions: 809 sessions in last 1 minute, 956 sessions in last 10 minutes, 1010 sessions in last 30 minutes Maximal NPU sessions: 844 sessions in last 1 minute, 1337 sessions in last 10 minutes, 1462 sessions in last 30 minutes Average nTurbo sessions: 597 sessions in last 1 minute, 731 sessions in last 10 minutes, 780 sessions in last 30 minutes Maximal nTurbo sessions: 621 sessions in last 1 minute, 989 sessions in last 10 minutes, 1103 sessions in last 30 minutes Virus caught: 0 total in 1 minute IPS attacks blocked: 0 total in 1 minute Uptime: 206 days, 15 hours, 19 minutes FGT # diagnose system mpstat
Gathering data, wait 5 sec, press any key to quit. ..0..1..2..3..4 TIME CPU %usr %nice %sys %iowait %irq %soft %steal %idle 03:08:20 PM all 0.44 0.00 25.52 0.00 0.00 0.15 0.00 73.89 0 0.20 0.00 5.18 0.00 0.00 1.00 0.00 93.63 1 0.20 0.00 99.80 0.00 0.00 0.00 0.00 0.00 2 0.60 0.00 7.37 0.00 0.00 0.20 0.00 91.83 3 0.20 0.00 88.65 0.00 0.00 0.00 0.00 11.15 4 0.39 0.00 0.00 0.00 0.00 0.00 0.00 99.61 5 0.79 0.00 0.00 0.00 0.00 0.00 0.00 99.21 6 0.00 0.00 0.79 0.00 0.00 0.00 0.00 99.21 7 1.18 0.00 0.79 0.00 0.00 0.00 0.00 98.03 Another symptom of this issue is the httpsd process showing high CPU usage. This can be verified using the 'diagnose system top' command. FGT # diagnose system top Run Time: 52 days, 2 hours and 40 minutes 3U, 0N, 24S, 73I, 0WA, 0HI, 0SI, 0ST; 1917T, 291F httpsd 2062 R 97.4 0.8 1 httpsd 2072 R -
0.7 3 In order to verify that the issue matches, the CPU profiler needs to be run. Because this issue can change CPU cores while running the profiler, it is best to profile all CPU cores of the device. To profile all CPU cores at once use the following syntax: diagnose system profile cpumask 0-X <----- Replace X with the highest CPU core number.
For example, a FortiGate 60F has 8 CPU cores so the syntax would be: diagnose system profile cpumask 0-7 To run the CPU profiler use the following CLI commands: diagnose system profile cpumask 0-X diagnose system profile start diagnose system profile stop diagnose system profile show order If the output is similar to the following, it is likely a match for this issue: 0xffffffc000084260:     9845 default_idle+0x10/0x20 0xffffffc0001478b8:     1339 prepend_path.isra.0+0x58/0x1f0 0xffffffc000151ca4:     1231 seq_list_start+0x20/0x3c 0xffffffc000152a1c:     724 seq_path+0x68/0x10c The issue can be caused by a FortiOS attempting to read many files stored on a connected USB disk. If there is a USB disk connected to the FortiGate disconnect it from the FortiGate. This issue is resolved in FortiOS 7.6.5, see the resolved issues in the release notes. This issue will be resolved in FortiOS 7.4.10 and 8.0.0 when released. |