|
When FortiGate experiences high CPU utilization at one core, follow the steps mentioned in this KB article: Troubleshooting Tip: How high CPU usage should be investigated.
get sys performance status
CPU states: 0% user 0% system 0% nice 99% idle 0% iowait 1% irq 0% softirq
CPU0 states: 90% user 0% system 0% nice 9% idle 0% iowait 1% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 42% user 0% system 0% nice 58% idle 0% iowait 0% irq 0% softirq
CPU3 states: 37% user 0% system 0% nice 63% idle 0% iowait 0% irq 0% softirq
CPU4 states: 16% user 0% system 0% nice 84% idle 0% iowait 0% irq 0% softirq
CPU5 states: 1% user 0% system 0% nice 99% idle 0% iowait 0% irq 0% softirq
CPU6 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU7 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
Memory: 1963896k total, 1022008k used (52.0%), 624128k free (31.8%), 317760k freeable (16.2%)
Average network usage: 118883 / 115211 kbps in 1 minute, 107489 / 103612 kbps in 10 minutes, 85740 / 81661 kbps in 30 minutes
Maximal network usage: 187908 / 181927 kbps in 1 minute, 231860 / 226202 kbps in 10 minutes, 231860 / 226202 kbps in 30 minutes
Average sessions: 8709 sessions in 1 minute, 8297 sessions in 10 minutes, 8542 sessions in 30 minutes
Maximal sessions: 8957 sessions in 1 minute, 9158 sessions in 10 minutes, 9663 sessions in 30 minutes
Average session setup rate: 80 sessions per second in last 1 minute, 83 sessions per second in last 10 minutes, 86 sessions per second in last 30 minutes
Maximal session setup rate: 127 sessions per second in last 1 minute, 214 sessions per second in last 10 minutes, 317 sessions per second in last 30 minutes
Average NPU sessions: 1920 sessions in last 1 minute, 1827 sessions in last 10 minutes, 1681 sessions in last 30 minutes
Maximal NPU sessions: 2024 sessions in last 1 minute, 2043 sessions in last 10 minutes, 2073 sessions in last 30 minutes
Average nTurbo sessions: 1826 sessions in last 1 minute, 1741 sessions in last 10 minutes, 1602 sessions in last 30 minutes
Maximal nTurbo sessions: 1926 sessions in last 1 minute, 1961 sessions in last 10 minutes, 1985 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 0 days, 0 hours, 50 minutes
Here, one core of the CPU is high at 90% at the user space. Check the output of 'diagnose sys top' and check which process is taking a high CPU.
Run Time: 0 days, 0 hours and 45 minutes
22U, 0N, 2S, 75I, 0WA, 0HI, 1SI, 0ST; 1917T, 597F
authd 183 R 7.5 0.9 1
dhcpd 198 R 92.0 1.2 0
src-vis 199 S 7.3 1.2 3
ipsengine 292 S < 1.1 2.5 6
ipsengine 294 S < 1.1 2.4 7
miglogd 259 S 1.1 0.9 7
httpsd 575 S 0.9 1.0 2
node 173 S 0.7 2.7 2
ipsengine 293 S < 0.7 2.4 5
httpsd 577 S 0.7 1.0 2
dnsproxy 214 S 0.7 1.0 3
updated 195 S 0.5 0.8 7
wad 237 S 0.3 1.2 6
httpsd 576 S 0.1 1.0 3
ipshelper 182 S < 0.0 2.5 7
cmdbsvr 135 S 0.0 2.3 0
miglogd 190 S 0.0 1.6 0
extenderd 233 S 0.0 1.5 2
cw_acd 216 S 0.0 1.3 1
forticron 180 S 0.0 1.2 6
In the 'diagnose sys top output', the dhcpd process is taking all CPU time.
Try to restart the DHCP process and observe if that helps reduce the CPU usage. If restarting dhcpd process does not help to reduce the CPU time, run a packet capture at port 67 or 68.
Check if there is a high amount of DHCP packets coming from any specific interface as below.
EXX-TX-BEL-00MBP-DMG1 # diagnose sniffer packet any " port 67 or port 68 " 4 0 l
interfaces=[any]
filters=[ port 67 or port 68 ]
2025-04-17 12:42:31.865898 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.866297 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.866858 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.867299 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.867867 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.868325 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.868886 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.869351 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.869381 VLAN10 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.869386 internal1 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.869951 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.870355 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.870978 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.871396 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.871964 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.872358 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.872928 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.873402 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.873597 VLAN10 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.873602 internal1 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.874011 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.874405 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.874969 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.875431 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.876057 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.876465 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.878052 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.878181 VLAN10 out 192.168.1.10.67 -> 192.168.1.3.67: udp 311
2025-04-17 12:42:31.878186 internal1 out 192.168.1.10.67 -> 192.168.1.3.67: udp 311
2025-04-17 12:42:31.878536 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.879103 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.879522 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.880080 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.880550 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.881116 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.881534 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.882166 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.882377 VLAN10 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.882383 internal1 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.882766 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.883322 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.883782 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.884342 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.884745 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.885312 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.885706 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.886290 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.886669 VLAN10 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.886674 internal1 out 192.168.1.10.67 -> 192.168.1.1.67: udp 311
2025-04-17 12:42:31.886748 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.887359 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.887824 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.888383 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.888801 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.889370 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.889812 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.890373 VLAN10 in 192.168.1.1.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.890774 VLAN10 in 192.168.1.3.67 -> 192.168.1.10.67: udp 304
2025-04-17 12:42:31.890856 VLAN10 out 192.168.1.10.67 -> 192.168.1.3.67: udp 311
Or complete Packet capture can be taken from the GUI to see the complete DHCP process:
There is an unexpected number of DHCP requests coming from the internal subnet, which is causing the high CPU for the DHCP process.
To resolve it, verify the IP address sending the DHCP packets from the internal subnet, and an investigation needs to be done in the internal network, which is sending thousands of packets in milliseconds. Try shutting down that specific interface to bring the CPU down.
Run also DHCP process debugs and check if there is any error showing in the process.
diagnose debug disable
diagnose debug application dhcpd -1
diagnose debug enable
Keep the debugs running and disable the debugs from the following command:
diagnose debug disable
|
