Troubleshooting Tip: Troubleshooting high SoftIRQ CPU utilization on specific cores
| Description | This article describes how to identify and troubleshoot high softirq CPU utilization on specific CPU cores of a FortiGate device. High softirq usage is typically related to heavy packet processing handled by the CPU, such as high traffic load, broadcast storms, deny traffic, or lack of NPU offloading (for example, IPsec traffic not being offloaded). |
| Scope | FortiGate v7.2, v7.4, NP7. |
| Solution | FortiGate devices experiencing high CPU usage under softirq on specific or multiple cores.
First, check which FortiGate (FGT) model is experiencing high CPU usage. Run the command get system performance status to verify the CPU utilization, particularly the softirq usage.
get system perf stat
diagnose system top Run Time: 0 days, 0 hours and 14 minutes 0U, 0N, 0S, 100I, 0WA, 0HI, 0SI, 0ST; 16010T, 12504F mvl.user 2153 S 2.4 0.5 7 httpsd 5294 S 0.6 0.2 11 httpsd 5295 S 0.2 0.2 0 initXXXXXXXXXXX 1 S 0.2 0.2 10 newcli 5229 S 0.2 0.2 0 snmpd 3312 S 0.2 0.1 3 fltund 3343 S 0.2 0.1 2 lnkmt_passive 3327 S 0.2 0.1 8 node 3304 S 0.0 0.4 1
If no process shows high CPU usage but the softirq count remains high, this typically indicates packet-processing load rather than user-space process consumption.
High softirq may be caused by:
If IPsec tunnels are configured, confirm whether traffic is offloaded to the NPU.
Sometimes, broadcast traffic, a high volume of traffic passing through the FortiGate, or a large amount of denied traffic may not be noticeable. However, high softirq utilization may be observed on certain CPU cores or across all CPU cores.
If any IPsec tunnels are configured on the FortiGate, ensure that NPU drop counters are checked.
diagnose vpn tunnel list name Test_Tunnel
list ipsec tunnel by names in vd 0 ------------------------------------------------------ name=H5_H4_ISP1 ver=2 serial=f 114.16.36.18:0->150.4.132.35:0 nexthop=114.16.36.17 tun_id=150.4.132.35 tun_id6=::150.4.132.35 status=up dst_mtu=1500 weight=1 bound_if=7 real_if=7 lgwy=static/1 tun=intf mode=auto/1 encap=vpnid-ipip/16952 options[4238]=npu create_dev frag-rfc role=sync-primary accept_traffic=1 overlay_id=5 proxyid_num=1 child_num=0 refcnt=5 ilast=0 olast=0 ad=/0 stat: rxp=76710 txp=119705 rxb=11516661 txb=11686688 dpd: mode=on-demand on=1 status=ok idle=5000ms retry=3 count=0 seqno=418 natt: mode=none draft=0 interval=0 remote_port=0 fec: egress=0 ingress=0 proxyid=H5_H4_ISP1 proto=0 sa=1 ref=65 serial=1 src: 0:0.0.0.0-255.255.255.255:0 dst: 0:0.0.0.0-255.255.255.255:0 SA: ref=57 options=10227 type=00 soft=0 mtu=1426 expire=1014/0B replaywin=2048 seqno=7b2c esn=0 replaywin_lastseq=00008e59 qat=0 rekey=0 hash_search_len=1 life: type=01 bytes=0/0 timeout=3298/3600 dec: spi=d69141f8 esp=aes-gcm key=36 yyyyyyyyyyyy ah=null key=0 enc: spi=92a09839 esp=aes-gcm key=36 xxxxxxxxxxxxx ah=null key=0 dec:pkts/bytes=19926/2976476, enc:pkts/bytes=31522/5418584 npu_flag=00 npu_rgwy=150.4.132.35 npu_lgwy=114.16.36.18 npu_selid=6 dec_npuid=0 enc_npuid=0
Sometimes, traffic may not be offloaded to the NPU even when NPU offloading is explicitly enabled for the IPsec tunnel. If the NPU flag is displayed as npu_flag=00 while NPU offloading is enabled, the traffic is processed by the CPU. If multiple IPsec tunnels are configured, this can significantly increase CPU load and eventually cause high softirq utilization. In such cases, the lack of NPU offload for IPsec traffic may also result in increasing NP7 drop counters.
Additionally, run the following command to check for packet drops on the NPU:
diagnose npu np7 dce-drop all
When packet drops are observed, rebooting the FortiGate may temporarily mitigate the issue.
Related documents: |