Staff

Troubleshooting Tip: High CPU with 100% softIRQ when creating or editing a VLAN Interface in the GUI

0 replies
448 views

Description

This article describes the high CPU utilization observed when creating or editing a VLAN interface in the FortiGate GUI. The 'httpsd' process consumes high CPU, resulting in interface bandwidth drops and potential traffic impact.

Scope

FortiGate-340xE, FortiGate-330xE, FortiGate-110xE, FortiGate-180xF.

FortiOS v7.0.x, v7.2.x, v7.4.11, v7.6.6.

Solution

Reproduction scenario:

Create or edit a VLAN interface via the web UI, selecting the parent interface before entering the IP address, with traffic running at approximately 4Gbps or higher. The issue is not observed when performing configuration changes using the CLI.

When creating a new VLAN interface via the web UI, the high CPU issue is triggered by a specific operation sequence: click "Add VLAN Interface", first select the parent interface, then fill in the VLAN ID and VDOM, and finally enter the IP address and leave the input field (e.g., clicking other areas or pressing Enter). The issue occurs as long as the parent interface is selected before entering the IP address of the VLAN interface, causing the httpsd process to occupy 100% of the CPU.

Traffic trigger points for the reported FortiGate models:

FGT3401E 64 * CPU 4 Gbps or higher
FGT1100E 12 * CPU 14 Gbps or higher
FGT3001F 16 * CPU 10 Gbps or higher
FGT1801F 16 * CPU 10 Gbps or higher

Symptoms:

The output of 'get system performance status' shows 100% softIRQ for some CPU core, as shown below.

The output of 'diagnose sys top' shows 'httpsd' consuming almost 100%, as shown below.

Interface bandwidth drops and potential traffic impact.

Diagnostic output:

FGT# get system performance status
CPU states: 0% user 1% system 0% nice 49% idle 0% iowait 0% irq 50% softirq
CPU0 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU2 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU3 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU4 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU5 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU6 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU7 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU8 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU9 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU10 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU11 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU12 states: 0% user 100% system 0% nice 0% idle 0% iowait 0% irq 0% softirq
CPU13 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU14 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU15 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU16 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU17 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU18 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU19 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU20 states: 4% user 1% system 0% nice 95% idle 0% iowait 0% irq 0% softirq
CPU21 states: 2% user 0% system 0% nice 98% idle 0% iowait 0% irq 0% softirq
CPU22 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU23 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU24 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU25 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU26 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU27 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU28 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU29 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU30 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU31 states: 0% user 0% system 0% nice 100% idle 0% iowait 0% irq 0% softirq
CPU32 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU33 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU34 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU35 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU36 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU37 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU38 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU39 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU40 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU41 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU42 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU43 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU44 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU45 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU46 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU47 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU48 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU49 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU50 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU51 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU52 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU53 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU54 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU55 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU56 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU57 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU58 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU59 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU60 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU61 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU62 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
CPU63 states: 0% user 0% system 0% nice 0% idle 0% iowait 0% irq 100% softirq
Memory: 98991556k total, 20693920k used (20.9%), 77927668k free (78.7%), 369968k freeable (0.4%)
Average network usage: 2531073 / 2526681 kbps in 1 minute, 2544900 / 2541424 kbps in 10 minutes, 2575916 / 2572419 kbps in 30 minutes
Maximal network usage: 2645222 / 2641596 kbps in 1 minute, 2776384 / 2772912 kbps in 10 minutes, 4264541 / 4258405 kbps in 30 minutes
Average sessions: 1032 sessions in 1 minute, 1032 sessions in 10 minutes, 819 sessions in 30 minutes
Maximal sessions: 1032 sessions in 1 minute, 1032 sessions in 10 minutes, 1036 sessions in 30 minutes
Average session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 0 sessions per second in last 30 minutes
Maximal session setup rate: 0 sessions per second in last 1 minute, 0 sessions per second in last 10 minutes, 4 sessions per second in last 30 minutes
Average NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal NPU sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Average nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Maximal nTurbo sessions: 0 sessions in last 1 minute, 0 sessions in last 10 minutes, 0 sessions in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 1 days, 10 hours, 37 minutes

FGT # diagnose sys top 1 40
09:03:20 AM up 1 days, 10 hours and 39 minutes
0U, 0N, 1S, 49I, 0WA, 0HI, 50SI, 0ST; 96671T, 76103F
          httpsd 16473 R 99.9 0.0 12
        bcm.user 456 S < 10.5 0.0 21
          fltund 612 S 1.0 0.0 4
       forticron 577 S 0.5 0.0 15
          newcli 16793 R 0.5 0.0 15
            node 2493 S 0.0 0.1 1
       ipshelper 625 S 0.0 0.0 6
          cw_acd 605 S 0.0 0.0 30
          flcfgd 610 S 0.0 0.0 4
      telemetryd 616 S 0.0 0.0 24
          newcli 16743 S 0.0 0.0 13
         cmdbsvr 520 S 0.0 0.0 0
        cid-scan 6856 S 0.0 0.0 14
            csfd 617 S 0.0 0.0 22
      http_authd 572 S 0.0 0.0 25
           fgfmd 604 S 0.0 0.0 4
             wad 720 S 0.0 0.0 27
             wad 756 S 0.0 0.0 6
             wad 748 S 0.0 0.0 24

Workaround:

Enable 'dedicated-management-cpu'.

config system npu
    set dedicated-management-cpu enable
end

Restart 'httpsd' process using the following options.

Using the process ID:

diagnose sys process pidof httpsd
diagnose sys process kill 11 <pid>

Using the process name. This command requires 'super_admin' administrator profile.

fnsysctl killall httpsd

Another option to perform a failover if FortiGate is in an HA pair: When HA override is disabled, refer to Technical Tip: Restoring HA Primary role after a failover using 'diagnose sys ha reset uptime' (ha 'set override disable' context).

When HA override is enabled, change the priority of the current primary to a lower priority value to force failover.

Resolution.

The issue is scheduled to be fixed in upcoming FortiOS release v7.4.12, v7.6.7, and v8.0.0.

Sign up

Login with SSO

Login to the community

Login with SSO