Troubleshooting Tip: High softirq CPU due to inoptimal drivers on VMWare

The first step to troubleshoot CPU issues is always to determine the CPU module of usage. This issue is regarding the softirq module.

   FGT-01 # get sys performance status

   CPU states: 6% user 6% system 0% nice 65% idle 0% iowait 0% irq 23% softirq

   CPU0 states: 3% user 2% system 0% nice 3% idle 0% iowait 0% irq 92% softirq

   CPU1 states: 10% user 12% system 0% nice 78% idle 0% iowait 0% irq 0% softirq

   CPU2 states: 4% user 6% system 0% nice 90% idle 0% iowait 0% irq 0% softirq

   CPU3 states: 6% user 6% system 0% nice 88% idle 0% iowait 0% irq 0% softirq

   Memory: 12332116k total, 2857160k used (23.2%), 9016844k free (73.1%), 458112k freeable (3.7%)

   Average network usage: 428015 / 433566 kbps in 1 minute, 447142 / 452540 kbps in 10 minutes, 415709 / 421098 kbps in 30 minutes

   Average sessions: 66776 sessions in 1 minute, 66971 sessions in 10 minutes, 68353 sessions in 30 minutes

   Average session setup rate: 512 sessions per second in last 1 minute, 515 sessions per second in last 10 minutes, 524 sessions per second in last 30 minutes

When performing a Troubleshooting Tip: FortiGate CPU Profiling, several commands can be seen starting with the name e1000. These references a specific network driver. While this specific driver does work, it causes extra load on the FortiGate and should be avoided.

   FGT-01 # diagnose sys profile show order

   0xffffffff80392bcc:                    254   __memcpy+0x6c/0x120

   0xffffffff805e9918:                    225   e1000_xmit_frame+0xa26/0xf24

   0xffffffff805e8c14:                    142   e1000_clean+0x31f/0x5fd

   0xffffffff805ea03c:                     99   e1000_clean_rx_irq+0x188/0x481

   0xffffffffa0006c68:                     50   ip_tuple_find_get+0xaf/0x18d

   0xffffffff805e799c:                     45   e1000_alloc_rx_buffers+0x22b/0x4b6

To confirm what drivers are used on the FortiGate, check any non-logical interface with get hardware nic <interface> or run diagnose hardware lspci -v which is also part of the Technical Tip: Download Debug Logs and 'execute tac report'.

   get hardware nic port1

   Name:            port1

   Driver:          e1000

   diagnose hardware lspci -v

   02:00.0 Class 0200: Device 8086:100f (rev 01)

                Subsystem: Device 15ad:0750

                Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 18

                Memory at fd540000 (64-bit, non-prefetchable) [size=128K]

                Memory at fdff0000 (64-bit, non-prefetchable) [size=64K]

                I/O ports at 2000 [size=64]

                [virtual] Expansion ROM at ebb00000 [disabled] [size=64K]

                Capabilities: [dc] Power Management version 2

                Capabilities: [e4] PCI-X non-bridge device

                Kernel driver in use: e1000   

Solution:

Instead of the e1000 driver the use the VMXNET 3 driver on the VMWare hypervisor.

Related document:

Choosing a network adapter for a virtual machine