High CPU Usage on FortiGate VM02 with VXLAN over IPSEC and VDOMs

[5q46n2te8jPWJY]

Hello everyone,

I’m experiencing an issue with very high CPU usage on my FortiGate VM02. The CPU is mostly consumed by software interrupts (softIRQ), and it seems related to the use of software switches. Here’s a summary of the performance status:

# get sys performance status
CPU states: 1% user 9% system 0% nice 24% idle 0% iowait 5% irq 61% softirq
CPU0 states: 1% user 9% system 0% nice 22% idle 0% iowait 4% irq 64% softirq
CPU1 states: 2% user 10% system 0% nice 23% idle 0% iowait 6% irq 59% softirq
Memory: 8155856k total, 1970056k used (24.2%), 5548792k free (68.0%), 637008k freeable (7.8%)
Average network usage: 1205906 / 1225484 kbps in 1 minute, 1299972 / 1323049 kbps in 10 minutes, 1260383 / 1282380 kbps in 30 minutes
Maximal network usage: 1385365 / 1411865 kbps in 1 minute, 1584081 / 1608140 kbps in 10 minutes, 1725782 / 1755302 kbps in 30 minutes
Average sessions: 50420 sessions in 1 minute, 50219 sessions in 10 minutes, 50779 sessions in 30 minutes
Maximal sessions: 51110 sessions in 1 minute, 51574 sessions in 10 minutes, 53986 sessions in 30 minutes
Average session setup rate: 415 sessions per second in last 1 minute, 409 sessions per second in last 10 minutes, 404 sessions per second in last 30 minutes
Maximal session setup rate: 745 sessions per second in last 1 minute, 745 sessions per second in last 10 minutes, 745 sessions per second in last 30 minutes
Virus caught: 0 total in 1 minute
IPS attacks blocked: 0 total in 1 minute
Uptime: 2 days,  8 hours,  14 minutes

My FortiGate is configured with VXLAN over IPSEC, and I’m using multiple VDOMs. I suspect that the software switches are significantly increasing the CPU load. The network traffic is quite high, with around 50,000 active sessions.

I’m looking for suggestions on how to optimize the configuration and reduce CPU usage. Has anyone encountered similar issues or have any recommendations for better managing software switches in this setup?

Thanks in advance for your help!

[bkrishnan]

Hi,
Please refer to the below article which might help you;
https://community.fortinet.com/t5/FortiGate/Technical-Tip-Software-switch-causing-high-CPU-softirq-u...

[5q46n2te8jPWJY]

Thanks, I actually read the article, but I can't do what it says.

Are there any other tips?

Can implementing SR-IOV help?

Thanks

[ffuchs]

Hi,

the article say that you´ve configured a Software Switch on your Fortigate. If there is heavy load / lot of traffic, the CPU of your VM has to handle it. Or if there are too many Software Switches configured.

I dont know why SR-IOV should help. The article say it´s a VM issue not a Hypervisor Issue. So shouldn´t be helpful. 

So heavy load, many software switches and not enough CPU to handle it.

[Atul_S]

Hi There,

SoftIRQ resembles to a bottle neck situation in VM environment where most of the time, it has to something to do with the device capacity management. Please refer to the below link to see if this could help in your case. I am not sure what Hypervisor solution this VM is hosted on but its worth looking at:

>> Considering VM upgrade to a higher capacity

>>Enable accelerated networking in vm

https://community.fortinet.com/t5/FortiGate/Technical-Note-Optimize-FortiGate-VM-performance-by-conf...

Thanks,