Uneven CPU Load Distribution on FortiGate VM02 Across Clusters

[5q46n2te8jPWJY]

Hello Fortinet Community,

I am experiencing an issue with CPU utilization on one of my FortiGate VM02 instances. I have two clusters, both configured similarly, but their CPU load behavior differs significantly.

1. Cluster 1 (Normal Behavior):
   The CPU load is distributed evenly across both cores during IPsec operations, as shown in the output of the # diagnose vpn ipsec cpu command:

# diagnose vpn ipsec cpu
Software crypto CPU distributions:
CPU#            enc             dec-in          dec             dec-out         
0               18524077877     0               7856389447      7856389447      
1               18703560081     0               198249897       198249897

• Cluster 2 (Problematic Behavior):
  On the second cluster, the load is uneven and concentrated on a single core, which is causing performance concerns. Below is the output for the same command:

# diagnose vpn ipsec cpu
Software crypto CPU distributions:
CPU#            enc             dec-in          dec             dec-out         
0               1334517         0               0               0               
1               2208255         0               11464826        11464826

The uneven load on Cluster 2 results in one core being underutilized while the other handles all the decryption tasks. This behavior seems suboptimal compared to Cluster 1.

---

Additional Information:

• FortiOS Version: 7.4
• Environment: VM02 running on Xen
• Configuration Parity: Both clusters are set up identically in terms of IPsec settings, general configurations and hypervisor configurations.

---

My Questions:

1. What could be causing the uneven CPU load distribution on Cluster 2?
2. Are there any specific configurations or settings I should verify to ensure proper load balancing across cores?

Thank you in advance for your assistance!

[alafrance]

IPSEC Load distribution is a bit of a complex topic to discuss without knowing all the variables in play like the traffic patterns and what tunnels they go into and the amount of tunnels , so this response will be general information to help frame things better.

In your working cluster output you have the encryption that's generally well balanced but on your decryption its unbalanced , which is by default a behavior I would expect.

To expand further on what I mean , we have to understand how packets are distributed to CPU cores when traffic comes in from the NIC or virtual-nic in this case.

Primarily this is hash based on the L3-L4 headers of the ingress packet depending on the protocol of the packet received. The general result is that packets from a session in one direction are generally expected to hit the same CPU core and that CPU handles the traffic.

When IPSEC and more specifically the ESP protocol which transports the encrypted tunneled
traffic enters a Fortigate , it's a single session transporting many sessions inside of it.

Depending on what's going in that tunnel , it can result in a high-packet rate ESP session that gets distributed to only one CPU core by default. Different tunnels may hit different CPU cores but one tunnel will always hit the same CPU by default.

This behavior on decryption _can_ be modified via the settings here , but its not known to generally increase performance , rather it helps take pressure off when a single CPU is being overwhelmed with packet decryption tasks due to a very busy tunnel. See the article below:

Technical Tip: How to distribute IPsec traffic to all CPU cores for decryption  

Now the above is true for the decryption direction generally (e.g. traffic inbound from the tunnel). For traffic that came in on a port and is egressing out of the IPSEC tunnel it's a bit different story.

In that scenario the same logic applies , however since this is the unencrypted/unencapsulated traffic that's not hidden behind an ESP header. Multiple sessions can hit different CPU cores and those cores will also handle encrypting/encapsulating traffic so the load distribution is typically better.

In your last output where its not even , there's two things I notice.

1. CPU#1 handles all decryption tasks.

This may mean theres one or few IPSEC tunnels actually in use , but that's a guess based on what I can glean.

2. Uneven distribution on encryption

This can be seen as a general packet counter , and as mentionned the packet distribution can be generally considered to be session-based in its ultimate results. The sessions going into the tunnel could be generally well distributed , but the packets sent by each may vary wildly which can influence the counter to look unbalanced.

In the end it may be best to open a ticket , since digging into this sometimes requires more details that I doubt many people want to share openly about their environment here.

[5q46n2te8jPWJY]

Thank you, I looked at the article, unfortunately the command does not seem to be available for me (VM02 FortiOS 7.4)

[alafrance]

Hi There,

My apologies , you are right that this setting is gone , it seems it was removed in 7.4.2 and sounds like we need to update the KB with this information.

There is a second method outlined here: IPsec support for round robin and RPS distribution that could be worth a try.

This is just a general recommendation however , as I alluded to earlier there could be some environment specific details that explain the differences in results between your two clusters. If that's the case it may be better to open a support ticket.