Uneven CPU Load Distribution on FortiGate VM02 Across Clusters

IPSEC Load distribution is a bit of a complex topic to discuss without knowing all the variables in play like the traffic patterns and what tunnels they go into and the amount of tunnels , so this response will be general information to help frame things better.

In your working cluster output you have the encryption that's generally well balanced but on your decryption its unbalanced , which is by default a behavior I would expect.

To expand further on what I mean , we have to understand how packets are distributed to CPU cores when traffic comes in from the NIC or virtual-nic in this case.

Primarily this is hash based on the L3-L4 headers of the ingress packet depending on the protocol of the packet received. The general result is that packets from a session in one direction are generally expected to hit the same CPU core and that CPU handles the traffic.

When IPSEC and more specifically the ESP protocol which transports the encrypted tunneled
traffic enters a Fortigate , it's a single session transporting many sessions inside of it.

Depending on what's going in that tunnel , it can result in a high-packet rate ESP session that gets distributed to only one CPU core by default. Different tunnels may hit different CPU cores but one tunnel will always hit the same CPU by default.

This behavior on decryption _can_ be modified via the settings here , but its not known to generally increase performance , rather it helps take pressure off when a single CPU is being overwhelmed with packet decryption tasks due to a very busy tunnel. See the article below:

Technical Tip: How to distribute IPsec traffic to all CPU cores for decryption  

Now the above is true for the decryption direction generally (e.g. traffic inbound from the tunnel). For traffic that came in on a port and is egressing out of the IPSEC tunnel it's a bit different story.

In that scenario the same logic applies , however since this is the unencrypted/unencapsulated traffic that's not hidden behind an ESP header. Multiple sessions can hit different CPU cores and those cores will also handle encrypting/encapsulating traffic so the load distribution is typically better.

In your last output where its not even , there's two things I notice.

1. CPU#1 handles all decryption tasks.

This may mean theres one or few IPSEC tunnels actually in use , but that's a guess based on what I can glean.

2. Uneven distribution on encryption

This can be seen as a general packet counter , and as mentionned the packet distribution can be generally considered to be session-based in its ultimate results. The sessions going into the tunnel could be generally well distributed , but the packets sent by each may vary wildly which can influence the counter to look unbalanced.

In the end it may be best to open a ticket , since digging into this sometimes requires more details that I doubt many people want to share openly about their environment here.