Technical Tip: The ipsec-round-robin setting on FortiGate has been deprecated (and what to replace it with)

[pjang]

Description	This article describes the ipsec-round-robin config option found under config system global (not to be confused with round-robin under config system affinity-packet-redistribution), its deprecation from FortiOS, and what should be used instead for spreading out CPU utilization on FortiGates handling IPsec without hardware-acceleration (such as FortiGate-VMs).
Scope	FortiGate, IPsec
Solution	As a quick primer: when IPsec traffic is incoming on a FortiGate without hardware-acceleration (such as FortiGate-VM), there are two main workflows that the CPU must handle: Receiving incoming packets on the network interface (e.g., the receive/RX queue, which generates softIRQ usage), and Decrypting the IPsec traffic. One major concern is that from a network interface perspective, a single IPsec tunnel represents just one network flow/tuple (e.g., one Source sending to one Destination, using a given Protocol/Port). Because it appears to be one network flow (even though it may tunnel many user traffic sessions within that one flow), that traffic is all processed by a single interface RX queue, and therefore only a single CPU core is handling that incoming traffic. This becomes a problem as the amount of user traffic increases on that IPsec tunnel, as it leads to a single CPU Core becoming overburdened as it processes and decrypts the incoming IPsec traffic. Important: the information in this article primarily applies to FortiGates that either have no hardware-acceleration capability for IPsec traffic (like FortiGate-VMs) or that have disabled hardware-acceleration for some reason. Devices with Network Processors and/or Content Processors are able to offload packet handling and IPsec encrypt/decrypt operations away from the CPU, which lowers CPU utilization substantially and improves traffic/IPsec throughput. Over time there have been several attempts to address this in FortiOS, typically by enabling some kind of Receive Packet Steering (RPS) that allows incoming IPsec traffic to be spread across multiple CPU cores, which reduces the burden on any single core and increases overall performance by scaling across cores. Here are some of the options that are/were available, as well as if/when they have been deprecated (as of the time of this writing). ipsec-round-robin (along with packet-redistribution under config vpn ipsec phase1-interface) was first introduced in FortiOS 7.0.8 via Change #789811 as a means of enabling Receive Packet Steering (RPS) for incoming IPsec traffic specifically. See also: IPsec support for round robin and RPS distribution This solution has had mixed performance results and notably did not work if incoming traffic was not directly IPsec (for example, IPsec encapsulated within PPPoE or GRE). As of FortiOS 7.4.5, 7.6.1, and later this config option has no effect whatsoever since changes/optimizations have been made to how RPS is handled in FortiOS, and as such, this option has been deprecated as of FortiOS 7.6.1 and later. ipsec-soft-dec-async was introduced in FortiOS 6.0.1 via Change #446684 with the purpose of allowing the FortiGate to scale IPsec decryption work across multiple CPU cores while still keeping packets handled by a single CPU core. This was designed to ensure that packets do not become out-of-order, which is a possibility with the other mentioned solutions, while still allowing some of the workload to be spread across CPU cores. This solution has also had a mixed impact when it comes to improving IPsec throughput, and so it has been deprecated as of FortiOS 7.4.2 and later. For more historical info on this option, refer to the following KB article: Technical Tip: How to distribute IPsec traffic to all CPU cores for decryption The options under config system affinity-packet-redistribution (specifically affinity-cpumask and round-robin) are currently the recommended methods to use when enabling RPS on the FortiGate (aka, spreading load for an incoming single IPsec tunnel across multiple CPU cores for packet-handling and decryption). The affinity-packet-redistribution section was added in FortiOS 6.4.8 via Change #696412, and round-robin was added later as of FortiOS 7.4.0 via Change #868801. These options fully replace ipsec-round-robin and should be used instead (disable ipsec-round-robin, enable round-robin instead). Refer to this KB article for in-depth information on this option and how it can be configured: Technical Tip: affinity-packet-redistribution when using single or multiple IPSec tunnels on FortiGa... Interestingly, ipsec-round-robin also required the affinity-packet-redistribution section to be enabled (but did not require the similarly-named packet-redistribution option under config vpn ipsec phase1-interface), which often led to misconfiguration due to the lack of clarity in documentation. CLI Locations of Above Commands: config system global set ipsec-round-robin [ enable | disable ] <----- No function in 7.4.5/7.6.1+, deprecated in 7.6.1+. set ipsec-soft-dec-async [ enable | disable ] <----- Deprecated in 7.4.2+. end config system affinity-packet-redistribution set affinity-cpumask {string} set round-robin [ enable | disable ] end