Technical Tip: FortiGate NPU does not hardware-offload IPsec tunnels terminated on NPU VDOM links (Known Issue)

[pjang]

Description	This article describes a known issue that occurs with hardware-offloading when IPsec VPN tunnels are terminated/listening on NPU VDOM links (also known as npu_vlinks). This issue will also occur if the IPsec tunnel is listening on a VLAN sub-interface that is configured on top of an npu_vlink.
Scope	FortiGate, NPU Hardware-Acceleration, IPsec.
Solution	When FortiGates are configured with Virtual Domains (VDOMs), it is common to utilize NPU VDOM links to interconnect the VDOMs, as it allows for traffic to be hardware-accelerated as it passes from one VDOM to another. Refer to the following KB article for more information on VDOM links in-general: Technical Tip: Difference and understanding between NPU Vdom link, NPU Vdom link with VLAN and Vdom .... As part of this design, administrators may consider configuring their VPN tunnels to listen/terminate on these NPU VDOM links. The following is an example of what this topology might look like: However, there is a known issue that occurs when IPsec VPN tunnels are configured to listen on NPU VDOM links or associated VLANs (for example, the issue would occur in the above topology if the VPN tunnel was set to listen on vlan100 in the root VDOM). In this configuration, the VPN tunnel will appear to be fully offloaded to the NPU, but in actuality, it will be handled entirely by the CPU. Important Note: This issue specifically occurs due to an issue with how the NPU handles VPN tunnels using UDP-encapsulated ESP traffic (aka UESP and UDP/4500), and this situation only occurs when NAT-Traversal is utilized. However, configurations that utilize IPsec tunnels terminated on NPU VDOM links are generally assigning private IP addresses to the NPU VDOM link pairs, and so NAT-Traversal becomes a nearly-universal requirement. The only way to avoid this issue would be to assign routable public IP addresses to the NPU VDOM links so NAT is not performed in-between the NPU VDOM link and the remote VPN peer. With the above in mind, the following symptoms may be observed as a result: When running diagnose vpn tunnel list name <VPN_Name>, the npu_flag line will show 03, which suggests that the VPN tunnel is being offloaded in both the encrypt AND decrypt directions. However, under heavy-load (such as a large network file-transfer or data backup operation), CPU usage on a single CPU core may increase as high as 99% in the softIRQ usage category when observed with get system performance status or diagnose sys mpstat. CPU load should generally be at 0% when the NPU is correctly offloading IPsec traffic. Additionally, the receive-side error counter (RXE) for the IPsec tunnel interface may increase significantly if softIRQ CPU usage is extremely high (as checked with diagnose netlink interface list <VPN_Name>), as it indicates that incoming VPN tunnel traffic is overwhelming the capabilities of the CPU. Peak VPN throughput may also be much lower than expected, depending on the FortiGate model and the CPU resources available (for example, VPN throughput might be capped at 500 Mbps instead of ~1 Gbps). Finally, when performing packet captures of ESP (IP Proto 50) or UDP/4500 IPsec traffic, it is possible to see both the outgoing AND incoming traffic (generally, it is only possible to see outgoing traffic when IPsec hardware-offload is taking place). At the time of this writing, this issue has been confirmed to occur for NP7 and NP6XLite (aka SOC4) equipped FortiGates, though it is likely that other NPU platforms (NP6lite, NP6, NP7lite) are affected. Additionally, given that this is a byproduct of the NPU hardware design, it cannot be corrected in software/firmware, and so the issue cannot be resolved for now. Workaround:  The current recommendation is to avoid FortiGate VDOM designs where VPN Tunnels are terminated on npu_vlink interfaces or VLANs associated with npu_vlinks. Consider some of these alternative designs that still allow for hardware-offload of IPsec traffic: Terminate the IPsec tunnel on a loopback interface (NP7/NP7lite only, see: NP7 session fast path requirements). Move the IPsec tunnels to a VDOM that has direct Internet access and have the tunnels listen on the WAN interface instead. Redesign the network interface assignments to provide direct Internet access to VDOMs that otherwise relied on NPU VDOM links to reach the Internet. Additional Notes: Keep in mind that this is an issue specific to hardware-offloading of UDP-encapsulated ESP IPsec traffic, and so software/CPU-based handling of IPsec traffic will behave as intended. For devices that do not have hardware-acceleration (such as FortiGate-VMs and some smaller models of FortiGates), this issue also does not apply. NPU VDOM links can still be used to accelerate traffic passing between the npu_vlink pairs (including IPsec) in-general. This issue only affects IPsec traffic for tunnels that are terminated on the FortiGate and on NPU VDOM links/VLANs.