Technical Tip: After upgrade to v7.0.14 and later, some FortiGate models may show incrementing TX and RX errors on IPsec interface

[Matt_B]

Description	This article describes a known change that causes TX and RX error counters to increase when an IPsec tunnel is not offloaded to NPU and FortiGate is handling bursts of IPsec traffic.
Scope	FortiGate v7.0.14, v7.2.8, v7.4.2 and later, Models using SOC4 and CP9XLITE such as FortiGate-40F, 60F, FortiGate-100F, IPsec tunnel not fully offloaded to NP6XLITE.
Solution	In  v7.0.14 and later, when IPsec encryption and decryption are handled by a Content Processor (CP) chip instead of NPU, TX and RX errors may increment on the IPsec tunnel interface, particularly if the device is operating near capacity. FGT-101F # fnsysctl ifconfig VPN-1 VPN-1 Link encap:Unknown UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1420 Metric:1 RX packets:123373532972 errors:1045311682 dropped:0 overruns:0 frame:0 TX packets:305161540153 errors:329422761 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7034817854543 (6551.7 GB) TX bytes:426827387197342 (397513.10 GB) FGT-101F # fnsysctl ifconfig VPN-1 VPN-1 Link encap:Unknown UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1420 Metric:1 RX packets:123375402638 errors:1045328594 dropped:0 overruns:0 frame:0 TX packets:305165390968 errors:329425232 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7035043998197 (6551.9 GB) TX bytes:426832376013781 (397518.6 GB) This is an expected change and does not indicate a performance drop, since before the change the same traffic triggers many less visible errors while still experiencing packet loss. There is an optimization in known issue 1054440 to further reduce TX and RX errors and drops by optimizing CP queue size, see FortiOS v7.2.11 Release Notes. This optimization is present in v7.2.11 and scheduled for v7.4.8 and v7.6.3. Detail: When an IPsec SA is offloaded to a Network Processing Unit (NPU), Content Processors (CPs) do not handle IPsec encryption and decryption. Instead, the NPU handles all eligible IPsec data encryption and decryption. config vpn ipsec phase1-interface edit "VPN-1" set npu-offload enable <-- default setting. next end When NPU offload is disabled or not available for an IPsec SA, the IPsec data packet is handled by the CPU but encryption and decryption are offloaded to the CP by default. NPU offload can be disabled in phase1-configuration, and is not available for certain interfaces, see 'Technical Tip: Interface not supported by NPU Offload'. In rare error cases, an IPsec SA may still fail to install in NPU even if the eligible for offloading, in which case the traffic continues to be handled by CPU and Content Processor. NP offload disabled: config vpn ipsec phase1-interface edit "VPN-1" set npu-offload disable next end CP offload enabled: config system global set ipsec-asic-offload enable <-- default setting. set ipsec-hmac-offload enable <-- default setting. end To verify if a tunnel is currently offloaded to NPU, see the article 'Technical Tip: Ensuring IPSec traffic is offloaded for improved throughput'. If 'diagnose vpn tunnel list' shows 'npu_flag=03', encryption and decryption are both offloaded for the tunnel. FGT-101F # diagnose vpn tunnel list list all ipsec tunnel in vd 0 ------------------------------------------------------ name=example ver=1 serial=1 10.255.200.2:0->10.255.200.1:0 tun_id=10.255.200.1 tun_id6=::10.255.200.1 dst_mtu=1500 dpd-link=on weight=0 bound_if=6 lgwy=static/1 tun=tunnel/255 mode=auto/1 encap=none/8 options[0008]=npu run_state=0 ... npu_flag=03 npu_rgwy=10.255.200.1 npu_lgwy=10.255.200.2 npu_selid=0 dec_npuid=1 enc_npuid=1 run_tally=0 Before v7.0.14 resolved the issue 897867 for SOC4 models, devices with NPU offload disabled for an IPsec tunnel would incorrectly pass encryption and decryption through to the CPU if CP was busy, leading to increased DCE and HIF drops. These drops represent packet loss but do not increment the tunnel TX and RX error counters. DCE and HIF drops for SOC4 units can be checked with 'fnsysctl cat /proc/net/np6xlite_0/hifdrop' and 'diagnose npu np6xlite dce 0'. These commands should be taken multiple times. Note that reading these counters also clears them. FGT-101F # fnsysctl cat /proc/net/np6xlite_0/hifdrop VHIF_RX0_DROP :0000000000002413 VHIF_RX1_DROP :0000000000000165 VHIF_RX2_DROP :0000000000000394 FGT-101F # diagnose npu np6xlite dce 0 DROP_PDQ_OSW_HRX0:0000000000002972[ae] FGT-101F # fnsysctl cat /proc/net/np6xlite_0/hifdrop VHIF_RX0_DROP :0000000000000014 FGT-101F # diagnose npu np6xlite dce 0 DROP_PDQ_OSW_HRX0:0000000000000014[ae] FGT-101F # fnsysctl cat /proc/net/np6xlite_0/hifdrop VHIF_RX0_DROP :0000000000000039 FGT-101F # diagnose npu np6xlite dce 0 DROP_PDQ_OSW_HRX0:0000000000000039[ae] Related document: FortiOS v7.0.14 Release Notes: Resolved Issues