Technical Tip: IPSec VPN Tunnel Interface Tx/Rx Error Counters Incrementing on FortiGate

[vpalli]

Description	This article describes an issue where IPSec VPN Tunnel Interface Tx/Rx error counters increment on FortiGate after the firmware upgrade to v7.0.14/v7.2.5 and above.
Scope	FortiGate.
Solution	At certain IPSec VPN bandwidth usage and concurrent super flows, it is observed that the IPSec Tx/Rx error counters increment on the SoC3 and SoC4 FortiGates. This increment is particularly noticeable when the IPSec tunnel phase1 npu-offload setting is disabled: FortiGate # fnsysctl ifconfig 'VPN ToSpoke1' VPN ToSpoke1 Link encap:Unknown inet addr:172.26.254.81 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1300 Metric:1 RX packets:721261 errors:924 dropped:0 overruns:0 frame:0 TX packets:668867 errors:104 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:185694792 (177.1 MB) TX bytes:117167132 (111.7 MB) FortiGate # fnsysctl ifconfig 'VPN ToSpoke1' VPN ToSpoke1 Link encap:Unknown inet addr:172.26.254.81 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1300 Metric:1 RX packets:2376420 errors:14735 dropped:0 overruns:0 frame:0 TX packets:2202102 errors:299 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1035748360 (987.8 MB) TX bytes:243505877 (232.2 MB) Additionally, the CP q_full counters may be observed increasing in the following output, indicating that the Content Processors(CP) are operating at near full capacity. FortiGate # diagnose cp soc4 vpn-stats 0 pq vq ipsec_enc ipsec_dec ssl_enc ssl_dec qfull nomem min_len clen_z clen_e offset_e offset_z no_proc pci_abort auh_f pad_e key_e key_abort --- --- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- 0 0 2779110579 3943675793 89 54 87501696 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3434650208 3858183367 85 58 88646322 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 2 3156079253 4068556350 91 52 89396117 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 3 2467766013 3777267200 84 59 82894867 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1 0 3881708492 3789696286 92 50 90067405 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 2720110272 3709654310 76 66 83096896 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 2 2949279999 3929719322 92 50 85826582 0 0 0 0 1 2 0 0 0 0 0 0 0 0 1 3 3063769112 4029068045 87 55 87087363 0 0 0 0 1 3 0 0 0 0 0 0 0 0 --- --- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------- err_irq: 0 This issue is currently being investigated by the development team, with further optimizations planned to prevent Tx/Rx errors. The article will be updated with the latest information once a fix is available. Workaround. Disable CP offload on FortiGate: config system global     set ipsec-asic-offload disable     set ipsec-hmac-offload disable end The above options are used to control if the CP processes the IPSec packets instead of the CPU. General debug information required by FortiGate TAC for investigation: Multiple iterations of the following debug: diagnose npu np6lite dce 0 diagnose npu np6lite anomaly-drop fnsysctl cat /proc/net/np6lite_0/hif-stats fnsysctl cat /proc/net/np6lite_0/hifdrop fnsysctl cat /proc/net/np6lite_0/osw fnsysctl cat /proc/net/np6lite_0/fos-perf fnsysctl cat /proc/softirqs fnsysctl cat /proc/interrupts diag vpn ipsec status fnsysctl ifconfig <intf>  <----- Replace <intf> to the IPsec phase1 interface. get sys performance status fnsysctl cat /proc/net/np6lite_0/mibs/gigeN  <----- Replace N with values from 0 to 15 respectively. diagnose vpn ipsec cpu diagnose cp soc3 vpn-stats 0 diagnose cp soc4 vpn-stats 0 diagnose sys vd list | grep fib diagnose sys bcm_intf cli 'show c' TAC Report: execute tac report Configuration file of the FortiGate.