|
At certain IPsec VPN bandwidth usage and concurrent super flows, it is observed that the IPsec Tx/Rx error counters increment on the SoC3 and SoC4 FortiGates. This increment is particularly noticeable when the IPsec tunnel phase1 npu-offload NPUs are disabled.
FortiGate # fnsysctl ifconfig 'VPN ToSpoke1'
VPN ToSpoke1 Link encap:Unknown
inet addr:172.26.254.81 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1300 Metric:1
RX packets:721261 errors:924 dropped:0 overruns:0 frame:0
TX packets:668867 errors:104 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:185694792 (177.1 MB) TX bytes:117167132 (111.7 MB)
FortiGate # fnsysctl ifconfig 'VPN ToSpoke1'
VPN ToSpoke1 Link encap:Unknown
inet addr:172.26.254.81 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1300 Metric:1
RX packets:2376420 errors:14735 dropped:0 overruns:0 frame:0
TX packets:2202102 errors:299 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1035748360 (987.8 MB) TX bytes:243505877 (232.2 MB)
Additionally, the CP q_full counters may be observed increasing in the following output, indicating that the Content Processors(CP) are operating at near full capacity.
FortiGate # diagnose cp soc4 vpn-stats 0
pq vq ipsec_enc ipsec_dec ssl_enc ssl_dec qfull nomem min_len clen_z clen_e
offset_e offset_z no_proc pci_abort auh_f pad_e key_e key_abort
--- --- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
0 0 2779110579 3943675793 89 54 87501696 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 1 3434650208 3858183367 85 58 88646322 0 0 0 0
0 1 0 0 0 0 0 0 0 0
0 2 3156079253 4068556350 91 52 89396117 0 0 0 0
0 2 0 0 0 0 0 0 0 0
0 3 2467766013 3777267200 84 59 82894867 0 0 0 0
0 3 0 0 0 0 0 0 0 0
1 0 3881708492 3789696286 92 50 90067405 0 0 0 0
1 0 0 0 0 0 0 0 0 0
1 1 2720110272 3709654310 76 66 83096896 0 0 0 0
1 1 0 0 0 0 0 0 0 0
1 2 2949279999 3929719322 92 50 85826582 0 0 0 0
1 2 0 0 0 0 0 0 0 0
1 3 3063769112 4029068045 87 55 87087363 0 0 0 0
1 3 0 0 0 0 0 0 0 0
--- --- ---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
err_irq: 0
Workaround:
Disable CP offload on FortiGate:
config system global
set ipsec-asic-offload disable
set ipsec-hmac-offload disable
end
The above options are used to control if the CP processes the IPsec packets instead of the CPU.
General debug information required by FortiGate TAC for investigation:
- Multiple iterations of the following debug:
diagnose npu np6lite dce 0
diagnose npu np6lite anomaly-drop
fnsysctl cat /proc/net/np6lite_0/hif-stats
fnsysctl cat /proc/net/np6lite_0/hifdrop
fnsysctl cat /proc/net/np6lite_0/osw
fnsysctl cat /proc/net/np6lite_0/fos-perf
fnsysctl cat /proc/softirqs
fnsysctl cat /proc/interrupts
diag vpn ipsec status
fnsysctl ifconfig <intf> <----- Replace <intf> to the IPsec phase1 interface.
get sys performance status
fnsysctl cat /proc/net/np6lite_0/mibs/gigeN <----- Replace N with values from 0 to 15 respectively.
diagnose vpn ipsec cpu
diagnose cp soc3 vpn-stats 0
diagnose cp soc4 vpn-stats 0
diagnose sys vd list | grep fib
diagnose sys bcm_intf cli 'show c'
- TAC Report:
execute tac report
- Configuration file of the FortiGate.
Solution:
Upgrade to v7.2.11, v7.4.8, or v7.6.3 as this has been resolved.
Note:
- The workaround is not necessary in version 7.2.11.
- Super Admin privilege is required to run the 'fnsysctl' command. Otherwise, FortiGate will return an error, as explained in Troubleshooting Tip: fnsysctl command returns Unknown action 0.
-
To configure npu-offload settings from the CLI, see the article Disabling NP offloading for individual IPsec VPN phase 1s, and to view the configuration of the VPN Tunnel from the CLI, execute the command show vpn ipsec phase1-interface.
|
