Description
This article describes troubleshooting for the speed or bandwidth throttling issues over the Site-to-Site IPsec tunnel.
Scope
FortiGate and all FortiOS Platforms.
Solution
The best way to troubleshoot speed-related issues on the IPsec tunnel is to compare the bandwidth over WAN.
As IPsec packets travel in the form of ESP(Encapsulated Security Payload) packets that are sent over WAN.
Therefore, the maximum throughput one can get over IPsec depends upon how fast the resource can be reached from location A to location B over the WAN.
In this scenario, port1 is the WAN interface on both sites, and port2 is the LAN interface on both sites.
There is a need to have some specific requirements to measure bandwidth over WAN.
Firstly, there is a requirement to have an unused public IP space on Site B so that a VIP can be configured in order to open the port to any server where an IPerf server can be configured in the server mode.
This server will be listening on the port and secondly needs a corresponding policy with VIP as the destination to allow the traffic from site A FortiGate to perform the test.
Then, from the FortiGate on site A, run the Iperf commands (as FortiGate can only act in client mode )
diagnose traffictest client-intf port1 <- Define FortiGate port (WAN port).
diagnose traffictest server-intf port1 <- Define FortiGate port (WAN port).
diagnose traffictest port 5209 <- Define iPerf3 port running on the iPerf3 server.
diagnose traffictest run -c 1.2.3.4 <- Remote site B WAN IP.
Sample output:
[ 14] local 4.3.2.1 port 5201 connected to 1.2.3.4 port 5209
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 14] 0.00-1.01 sec 1.78 MBytes 14.8 Mbits/sec 2 198 KBytes
[ 14] 1.01-2.01 sec 3.56 MBytes 29.9 Mbits/sec 37 256 KBytes
[ 14] 2.01-3.01 sec 6.01 MBytes 50.4 Mbits/sec 0 304 KBytes
[ 14] 3.01-4.01 sec 6.73 MBytes 56.6 Mbits/sec 0 335 KBytes
[ 14] 4.01-5.01 sec 6.73 MBytes 56.4 Mbits/sec 0 354 KBytes
[ 14] 5.01-6.01 sec 6.78 MBytes 56.9 Mbits/sec 0 354 KBytes
[ 14] 6.01-7.01 sec 6.65 MBytes 55.8 Mbits/sec 0 363 KBytes
[ 14] 7.01-8.01 sec 6.77 MBytes 56.8 Mbits/sec 0 363 KBytes
[ 14] 8.01-9.01 sec 4.58 MBytes 38.4 Mbits/sec 5 187 KBytes
[ 14] 9.01-10.00 sec 6.07 MBytes 51.1 Mbits/sec 0 301 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 14] 0.00-10.00 sec 55.7 MBytes 46.7 Mbits/sec 44 sender
[ 14] 0.00-10.00 sec 55.5 MBytes 46.6 Mbits/sec receiver
iperf Done.
iperf3: interrupt - the server has terminated
After confirming the speed over the WAN, In order to confirm the same via the IPsec tunnel route, run the Iperf commands on Site A again with the private IP of the server which is reachable via IPsec.
diagnose traffictest client-intf port2 <- Define FortiGate port (this is the lan port for the subnet allowed over phase-2 selectors).
diagnose traffictest server-intf port2 <- Define FortiGate port (this is the lan port for the subnet allowed over phase-2 selectors).
diagnose traffictest port 5209 <- Define iPerf3 port running on the iPerf3 server.
diagnose traffictest run -c 192.168.10.2
[ 14] local 192.168.1.1 port 5201 connected to 192.168.10.2 port 5209
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 14] 0.00-1.01 sec 1.78 MBytes 14.8 Mbits/sec 2 198 KBytes
[ 14] 1.01-2.01 sec 3.56 MBytes 29.9 Mbits/sec 37 256 KBytes
[ 14] 2.01-3.01 sec 6.01 MBytes 50.4 Mbits/sec 0 304 KBytes
[ 14] 3.01-4.01 sec 6.73 MBytes 56.6 Mbits/sec 0 335 KBytes
[ 14] 4.01-5.01 sec 6.73 MBytes 56.4 Mbits/sec 0 354 KBytes
[ 14] 5.01-6.01 sec 6.78 MBytes 56.9 Mbits/sec 0 354 KBytes
[ 14] 6.01-7.01 sec 6.65 MBytes 55.8 Mbits/sec 0 363 KBytes
[ 14] 7.01-8.01 sec 6.77 MBytes 56.8 Mbits/sec 0 363 KBytes
[ 14] 8.01-9.01 sec 4.58 MBytes 38.4 Mbits/sec 5 187 KBytes
[ 14] 9.01-10.00 sec 6.07 MBytes 51.1 Mbits/sec 0 301 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 14] 0.00-10.00 sec 55.7 MBytes 42.7 Mbits/sec 44 sender
[ 14] 0.00-10.00 sec 55.5 MBytes 42.6 Mbits/sec receiver
iperf Done.
iperf3: interrupt - the server has terminated
If the difference is minimal between both tests, it can be concluded that the bandwidth difference is due to the fact that traffic is encrypted & decrypted over the tunnel. But if the difference is significantly noticeable, further troubleshooting should be undertaken with respect to the npu-offloading being enabled/disabled, drops on the NPU chips, and CPU/Memory utilization on the FortiGate.
In some cases, ISPs might be applying traffic shaping for ESP traffic. One solution is to switch to using UDP/port 4500 packets for Phase 2 by enabling Forced NAT-T on both sides and checking if there is any improvement in throughput via iPerf.
To check if NAT-T is being used, run the following commands:
diagnose vpn tunnel list
If LocalIP:0->PeerIP:0 on the first entry where the tunnel name is also contained, it means that the tunnel is not using NAT-T. Otherwise, if LocalIP:4500->PeerIP:4500, NAT-T is being used.
The latency between communicating endpoints should also be checked. With TCP there is a direct relationship between latency and throughput. As latency increases, TCP throughput drastically decreases. The ping command assists in quickly checking the round trip end-to-end latency.
Note: The VPN traffic can get affected if there is a high CPU issue stemming on one of the core CPU 0 core,s for example. Check if the proposals used in IPsec VPN configuration are supported by NP6 hardware accelerator. The GCM proposal is not supported by the NP6 processor. To reduce the load for management, use the following commands to dedicate core 0 using below commands:
config system npu
set dedicated-management-cpu enable
end
It is recommended to change the proposal to supported proposal that the NP6 can offload so that the CPU and vpn speed can recover to a better stable performance.
Related documents:
- NP6XLite: NP6XLite processors
- NP6 requirements: NP6 session fast path requirements
- NP7 requirements: NP7 session fast path requirements
- Site-to-site VPN
- Site-to-site IPsec VPN with two FortiGates
