IPSec VPN Issue

[Daryaya]

Hi, I’m running into a strange issue with an IPsec tunnel to a Cisco firewall and I’m hoping someone can help.

The tunnel comes up fine and traffic does pass through, but the connection slows down significantly. For example:

• If I ping 8.8.8.8 directly from an end-user machine, the replies are just a few ms.
• If I ping a remote address over the tunnel, the latency jumps to about a second.
• Even pinging the remote VPN gateway itself shows very high delay. As soon as I bring the tunnel down, latency immediately improves to under 10 ms.

Setup details:

• Both the Internet and VPN zones share the same physical interface on the firewall (different VLANs) and use the same ISP.
• I don’t believe the issue is on the remote side, because if I establish same tunnel to the same remote gateway from a different firewall, it works normally.

So far, I’ve tried flushing the tunnel on both ends, but that didn’t help.

Does anyone know which commands I could run to troubleshoot this further, or what else I should be checking?

Thanks

[Atul_S]

Hi Daryaya,

It's worth checking the MSS size so as to isolate if fragmentation is causing the delay. Have you tried disabling NPU and test? Pls help review the below docs for your reference:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-VPN-ESP-traffic-dropped-due-to-NP6-PBA-lea...

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-NP6-Out-of-order-packets/ta-p/312315

https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-troubleshoot-speed-or-bandwid... 

[Daryaya]

Thank you for your response. I have disabled NPU offload, but this did not improve the latency. I considered packet fragmentation as a possible cause and tested using ICMP with the DF bit set and a reduced MTU; however, the delay remained consistently high. Regarding running below debugs, could you please advise what they might prove?

diagnose npu np6 port-list

diag npu np6 dce <id>

diag npu np6 pdq <id>

diag npu np6 register <id>

diag npu np6 sse-stats <id>

[Atul_S]

Hi Daryaya,

If the issue still remains after adjusting MSS and disablng NPU, pls consider retest with TCP-MSS clamping applied and perform a single-stream iPerf test. If possible, the testing should be done using Linux clients/servers to avoid potential discrepancies introduced by Windows iPerf auto-window handling. 

Configure server-client set up for the iperf test as below:

At Fortigate side:

-----------------

Step 1:

-Visit the link https://iperf.fr/iperf-download.php and download iperf software at the destination server

-Now locate the directory path(something like c:\users\name\onedrive\desktop\name) where you have saved the downloaded iperf files(I would recommend to place the files inside a folder) and copy the path.

-Go to command prompt and run the following:

cd <directory path> iperf3.exe -s    //hit enter. 

At Cisco side :

--------------

Step 1:

-Visit the link https://iperf.fr/iperf-download.php and download iperf software at the source machine

-Now, locate the directory path(something like c:\users\name\onedrive\desktop\name) where you have saved the downloaded iperf files(I would recommend to place the files inside a folder) and copy the path.

-Go to command prompt and run the following:

cd <directory path> iperf3.exe -c   x.x.x.x  //hit enter. x.x.x.x is the destination server IP address at FGT.

cd <directory path> iperf3.exe -c  x.x.x.x -R  //hit enter. 

Step 4: On Fortigate

a)

# diag traffictest server-intf <tunnel interface>        <----- Define server port.

# diag traffictest client-intf port1        <----- Define the port from where source is coming

# diag traffictest run                      <----- Run iPerf3.

b)

# diag traffictest server-intf <tunnel interface>

# diag traffictest client-intf <tunnel interface>

# diag traffictest port 5209

# diag traffictest run -c  y.y.y.y     //where y.y.y.y is the src IP behind Cisco

# diagnose traffictest run -R -c  y.y.y.y -p 5209

# diagnose traffictest run -c  y.y.y.y -u

For your query related to the NP commands, below are the explaination:

diagnose npu np6 port-list- This command shows the list of ports on the NP6 processor. It can help verify if the correct ports are being used and if there are any configuration issues.

diag npu np6 dce <id>- This command displays the number of dropped packets for the selected NP6 processor. It can help identify if packet drops are contributing to the latency issue.

diag npu np6 pdq <id>- This command shows packet buffer queue counters. It can help determine if there are any queuing issues that might be causing delays.

diag npu np6 register <id>- This command shows NP6 registers. It is typically used for low-level diagnostics and might be more useful for Fortinet support or developers to identify hardware-related issues.

diag npu np6 sse-stats <id>- This command shows hardware session statistics counters. It can help identify session-related issues, such as session offloading problems, which might be affecting performance.

All the above commands helps identify any botleneck situation for the traffic flow.