New Member

Question

Site to site IPsec traffic very slow

Forum|Forum|4 years ago
November 23, 2021
11 replies
60571 views

Hi everyone !

I'm facing a really strange problem with IPSec VPN. I configured IPSec tunnel FortiGate to FortiGate on different models (40F - 80F and 100F) all of my VPN tunnels are slow and they not reflecting my bandwidth throughput. I'm on FortiOs 7.0.1

For exemple I have :
- FortiGate-80F <-> FortiGate-80F with a bandwidth of 1Gb/1Gb on both sites. Trough my tunnel, I reach with difficulties about 200Mb/200Mb

- FortiGate-100F <-> FortiGate-80F with a bandwidth of 1Gb/1Gb on 100F sites and a bandwidth of 500Mb/500mb. Trough my tunnel, I reach with difficulties about 200Mb/200Mb

- FortiGate-100F - Fortigate-100F with a bandwidth of 500Mb/500Mb on both sites. Trough my tunnel, I reach with difficulties about 200Mb/200Mb

I tried many options to optimise my tunnel but nothing woks. I tried :
- Set correct MTU on WAN Interface and MSS on Firewall rules (for exemple MTU of 1500 and MSS of 1380) --> no change
- Set different encryptions on my tunnels --> no change
- Disabled ipsec-asic and ipsec-hmac --> no change

This slowness on IPSec seems to be the same on every models and on very configurations... Here is for exemple one of my phase1 config

config ipsec phase1-interface
edit "vpn"
set interface "wan1"
set ike-version 2
set local-gw 1.2.3.4
set keylife 28800
set peertype any
set net-device disable
set proposal aes128-sha256 aes256-sha256 aes128gcm-prfsha256 aes256gcm-prfsha384 chacha20poly1305-prfsha256
set dhgrp 19 20
set nattraversal forced
set remote-gw 4.3.2.1
set add-gw-route enable
set psksecret Secret
next
end

I really need your help. I don't understand what I'v missed in configuration.

Thanks !

FortiGate

Golle

New Member

Why is nattraversal forced? Let it automatically figure out if NAT is needed or not, or better yet disable it if you know both sites are directly connected to the internet via public IP-addresses.

Also, what's the latency between your sites? Latency has a big impact on TCP performance, and they further away your sites are the worse single-session TCP will perform. Splitting up the workload into multiple TCP sessions is often necessary to get higher bandwidths over long distances.

Also, how are you testing the bandwidth over the tunnel? I personally like to use iperf3 since it allows me to test network throughput without having to rely on factors like harddrives or file transfers. It also allows me to run several sessions in parallel to find the true maximum throughput between two sites.

Sending your traffic with a higher TCP Window size should also improve throughput assuming there is no loss on the link.

latos1224

New Member

Hello Golle, I showed my client the ping response of 7ms (a very low value in itself), however, they have other clients and they told me that one of those clients has a Forti 40F in the DC and has 3 triangulated locations and the ping between the locations and the DC is 1ms (they have an ERP in the DC and computers at the client's location) and the ping between locations and DC is 1ms.

zoriaxAuthor

New Member

Hi Golle,

Thanks for your return. Nat-t was set to force because Forti Support said to me it will be better to have it forced. But I can disable it because my router is directly connected to internet.

The latency between two sites (for exemple between Foti-100F) is around 22ms. For me it's low.

With iPerf an 10 parallel client threads I can reach the max : [SUM] 9.00-10.00 sec 53.0 MBytes 445 Mbits/sec

So... seems to be related to SMB

Any idea how to optimize this protocole trough VPN IPsec ?

Stelios_FTNT

Staff

Hi Zoriax,
What about the MTU used on all your equipment between the client and the FortiGate but also between the FortiGate and the server? Are you sure you're not using Jumbo frames on these segments, when as you mention, you have an MTU of 1500 bytes on the interfaces of the FortiGate?

For SMB transfers we often see jumbo frame configurations on all the equipment of the path and of course if you haven't done that on your firewall, maybe this is the reason the performance decreases no matter which firewall you have in the middle of the path.

If this is the root cause of your issue, you need to override the MTU of your incoming and outgoing interfaces and verify that the MTU of the tunnel is also increased with the use of the following debug:

# diagnose vpn tunnel list
list all ipsec tunnel in vd 4
------------------------------------------------------
name=IPsec_name ver=2 serial=8 10.1.1.1:0->10.2.2.2:0 dst_mtu=9100
bound_if=40 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc accept_traffic=1 overlay_id=1
proxyid_num=1 child_num=1 refcnt=311 ilast=0 olast=0 ad=r/2
stat: rxp=1397676 txp=1201995507 rxb=502387654 txb=680787953
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=2
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=IPsec_name_0 proto=0 sa=1 ref=721 serial=1
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=6 options=27 type=00 soft=0 mtu=9038 expire=1247/0B replaywin=2048

zoriaxAuthor

New Member

As I can see my MTU is correct : I have

dst_mtu=1492

SA : mtu=1422

Jumbo is of course allowed on switches between my server and the FortiGate but if I look into for exemple a Windows machine, the MTU is set to 1500 (default).

For me here it's correct too....

Stelios_FTNT

Staff

The debug output you display is just a reflection of your current configuration which doesn't give any information about potential TCP retransmissions due to lower MSS in the path. A valid test would be to change/increase the MTU configuration of your interfaces where the IPsec tunnels are bound, and verify if the performance is better. Once you change the MTU, the same diag command will give you dufferent "dst_mtu" and "mtu" values.

If you want to debug deeper and have a better visibility, you need to capture a trace on the interface the traffic is incoming and analyze the TCP performance.

zoriaxAuthor

New Member

Ok nice ! I'm just a bit confuse how I need to change this value.

For MTU on WAN interface or VPN Interface ? And should I adjust MSS on these interfaces too or only on firewall rules ?

Thanks

Stelios_FTNT

Staff

Assuming the WAN interface is the ingressing interface of your traffic, you can change the MTU only on the WAN interface and this change will be reflected on the IPsec interfaces bonded to this physical interface as well. No need to touch the firewall policies. Don't forget to also change the MTU on the egressing interface of the firewall to allow the same amount of bytes being sent out of the firewall.

Once this is done, you can verify the MTU of your IPsec tunnel with the above command and then you can do a traffic test to verify if you get better performance.

zoriaxAuthor

New Member

I think I need to decrease the MTU because of my ISP. I can't have MTU bigger than 1500.

After a few test I tried to adjust MTU on 2 Windows Machine (one on each side of my VPN) but nothing change.

If I tried to ping through the tunnel, my MTU max is 1422 (1394 + 28). I adjust my Ethernet card to this value but SMB continue to be very slow (on both side of the tunnel)...

I'm sure it's correct like that but I continue to be confuse about this behaviour and how to solve this problem.

Anyone could help me to have the right process to debug and solve this kind of slowness ?

Thanks

zoriaxAuthor

New Member

Hi !

I think I need to decrease the MTU because my ISP is limited to a MTU of 1500.

I tried many configuration but nothing seems to work. My WAN MTU is set to 1492 (reflecting PPPoE and validating by a ping). I think my IPSec will automatically adjust its MTU regarding of the WAN. So I don't understand where is the problem.

If a run a wireshark, I can't see any fragmentation but maybe it's related to honor-df in global setting. What do you think about this feature ? Should I disabled it ?

Thanks

zoriaxAuthor

New Member

Hi everyone !

I continued my investigations. As I can see, the slowness seems to be related to PPPoE interfaces. When I have an interface with static public IP I don't have any slowness...

My question is why on PPPoE it's slow and why not. Is it a MTU problem or ike proposal problem ?

Thanks

Stelios_FTNT

Staff

Most probably you pinpointed the issue. Your FortiGate F-devices come with a NP6Xlite (SOC4) processor, which like all other NP6 units, can't accelerate PPPoE traffic. See here:
https://docs.fortinet.com/document/fortigate/7.0.1/hardware-acceleration/149012/np6-session-fast-path-requirements

Sessions that are offloaded must be fast path ready. For a session to be fast path ready it must meet the following criteria:

Layer 2 type/length must be 0x0800 for IPv4 or 0x86dd for IPv6

According to Wikipedia PPPoE uses EtherTypes of 0x8863 and 0x8864 so this traffic won't benefit of the SoC4 acceleration of your device and hence handle all the traffic on the CPU. You should be able to monitor CPU usage while using PPPoE with "get sys perf status" and compare with the CPU usage when using plain IPv4 traffic (if possible). You are probably reaching high levels of CPU usage when PPPoE is in place and potentially reach the limits of these models.

zoriaxAuthor

New Member

Hi ! Thanks for your return. On my FortiGate-100F my CPU reach approx 4% when I download a file trough SMB and VPN... Seems not related to that for me.

Maybe MTU but I tried 1492 - 1452 and I tried to adjust MSS on WAN and LAN interfaces of the fortigate and there is no change... It drives me crazy :)

Character_Flamingo_1

New Member

Hi,

If you are using PPPoE try switching to DHCP that worked for me.

Had the exact same problem.

zoriaxAuthor

New Member

Hello,

I don't understand what you have done. II can't switch to DHCP because it's PPPoE and I need to be directly connected with a public IP address.

Fett

New Member

Did you happen to ever get anywhere with this issue? I am having a similar problem but seeing it with 100MB/100MB pppoe connection having IPSEC traffic going as low as 17MB.

TG1

New Member

I was dealing with a similar situation. I have a FortiGate 100E cluster running on one side and a pfsense running in the other point in different countries, at the pfsense side I have 1Gb/s internet link and on the FortiGate 500Mb/s up and down (dedicated link). I change all the configurations already mentioned and other ones in this forum and I was still getting up to 250Mb/s for the upload from the FortiGate to pfsense using ipsec, without ipsec I can reach 500Mb/s normally, I'm using iperf server package on pfsense side and a linux server on the private network behind the FortiGate firewall as iperf client. After all the changes, tests, I tried to run the iperf test from two servers on the FortiGate private network side and comes out I get the result I was looking for, reaching the 500Mb/s for upload because for the download was working on the first try. Now, I have another thing to look at but I'm sure about the structure supporting the 500Mb/s up and down using the ipsec tunnel.

Show more replies

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded