Slow iPerf (and generall traffic) through firewall
Before I start, I want to clarify that the remote iPerf server I’m testing against is on a 10Gbps link.
I have a Fortigate 61F with the wan2 interface connected to a 1Gbps (synchronous) Internet connection. If I run an iPerf test from that interface, I can achieve ~740mbps. However, if I run the same test from a machine on the inside of the firewall, I can never get past 300Mbps.
I'm not using a VPN tunnel, I don’t have any strange routing (no PBR), and I don’t even run a routing protocol, just a single static default route. There is no traffic-shaping, and I did not create any software switches. I have a single physical interface is connected to a Cisco switch (no errors on the ports), and it’s configured for VLAN trunking.
I thought maybe the outbound policy might be slowing it down, so I’ve removed all UTM features, set the SSL inspection to “no-inspection”, and even disable traffic logging but it doesn’t make the slightest difference.
config firewall policy edit 2 set name "outbound default" set uuid 31503162-25f6-51eb-f268-20a2aadc8c96 set srcintf "Servers" "Workstations" "Lab-1" set dstintf "Outside" set action accept set srcaddr "all" set dstaddr "all" set schedule "always" set service "ALL" set logtraffic disable set logtraffic-start enable next
I’m not sure where to start looking next so any suggestions would be appreciated!
Resolution Check lInk speed WAN interface TIP: Fig. 1. Optimizing the Link Speed and MTU on the Advanced tab of the WAN interface where the defaults fail to establish a compatible ISP connection.
Check the MTU. A mismatch in the maximum transmission unit (MTU) between the firewall and the ISP device can impact the bandwidth. In one of my cases, just by optimizing the MTU we were able to regain the bandwidth (5% to 90%). An optimal condition for the test would be to connect a computer directly to the firewall (Fig.2). This is to rule out uncertainties about latency due to a network. Image
TIP: Fig.2. Checking MTU on a directly connected computer is my preferred way to minimize uncertainties about latency involved in a complex network.
A typical MTU optimization test involves doing a ping with the options of -f (don't fragment) and -l (size) as summarized in Fig. 3. I start with an MTU of 1500 and find out a value where there is a successful ping. Then I add 28 bits to derive an MTU value I would be using on the WAN interface.
TIP: Ping Test on a Windows Computer directly connected to the Firewall. I would add 28 to the final MTU value that resulted in a successful ping.
Check the DNS settings of the Computer where you are testing. On a few occasions, although you see a reasonable bandwidth, web pages take long to initially load. First suspect was the DNS. When you type abc on a browser, the first thing we do is a DNS name resolution to get the IP. In Fig. 4, the host is trying to resolve names by accessing local DNS- one local and another across a VPN tunnel. This is a security measure. However, if the local DNS is having an issue (overwhelmed, network latency...), it will slow the DNS name resolution. Just by using a public DNS that is readily available, we were able to overcome the slow page load issue.
Thank you both Graham, and Rachel for your suggestions!
The default protocol for iPerf is TCP (unless you specify -u) but I did verify that the remote server was started using TCP (the remote server is actually an Exfo MAX-880 tester).
I did test different window sizes (64K, 512K, 1M, 5M, 20M) but they make no difference at all.
The MTU on all the Fortigate interfaces is set to 1500, but the actual usable MTU is 1472. I did verify with the ISP that their ethernet side is also configured for 1500 so there are no mismatches there.
The biggest problem we’re having is with sustained throughput. A simple speed test does show good internet speeds, but when we’re doing any kind of heavy file transfers, those speeds average ~10% of the capacity.
For instance, we have a daily transfer that runs between our on-site S3 storage to a remote location. The total size is usually 300-400 GB/day (with no files being less and 5GB) and it’s accomplished using Rclone. Without the Fortigate, we can sustain 57MB/sec (~450Mbps), but through the Fortigate that speed drops to ~10MB/sec (~80Mbps).
TEST Transfer bypassing the Fortigate
TEST Transfer through the Fortigate
Internally (between VLANs) on the Fortigate, we can sustain ~90MB/s (720Mbps), and all the ports on our internal switches are clean (no errors/ no drops). The speed issue only seems to happen when we’re running through the wan interface.
My only thought at this point is the FGT is busy servicing other traffic. It sounds like you are doing East-West inter-vlan routing through the FGT? That's a fairly small box to be doing edge FW and internal FW at the same time.
Are you running these tests when there is absolutely no other traffic flowing through the gate?
This is a very small site, and east-west traffic happens during the day. The Internet transfers happen at midnight when nobody is here (and there is no traffic on the network). For my testing, I did them with no other traffic (or negligible traffic) on the network.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.