|
Low throughput can affect a network in a variety of ways, from being protocol or application-specific to impacting all services and hosts.
Note:
Back up the FortiGate config file to a safe location before following the steps below.
- Identify the scope of the problem.
- Which hosts are affected and which services.
- Document the average throughput observed. Some services can take time to buffer efficiently so the average is often more reliable after a minute or two of continuous testing.
- Consider using iPerf or another tool such as the built-in traffic test.
- To test the full circuit, iPerf3 should be used.
- Verify connectivity is reliable.
Document the hosts involved, such as the test client IP and server IP.
Is only Internet-bound traffic affected? If so it may be a good idea to schedule a maintenance window to reboot the ISP modem before continuing through this article.
- Does it affect hosts from other networks and VLAN too?
- Is it only for traffic over a specific VPN, VLAN, or other segments?
- If the test client continuously pings the server over several minutes are any packets lost?
- In Windows: ping -t <server ip>.
- Alternatively: pathping <server up>.
If there is packet loss (especially above 2% total) it is recommended to address it before continuing.
Throughput can suffer drastically if packet loss occurs.
Some common causes could be bad cabling, low system resources on the client/server, or intermediate network equipment.
- Verify duplex and negotiated speed.
- Check every node in the path from the client to the server to ensure the cables and devices involved are all negotiating to both full duplex and the proper speed setting.
- For example, if an intermediate router only has 100 Mbps ports but all other devices support gigabit then the router will be a choke point for the traffic reducing to roughly one-tenth of the expected gigabit speed.
To check a FortiGate port (which should be done for both ingress and egress interfaces) run:
diag hardware deviceinfo nic <portname>
Example:
diag hardware deviceinfo nic port1
Name: port1
Driver: virtio_net
Version: 1.0.0
Bus: 0000:00:06.0
Hwaddr: 00:09:0f:09:26:00
Permanent Hwaddr:00:41:72:74:16:01
State: up
Link: up
Mtu: 1500
Supported: 1000full 10000full
Advertised:
Speed: 10000full
In the above example, it can be seen that the port1 interface negotiated to a 10 Gbps speed.
- Fine-tune MSS to ensure it is not contributing:
See the article Setting TCP MSS value.
- Check the system performance.
Ensure the FortiGate has sufficient resources, and that anomalous behavior is not contributing to the low throughput.
Run the following during testing:
get sys perf st
diag sys top summ
diag sys top-mem
di de crashlog read
Note:
If the system is in conserve mode that lower throughput can be expected. The source of the conserve mode would need to be addressed first.
-
Remove traffic inspection and Traffic Shaping during testing.
- If security profiles are in use, backup the configuration and create a new test firewall policy.
- If Traffic Shaping Policies are configured, disable them and test.
- Restrict it to the source and destination IPs involved so that it will not affect other traffic. Then move it above existing policies and re-test.
- Use flow-based inspection.
- If traffic inspection is mandatory for testing, consider cloning the existing UTM profiles and toggling the cloned ones all to flow-based.
- Flow-based is more efficient and faster at processing traffic while also using less memory
- Conversely, if the proxy inspection is mandatory, consider changing the tcp window to dynamic.
- Disable NP offloading:
- On the new test firewall policy from above disable NP offloading functionality.
- This can help in cases where an NP may be faulty or encounter aberrant traffic.
- See this article for instructions. Then re-test.
- Check bandwidth availability.
- This is especially important for traffic traversing the internet.
- Check that the internet service at both ends of the connection is not saturated for both ingress and egress traffic.
- The best way to do this is likely by reviewing logs or running a report based on log data.
Also, under FortiView -> Policies/Sources/Destinations there is some bandwidth information available.
Example:
If traffic is saturated, consider implementing a guaranteed traffic shaper for the traffic in question.
- Isolate from the internal network.
- In extreme cases, it may be beneficial to connect a test client and server directly to spare firewall ports and re-test after creating appropriate policies.
-
Impact of Latency on TCP Download Performance.
- TCP is a reliable protocol for sending data accurately and in order, but its performance is affected by latency.
- The higher the round-trip time (RTT), the lower the TCP download throughput will be, even if the connection speed is high. For example, with a 1 Gbps link, the maximum possible throughput might only be 17.4 Mbps if the latency is 30 ms. This shows how TCP performance can be significantly constrained by latency.
- To estimate the maximum theoretical throughput of a TCP session, the following formula can be used:
This ensures internal infrastructure is not contributing to the issue.
|
