Bad bufferbloat on WAN link. How to shape with Fortigate
Have recently started a new contracting gig. part of the role is implementing a voip telephone system , and I've been investigating the network a little as there are some problems with jitter and large latency spikes to handsets. Anecdotally users are also reporting "slow" internet , often when we are no where near peak capacity.
a (not managed by us) telco router/media convertor is onsite (either one or both , I see a cisco MAC from the fortigate WAN interface, its near the MDF in the building, which we don't have access to) . with a 50/50 fibre link.
RRUL testing shows pretty bad bufferbloat
I'm not very familiar with fortigate products, I don't see any option for fq_codel , HTB etc as such , which I have had some success implementing on linux based routers etc before.
Im thinking much of this problem is either because of how the ISP internet gear is buffering traffic (if its a router I can see in ARP), or its just discarding everything above 50m. I see spikes over 50mbit when the link is saturated that drop off quickly, I dont think they are letting us burst traffic though, I think its just being dropped so I need to setup some shaping outbound.
There is pretty much zero setup on the fortigate right now from the outfit that installed it. No QoS. There are Vlans but they do nothing except have slightly different subnets (all route to each other, no tagging or QoS). There are stacked DELL switches attached to the LAN, everything in the office goes through these.
Anyone have some experience trying to solve this on fortigate gear , or some tips on config?
in the past ive worked with mid band ethernet type services where its fairly essential to shape traffic before handing it off to the NTU ( a dumb layer 2 device thats just mirroring the mac from the switch in the exchance). I'm thinking if I can just shape everything at the LAN interface to slightly less than 50 this will improve, then I can work on QoS for the voice vlan etc.
Any ideas or tips? I think we can get much better performance from this service.
There are CLI options to tell the Fortigate the bandwidth that you are subscribed to. (Both inbound and outbound. On the interface in the cli, type 'set?' and see the list of available options.) That along with proper policy shaping should quell those traffic drops and hopefully help get all you traffic through during periods of high traffic.
I have tried setting the bandwidth on the interfaces and it doesn't change the speed in my testing at all. I just used the gui, under interfaces. I continually get 120mbps down and about 12mbps up, even if I set the interface bandwidth to 50mbps and 5mpbs I still get 120/12? It is like it isn't working.
Additionally, do you have a good link on the traffic shaping piece that I could read in order to get it setup correctly?
On the Fortigate there is two options on the cli for setting the bandwith. The option that you can set from the GUI sets the parameter of "estimated-upstream-bandwidth" and "estimated-downstream-bandwidth" which is only used to estimate the links utilisation. There is another option which you can only set from the cli called "inbandwidth" and "outbandwidth" which is the bandwidth limit to apply to the interface.
I use these settings in association with dscp and the traffic shapers on the fortigates to provide QoS for phone and video when required and it works well. If you have not already, download the cli guide from the docs.fortinet.com site as it will explain a lot of these cli only commands in more detail.
Just an update/query , I haven't had time to revisit this for a while.
Still have the latency under load issues, I'm now suspecting its could be way too large buffer sizes on the layer 3 switch stack. I don't see a way to fix this easily unfortunately. I've qos'd the voice vlan now which has helped a little, but I still get extensions lagging (up to 2000ms) occasionally and it doesn't correlate with WAN load. Theres plenty of room left on the WAN link , and latency is still just appalling at times. Its less related to load and more to the types of traffic (big flows seem to mess with latency) as far as I can see.
This isn't really a QoS or shaping issue. This is a buffering issue that manifests well before the WAN link is saturated.
I'm going to try and experiment on the fortigate by manually tweaking txquelen on the WAN interface during some planned works coming up afterhours. It could have some unforeseen consequences so I don't recommend trying this without a console cable handy or during business hours ;) It also may not work on your particular model (we have an 80E here). I've verified this works quickly and doesn't crash the firewall , but I need to test it. Your mileage may vary etc.
you have to execute ifconfig using fnsysctl to manually set these values.eg:
fnsysctl ifconfig wan1 txquelen 100
It's current set to 1000 (just run fnsysctl ifconfig to see what its set to) , which is probably very excessive for a smallish WAN link. I can verify setting the interface bandwidth has no bearing on this value. I'll try it on the wan link and the ethernet back into the switches.
I'll report back if this makes any difference whatsoever to my flent tests.
My question - are fortigate planning to implement any sort of SQM in the future? linux / BSD / mac os , openwrt, pfsense, palo alto and many others have all implemented something like fq_codel.... ;)
This would probably go a long way to resolving problems like this...
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.