Bad bufferbloat on WAN link. How to shape with Fortigate

[richg]

Hi all,

Have recently started a new contracting gig. part of the role is implementing a voip telephone system , and I've been investigating the network a little as there are some problems with jitter and large latency spikes to handsets. Anecdotally users are also reporting "slow" internet , often when we are no where near peak capacity. 

a (not managed by us) telco router/media convertor is onsite (either one or both , I see a cisco MAC from the fortigate WAN interface, its near the MDF in the building, which we don't have access to) .  with a 50/50 fibre link.

RRUL testing shows pretty bad bufferbloat 

I'm not very familiar with fortigate products, I don't see any option for fq_codel , HTB etc as  such , which I have had some success  implementing on linux based routers etc before. 

Im thinking much of this problem is either because of how the ISP internet gear is buffering traffic (if its a router I can see in ARP), or its just discarding everything above 50m. I see spikes over 50mbit when the link is saturated that drop off quickly, I dont think they are letting us burst traffic though, I think its just being dropped so I need to setup some shaping outbound. 

There is pretty much zero setup on the fortigate right now from the outfit that installed it. No QoS. There are Vlans but they do nothing except have slightly different subnets (all route to each other,  no tagging or QoS). There are stacked DELL switches attached to the LAN, everything in the office goes through these.

Anyone have some experience trying to solve this on fortigate gear , or some tips on config?

in the past ive worked with mid band ethernet type services where its fairly essential to shape traffic before handing it off to the NTU ( a dumb layer 2 device thats just mirroring the mac from the switch in the exchance). I'm thinking if I can just shape everything at the LAN interface to slightly less than 50 this will improve, then I can work on QoS for the voice vlan etc. 

Any ideas or tips? I think we can get much better performance from this service.  

[pireality]

This is an excellent question, I wonder why nobody has responded in over 3 weeks.  I would like the same assistance.

[rwpatterson]

Welcome to the forums guys.

There are CLI options to tell the Fortigate the bandwidth that you are subscribed to. (Both inbound and outbound. On the interface in the cli, type 'set?' and see the list of available options.) That along with proper policy shaping should quell those traffic drops and hopefully help get all you traffic through during periods of high traffic.

[pireality]

I have tried setting the bandwidth on the interfaces and it doesn't change the speed in my testing at all.  I just used the gui, under interfaces.  I continually get 120mbps down and about 12mbps up, even if I set the interface bandwidth to 50mbps and 5mpbs I still get 120/12?  It is like it isn't working.  

Additionally, do you have a good link on the traffic shaping piece that I could read in order to get it setup correctly?

Thanks for your speedy response BTW!

[rwpatterson]

One caution: The speed may be in mega BYTES. Look for a small or capital B.

Divide the speed you thought you have by 8 and see if that improves things.

[pireality]

I confirmed mbps, oddly, the max mtu size changed from this morning's 1472 to now it is 1444....why would that change dynamically during the day?  Weird.

[crispy]

Hi All,

On the Fortigate there is two options on the cli for setting the bandwith. The option that you can set from the GUI sets the parameter of "estimated-upstream-bandwidth" and "estimated-downstream-bandwidth" which is only used to estimate the links utilisation. There is another option which you can only set from the cli called "inbandwidth" and "outbandwidth" which is the bandwidth limit to apply to the interface.

I use these settings in association with dscp and the traffic shapers on the fortigates to provide QoS for phone and video when required and it works well. If you have not already, download the cli guide from the docs.fortinet.com site as it will explain a lot of these cli only commands in more detail.

crispy

[richg]

Just an update/query , I haven't had time to revisit this for a while. 

Still have the latency under load issues, I'm now suspecting its could be way too large buffer sizes on the layer 3 switch stack. I don't see a way to fix this easily unfortunately. I've qos'd the voice vlan now which has helped a little,  but I still get extensions lagging (up to 2000ms) occasionally and it doesn't correlate with WAN load.  Theres plenty of room left on the WAN link , and latency is still just appalling at times. Its less related to load and more to the types of traffic (big flows seem to mess with latency) as far as I can see. 

This isn't really a QoS or shaping issue. This is a buffering issue that manifests well before the WAN link is saturated. 

I'm going to try and experiment on the fortigate by manually tweaking txquelen on the WAN interface during some planned works coming up afterhours. It could have some unforeseen consequences so I don't recommend trying this without a console cable handy or during business hours ;) It also may not work on your particular model (we have an 80E here). I've verified this works quickly and doesn't crash the firewall , but I need to test it. Your mileage may vary etc. 

you have to execute ifconfig using fnsysctl to manually set these values.eg:

fnsysctl ifconfig wan1 txquelen 100 

 It's current set to 1000 (just run fnsysctl ifconfig to see what its set to) , which is probably very excessive for a smallish WAN link. I can verify setting the interface bandwidth has no bearing on this value.  I'll try it on the wan link and the ethernet back into the switches. 

I'll report back if this makes any difference whatsoever to my flent tests.  

My question - are fortigate planning to implement any sort of SQM in the future? linux / BSD /  mac os , openwrt, pfsense, palo alto and many others have all implemented something like fq_codel.... ;)

This would probably go a long way to resolving problems like this... 

[wallky]

Was there any outcome on this? I have a very similar if not exactly the same issue:

One ISP here in Australia uses Cisco as their NTU and we plug Fortigate's into the Cisco. The link is a 50/50Mbps, and we get 50Mbps down, but up is only getting 25Mbps.

I have an egress shaper applied to the WAN interface, have set all of the estimated speeds etc but it makes no difference.

I've set the egress shaper to 10Mbps and it only gets 10Mbps. I set it to 20Mbps and it only gets 20Mbps, but anything above 25 just stops at between 25-30Mbps.

It only happens on this one ISP, as I have many Foritgates out there with other ISP's and we don't have this issue.

They are saying it's our CPE.

I'm interested to see if changing the txquelen made any difference to your problem, as I suspect it's something similar to my issue.

Thanks!

[richg]

txque change minimal impact.

shaping outbound to a bit less than full speed and lowering that made a slight difference.

in the end the solution was to put a netgate (pfsense) firewal in and setup fq_codel

voip latency is around 25ms now solid , even under load.

this is in australia too. its a telstra service resold by someone else, I suspect they are doing something screwy in the interconnect because its not the only one I've seen now from the same ISP.