Bad bufferbloat on WAN link. How to shape with Fortigate
Have recently started a new contracting gig. part of the role is implementing a voip telephone system , and I've been investigating the network a little as there are some problems with jitter and large latency spikes to handsets. Anecdotally users are also reporting "slow" internet , often when we are no where near peak capacity.
a (not managed by us) telco router/media convertor is onsite (either one or both , I see a cisco MAC from the fortigate WAN interface, its near the MDF in the building, which we don't have access to) . with a 50/50 fibre link.
RRUL testing shows pretty bad bufferbloat
I'm not very familiar with fortigate products, I don't see any option for fq_codel , HTB etc as such , which I have had some success implementing on linux based routers etc before.
Im thinking much of this problem is either because of how the ISP internet gear is buffering traffic (if its a router I can see in ARP), or its just discarding everything above 50m. I see spikes over 50mbit when the link is saturated that drop off quickly, I dont think they are letting us burst traffic though, I think its just being dropped so I need to setup some shaping outbound.
There is pretty much zero setup on the fortigate right now from the outfit that installed it. No QoS. There are Vlans but they do nothing except have slightly different subnets (all route to each other, no tagging or QoS). There are stacked DELL switches attached to the LAN, everything in the office goes through these.
Anyone have some experience trying to solve this on fortigate gear , or some tips on config?
in the past ive worked with mid band ethernet type services where its fairly essential to shape traffic before handing it off to the NTU ( a dumb layer 2 device thats just mirroring the mac from the switch in the exchance). I'm thinking if I can just shape everything at the LAN interface to slightly less than 50 this will improve, then I can work on QoS for the voice vlan etc.
Any ideas or tips? I think we can get much better performance from this service.
I don't know if that's the same issue exactly, that was clearly bufferbloat. latency seesawed stupidly after around 40% load.
try running a RRUL test with flent just to be sure. you will need a MAC/linux box (or a VM and good NIC in your computer) and a netperf/iperf server.
what sits behind the fortigate? I also found some issues in the dell switch stack they had there as well. it was cabled wrong for a start. this was also causing problems.
where abouts are you speed testing from? directly off the fortigate?
have the telco given you any indication why they believe it's CPE, besides the Juniper getting full speed? I'd be curious to know how they tested that exactly..
what do your traffic reports show (telco ones) ?
how do those line up with the fortigate reports?
do you have netflow/sflow setup ?
You could try hard shaping upstream to link speed minus a delta (try 10%) and see if that makes a difference.
also make sure you have set the link speed correctly , from memory there are two places to do it, ones effectively a label in the GUI , but the other is set via cli like this:
config system interface
set inbandwidth <kb>
set outbandwidth <kb>
..may vary depending on your fortios version.
if it is a buffer issue then there are unfortunately a lot of places it can happen. NIC's (OS buffer and hardware buffers) , your switches , the CPE, the telcos gear at other end. etc. Its often telcos, because everyones got obsessed with not dropping packets, which is counterproductive. if you are maxxing something you want packets to drop so TCP will backoff.
I know its not particulary helpful, but swapping the fortigate for a pfsense box improved the situation significantly.
I really hope fortigate follows some of the other major vendors in implementing some form of SQM at some stage.. but I'm not sure that this is the same problem.
from my telco days certain types of handoff require hard policing on customer end, usually layer 2 ones. else packets get dropped in access network .
I did some testing with the DF bit set and found that packets > 1472 fail, so I set the WAN interface max mtu to 1472, however, this hasn't had any effect, like the other changes to the traffic rate on the interface.
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.