Weird slowness on FG 200D

[netwrkr]

I'm working (albeit very slowly) with FN support on this but wanted to throw the issue out to a wider audience to see if anyone had any ideas.

I have 2 x FG200D in a active-active HA pair.  Was running 5.2.10 with the below issue.  Currently running 5.4.3.

Within the last 2 weeks, traffic to the Internet has become sporadically slow.  If I disable all the UTM settings (AV, IPS, APP, WEB), web pages start fully loading again, and everything is back to speed consistent with a 300/300Mbps Internet connection.

I have verified on both the switch and FG sides that none of the ports are having issues (all are 1Gbps and no errors reported).

I have rebooted both firewalls several times.

I have broken the a-a pair into a a-p.

I have selectively turned on the UTM settings one at a time - it doesn't matter which one is enabled, any and all of them make things slow to the Internet.

I have changed the VDOM from proxy to flow mode.

I have changed all VDOMs from proxy to flow mode.

Load on the firewall (both memory and cpu) is typically less than 30%.

I have disabled every UTM setting applied to any other rule, in every VDOM.

I can consistently replicate the issue by trying to load a page like yahoo.com; aol.com, cnn.com, etc.

I have tried turning off asic-offloading.

I am perplexed to say the least.  I have ~10 more pairs of 200D's, in near identical configuration running in other locations without issue.

Thanks for your feedback.

Tom

[MikePruett]

Couple of questions, most are "duh" type questions I just want to cover my bases.

- Has the firmware always been updated using proper firmware upgrade path?

- Has this particular cluster worked appropriately before? If so, what Firmware was it running then?

- What firmware did this issue begin with?

- Are your logs showing any system / event triggers that would lead you to thinking the device is having issues?

- I would try replacing some cables (the HA cable in particular)

- I would try an HQIP if possible

I ran into a similar issue with a pair of 500D's and it ended up being the slave 500D experiencing issues. Disabling UTM resolved the issue but that was because UTM is carried out by the secondary unit when FortiGates are in HA.

The slave unit passed all hardware tests so just a heads up. After tons of Fortinet bashing their heads into the wall trying to figure out what is wrong they will eventually RMA it and send you on your way (if it is the same situation I was in)

[netwrkr]

- Has the firmware always been updated using proper firmware upgrade path?

Yes.

- Has this particular cluster worked appropriately before? If so, what Firmware was it running then?

To the best of my recollection, the cluster started life around 5.2.3 and has been upgraded over time.  The issue arose approx 2 weeks ago at which time it was running 5.2.8.  I upgraded the cluster to 5.2.10, and then to 5.4.3.

- What firmware did this issue begin with?

5.2.8

- Are your logs showing any system / event triggers that would lead you to thinking the device is having issues?

Negative.  I've even gone line by line through the debug logs.

- I would try replacing some cables (the HA cable in particular)

I will make it so.

- I would try an HQIP if possible

Ok.

I ran into a similar issue with a pair of 500D's and it ended up being the slave 500D experiencing issues. Disabling UTM resolved the issue but that was because UTM is carried out by the secondary unit when FortiGates are in HA.

Yeah, I was chasing this line of thought also as I too have seen similar issues when one of the HA pair 'wedges' or otherwise gets into a weird state - both times this has happened to me before, its because of disk errors on one of them.

Thanks for the feedback.  More to follow...

[James_G]

Try running the units in isolation - ie power off one unit at a time - can you fix the issue to one or both of the units

[netwrkr]

So  I just broke the HA pair and ran each one for a period of time in standalone mode.  No change.