1500D Bug Found and Confirmed by TAC

[Camshaft007]

All, we have discovered a bug in the 1500D’s with Fortinet TAC. I’m in the process of deploying 2 1500D clusters in our Primary and Secondary datacenters, while recycling our 800c clusters to all of our sister sites. The bug we discovered has to do with Aggregate and Redundant Interfaces… In short, they don’t work… The Aggregate ports are not negotiating LACP correctly with the Port Channel on the Cisco 2960s switch (Datacenter Switches will be Nexus 7Ks), but the 800c’s work flawlessly. Fortinet TAC and I looked at this problem from all sides and ultimately had this issue escalated to the engineers, which in turn replied to my open ticket that this is a known issue with the 1500D’s and has been reported to the Dev. Team(s).

---

Aggregate Interface issue: 1 port negotiates LACP fine, and comes on line, but other port sits in a “negotiating” state and never negotiates; thus causing the whole Aggregate to not work. Both Aggregate members are assigned different Aggregate IDs as well. Work around: only have 1 Member of the Aggregate Interface, all works fine.

---

Redundant Interface Issue: 1 port comes up, but the other does not, causing intermittent connectivity or no connectivity. Work around: only have 1 Member of the Redundant Interface, all works fine.

---

I’m glad we were able to Lab this scenario prior to pulling the trigger on the cut-over, because I would have been one upset engineer had this issue been discovered during a weekend cutover. Thoughts about a work around basically mean creating the Aggregate/Redundant Interfaces with a single member and moving forward with my cutover plans, and wait for a patch to correct this. Hope this helps someone in the future. Cam

[Matthew_Mollenhauer]

TAC have opened another Bug for our issue, 0242222. Support did note that there is another similar bug, " There is a reported issue for 10GE ports on 1500d where LACP negotiation failed and developers are working on fixing that." Our issue is related to 1G copper ports. Regards, Matthew Mollenhauer

[Camshaft007]

I also discovered another bug today. It does not affect functionality of the system, but it is still annoying. For some reason or another you can' t add comments to the VIPs via the GUI; this admin function must be done via the CLI/SSH. I know it' s a real " ticky" thing to complain about; however I use the comments/description fields when possible.

[Adrian_Lewis]

I' ve been waiting for a fix for the redundant interface issue since Dec ' 13. This was on a 200D with basic active-passive redundant pairs but I assume it' s the same root cause or at least related. Currently working on 5.0.7 with only a single port connected out of each pair. I had been told that 5.0.7 would fix the issue but then heartbleed happened and what was going to be 5.0.7 got pushed back. 5 months of waiting despite being confirmed as a bug within a few days of opening the ticket. Not impressed. I have a pair of 3700Ds to deploy in July so I' m just desperately hoping this will be resolved before then.

[AndreaSoliva]

Hi all if you do not know it.......1500D as 3700D final release was put on the official donwload/image server by yesterday evening. This release is -as of Forti info- based on 5.0.7 and will be found under 5.0.7. hope this helps have fun Andrea

[Camshaft007]

I have applied the patch to the 1500Ds and it has resolved the LAG Port issue(s) I was having. Thanks for the heads up Andrea!

[Camshaft007]

I hate to state the obvious, but I highly recommend simply rebuilding your config(s) from scratch when upgrading to new models. This will save you tons of time and heartache in the future. Doh! Yes, I can admit I tried the lazy conversion (800c->1500D) and got burned. Lesson learned. After a new build of my 1500D cluster, the cluster seems to be very happy with me and the Nexus 7K! Now and I can Chive On and start the next build.

[Tweety123]

Hi everybody,

I am currently experience some problems with LAG and Fortigate forgetting to answer LACP heartbeats. Anyone had a similiar issue before? I am running latest 5.0 firmware.

The issue happens when sending BGP routed traffic through a VPN tunnel which resides on a  VLAN interface which depends on a LAG interface.

I already opened a ticket at TAC but maybe someone had the same issue with a FGT1500D.

Cheers

[Matthew_Mollenhauer]

We haven't seen anything like that happening on either of our 1500D clusters, (5.0.7 & 5.0.9).

Is the issue you have is that the switch (or other device) connected to the 1500D tearing down the LACP bundle? The only thing I can think of is your switch is doing fast lacp and your Fortigate is doing it's default of slow LACP.

config system interface

  edit interface_name

    set lacp-speed {fast|slow}

  next

end

http://docs-legacy.fortinet.com/fgt/handbook/cli52_html/index.html#page/FortiOS%205.2%20CLI/config_s...

Regards,

Matthew

[Tweety123]

According to the Switch the Fortigate is missing LACP and tearing it down. 

I also see the following messages when this event occurs:

msg="NSM: pal_kernel_if_get_bw:Operation not supported

vd="root" action=crash msg="Pid: 00180, application: fgfmsd, Firmware: FortiGate-1500D v5.0.9,build0292b292,140801 (Release), Signal 11 received, Backtrace: [0x00606094] [0x0060787c] [0x00603b63] [0x006040f1] [0x0061720e] [0x00615a1b] [0x00611262] [0x00605e81] [0x0060f99a] [0x0043bf30] [0x0043ba0b]

And OSPF is going nuts as well as hearbeat and IPSEC. The problem for me at the moment is to find out what happens first, everything going nuts or LACP event missing. I also see CPU usage raising in this moment. The only thing to recover from this action is to switch to a VPN tunnel which is not running through the same LAG as other VLAN tagged traffic.

We have an FGT3040B with the same firmware but without LAG running at the other side of the tunnels with BGP over them which has no issues.

[Matthew_Mollenhauer]

From the 5.0.9 release notes, there are a couple of known issues for the 1500D & 3700D (both NP6 units):

[ul]

242298 - When the FortiGate unit is very busy with high CPU usage, IPsec VPN packets

may be lost

242012 - IPsec VPN traffic throughput is highly unstable

Workaround: this only happens on IPsec interface added to a 40G LAG. Don't use IPsec VPN over a 40G LAG.

[/ul]

There were other issues for the NP6 units in 5.0.7, such as using interfaces in a LAG that are attached to different NPU's.

Could you post the LAG interface config you're using?

The other question I have, the fgfmsd is daemon is responsible for communications with FortiManager & I think FortiCloud, is this a managed unit?

Regards,

Matthew