LACP recommandation between Fortigate FortiOS 5 and Cisco switch

[miky]

Hello, I would like to know if some of you have a recommendation for a configuration between a Cisco switch port-channel and a Fortigate Agg FortiOS5 On my Cisco configuration I' ve used this for the physical interfaces channel-group 1 mode active switchport nonegotiate On the Fortigate I have edit " Agg1" set vdom " root" set type aggregate set member " port1" " port2" set lacp-mode passive So LACP active on the Cisco switch and passive on the Fortigate. Thank you

[emnoc]

You really need the show lacp commands. What you have with the " lacp-mode-static" is a static bundling no LACP protocol or no-Negotiation. No-Negotiation is not necessary a bad thing, and some device just don' t do LACP that good. I never heard of a cisco or fortigate being one of those devices tho. I' m curious, did you do this without the cisco config of QoS enabled and without the fortigate configuration of; set broadcast-forward enable set l2forward enable set stpforward enable I' m assuming this is a transparent L2 firewall ? I never done lag bundles within any Layer2 transparent-firewalls but I don' t see why it would make a difference.

[ShrewLWD]

I must confess to not being proficient in Cisco IOS anymore, so we had Cisco TAC SSH' d into the 3750s while I made changes to the Fortinet on my end. He was watching running lacp commands, including show lacp event, show lacp 25 (counters and internal detail), show spanning-tree, etc. We did not try turning off QoS, nor turning off the Fortinet ' sets' . We don' t do non-IP traffic, so we don' t need the l2 line, and I don' t believe we need the stpforward either, since there are no additional cisco devices behind the Fortinet needing to send traffic down. I was just turning those on thinking they were causing the lacp to fail originally, before getting Fortinet and Cisco TACs involved. Yes, the VD_PWAN is a transparent VDOM, with VDOMLinks from several other transparent VLANs accessing it. With only port15 enabled, and all firewall rules pointed to it, traffic passed no problem. Same with switching to 16. As soon as those two were LAGd (and firewall policies updated to point to the LAG), all traffic stopped. Additionally, when we put it back the way it was, for a period of time (5-10 minutes) traffic still did not flow.

[emnoc]

It sounds like a L2 STP issue if I had to guess. Once again I never seen a transparent firewall using LACP bundle. I think fortinet TAC could clarify if this is acceptable with the config you have and with the with the cmd; set l2forward enable set stpforward enable Both of which should be used with caution. I ' m betting most of your problems are in these 2.

[ShrewLWD]

Possibly, and, because I have a known good configuration from each end, I will certainly take a stab at removing those lines and putting both devices back into ACTIVE - ACTIVE, but before I got the TACs involved, those two lines weren' t in there. So unless the Cisco TAC found an additional setting he changed (possible), I don' t see it making a difference. The configs changed in this order (none worked except the last); both devices were set to ACTIVE ACTIVE I added those lines I tried Active Passive I tried Passive Active I had read on a recent previous lacp discussion that STATIC is not a good idea, so I didn' t try that, I got the TACs involved instead. We tried static static So those lines were added early in my tests, and again, I don' t know if the Cisco TAC found and changed something else. But ' I will submit myself to these experiments in the name of science!' -Allan Walker Blair ;)

[emnoc]

FWIW: You should open a new thread and not hijack this one. But I would review your configuration to ensure the LACP neighbors counters

[ShrewLWD]

How is this a hijack? 1)The OP requested recommendations for Cisco to Fortinet LAG. That' s what I have, and this have been my experience, and what works. 2) I' m even willing to continue troubleshooting for the sake of the community in general, and the OP specifically, so that he can have an even more informed decision.

[miky]

Hello, I don' t mind people talking about the LACP counters problem on this thread because I asked the Cisco vs Fortigate configuration recomendation as I see output drops on the Cisco side. The latter statement is my real problem. emnoc: My configuration is vers simple, I have no QoS enabled except the command mls qos globally and mls qos trust cost at the interface level. I didn' t allow giant frames but usually giant frames show up in the " giant" counter #sh int po1 | i giant 0 runts, 0 giants, 0 throttles And there is no ACL. Unknown datagrams might show up in the " unknown protocol drops" counter #sh int po | i unknown protocol drops 0 unknown protocol drops My assumption towards the Fortigate is biaised I agree. That' s because I have been working for a decade with Cisco switches and much much less with Fortigates. And the FortiOS 5 is buggy, it keeps crashing !!! (case opened) So I confirm that Cisco is using LACP protocol #sh etherchannel 1 summary Number of channel-groups in use: 2 Number of aggregators: 2 Group Port-channel Protocol Ports ------+-------------+-----------+----------------------------------------------- 1 Po1(SU) LACP Gi1/0/46(P) Gi2/0/46(P) On the Fortigate # diagnose netlink aggregate name Agg1 LACP mode: passive LACP speed: slow LACP HA: enable slave: port1 actor state: PSAIEE partner state: ASAIEE aggregator ID: 2 slave: port2 actor state: PSAIEE partner state: ASAIEE aggregator ID: 2 #show lacp 1 counters LACPDUs Marker Marker Response LACPDUs Port Sent Recv Sent Recv Sent Recv Pkts Err --------------------------------------------------------------------- Channel group: 1 Gi1/0/46 54870 50720 0 0 0 0 0 Gi2/0/46 54868 50723 0 0 0 0 0 My firewall is not transparent, it' s a L3 firewall so it should block bpdu' s My next test will probably be to configure the port-channel static without LACP. I' ll keep you updated.

[emnoc]

For the output drops are going to be cisco specific. I would start by ensuring flow control is not enable show int por 1 flow These 2 commands might help with showing you the number of process switched packets. show interface stats show interfaces switching e.g SW1#show interface port 1 switching Port-channel1 Etherbundle to MACORE1 Throttle count 0 Drops RP 0 SP 0 SPD Flushes Fast 0 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 0 Drops 0 Protocol Path Pkts In Chars In Pkts Out Chars Out Spanning Tree Process 0 0 23025 2210400 Cache misses 0 Fast 0 0 0 0 Auton/SSE 0 0 0 0 SW1# I would disable anything that' s process switch ( cdp, lldp, dynamic routing protocols, GRE tunnels, basically anything sourced from the switch directly) And Yes I realize this is not a transparent. The other gentlemen is using LACP in a layer2 firewall configuration. Now for some more Qs: Do you see any performances with users traffic or just monitoring the output drops? Have you tried to disable any QoS as temporal fix to see if that effects anything or decreases the drops? Do the drops increase/decrease if you check all of the above items mention above ? FWIW, the drops probably has nothing to do with LACP btw and I disagree that 5.0.x is buggy. If it is than we are not effected by anything if it was as buggy as some claims than none of the 18 devices I mention that' s on all versions of 5.0.x would be buggy.

[miky]

emnoc, thank you for your answer. I had forgotten to mention that flow control is disable by default (input direction) on Cisco not available on the output direction. http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst2960/software/release/12-2_58_se/configuration/guide/2960scg/swint.html#wp2071785 #sh int Gi1/0/46 flowcontrol Port Send FlowControl Receive FlowControl RxPause TxPause admin oper admin oper --------- -------- -------- -------- -------- ------- ------- Gi1/0/46 Unsupp. Unsupp. off off 0 0 Gi2/0/46 Unsupp. Unsupp. off off 0 0 LLDP is not enabled, CDP is disabled on the physical interfaces connected to the Fortigate, there are no routing protocol and no gre. The switch is pure L2 and the Fortigate is taking care of the L3 #show interfaces port-channel 1 switching Port-channel1 FG_Agg1 Protocol Spanning Tree Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 19405256 1238882128 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 NOTE: all counts are cumulative and reset only after a reload. #show interfaces port-channel 1 stats Port-channel1 Switching path Pkts In Chars In Pkts Out Chars Out Processor 0 0 7260046 492528964 Route cache 0 0 0 0 Total 0 0 7260046 492528964 Do you see any performances with users traffic or just monitoring the output drops? => Yes users complain about slow network (as usual you' ll tell me) but that how I ended searching on the root cause of these output drops Have you tried to disable any QoS as temporal fix to see if that effects anything or decreases the drops? => No but I will add it in my action plan. However, I just wanted the switch to keep the marking. And also I see no reason why 100M traffic would be queued on a 2x1G port-channel About the 5.0, trust me =) There is at least one real nasty bug on it, maybe related with dhcp relay. In many cases we had kernel panic, we are able to reproduce it though and we' re currently investigating this.

[miky]

Hello, Just to let you know, the port-channel is now static/mode on but we still have output errors seen on the Cisco Po Po1 => 0,022% (expressed in packets) Gi1/0/46 => 0,017 % Gi2/0/46 => 0,039 % I know the percentage of error is quite low but there just should not have these errors in theory. Regards