1-tier MCLAG network resets when access switches connected to both dist switches

[76maverick]

My network (currently 1x GW, 2x dist switches, and multitude of access switches) is set up for mclag. Our ultimate goal is to have a full HA system. Our 2nd gateway hasn't been added till I resolve the layer 2 issues.

Both dist switches connect to the gateway over a combined 8x ports, 4 per dist switch. The dist switches are connected to each other via 2x physical links (mclag-icl). 

All access switches were initially connected to one dist switch. As a second link is activated on the second dist switch, the entire network resets and all switches lose conn. It takes almost 45mins for the network to normalize. Traffic flows over both links when things work but we haven't tested removing one link for fear of generating another 45min long "hold your breath that things normalize" event. Any help on what we're doing wrong would be appreciated.

Here's a config from one dist switch:

edit "DIST-ICL"
set port-selection-criteria src-dst-ip
set description ''
set mode static
set bundle disable
set auto-isl 0
set fortilink 0
set isl-fortilink 0
set mclag-icl enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port1" "port2"
next

edit "GATEWAY"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 1
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port22" "port21" "port24" "port23"
set lacp-speed slow
set aggregator-mode bandwidth
next
edit "Access1"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 0
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port9"
set lacp-speed slow
set aggregator-mode bandwidth
next

And here's from the other dist switch:

edit "DIST-ICL"
set port-selection-criteria src-dst-ip
set description ''
set mode static
set bundle disable
set auto-isl 0
set fortilink 0
set isl-fortilink 0
set mclag-icl enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port1" "port2"
next
edit "GATEWAY"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 1
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port22" "port24" "port21" "port23"
set lacp-speed slow
set aggregator-mode bandwidth
next
edit "Access1"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 0
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port7"
set lacp-speed slow
set aggregator-mode bandwidth
next

Also running a check shows some mismatches but I'm not sure where to start tshooting that:

diag switch mclag peer-consistency-check

Running diagnostic, it may take sometime...

** Comparing "switch.global.mclag-split-brain-detect" config ....OK
** Comparing "switch.global.mclag-split-brain-all-ports-down" config ....OK

mclag-trunk-name peer-config lacp-state stp-state local-ports remote-ports
__________________ ___________ __________ _________ _____________ _____________

Access1-0 MISMATCH UP OK port7 port11
GATEWAY MISMATCH UP OK port21 port22 port21 port22
DIST-ICL* MISMATCH UP OK port1 port2 port1 port2

[76maverick]

What I've seen so far from the fortiswitch documentation, is I'm missing lacp-active on the DIST-ICL as well as auto-isl. I also shouldn't have these on the links facing the gateway. That said, I'm not sure if this is just an ommission in the document or they shouldn't be there.

[76maverick]

Any takers on help with this?

[gfleming]

Are you running Standlone or FortiGate Integrated?

What is the trunk config on the access switch?

[76maverick]

Apologies for the late reply. I need to check more frequently. Switches are all fortigate integrated and not standalone. Here's the config on one switch.

config switch trunk
edit "_FlInK1_MLAG0_"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 0
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port52" "port50"
set lacp-speed slow
set aggregator-mode bandwidth
next
end

And from a different switch:

edit "_FlInK1_MLAG0_"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 0
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port47" "port52"
set lacp-speed slow
set aggregator-mode bandwidth
next

[76maverick]

And from one 1024E in our environment:

edit "_FlInK1_MLAG0_"
set port-selection-criteria src-dst-ip
set description ''
set mode lacp-active
set member-withdrawal-behavior block
set bundle disable
set auto-isl 1
set fortilink 0
set isl-fortilink 0
set mclag enable
set static-isl disable
set static-isl-auto-vlan enable
set members "port23" "port24"
set lacp-speed slow
set aggregator-mode bandwidth
next

[76maverick]

Our fortigate is in (HA) though the second fortigate has not been introduced. It's defacto operating in standalone regardless of the "HA Primary" shown in the gui. The issue precedes us movign the fortigate into HA - it's been going on for a long while.

We upgraded the fortigate this weekend and the network never quite recovered. We ultimately powered off one of our distro switches and things stabilized.

[76maverick · Accepted solution]

So for those wondering, and for posterity, the issue stemmed from the fact I created the MCLAG ICL between my two switches, rather than letting the switch do so itself. Initially when I configured this, the switches created their own mclag however I didn't like the name and so I recreated - or thought I did.

So we deleted the ICL mclag, rebooted one of the switches and let the network create things itself. The only action needed from my part was to ensure the proper lldp-profile was applied to the ICL ports. This ensured the ports created an MCLAG ICL trunk rather than a lag or simple ICL trunk (where one link would remain down to prevent a loop.)

Once complete I was able to turn down ports and turn them up, perform firmware upgrades and more on the l2 stack without the entire network flapping. Thanks to the Fortinet phone support person that helped with this.