fortiswitches show offline in fortigate - FortiLink - MCLAG config

[UnderscoresAndDashes]

              I recently deployed 11 Fortiswitches into an environment. I am using 6 248E-FPOE's, 1 124E-FPOE, and 4 108F-FPOE's. I have 2 of the 248's setup in a MCLAG config and the reset are downstream. The issue I am having is that after initial deployment, all switches were online and doing what they are supposed to do, but now I have 8 switches, including one of the Lag switches showing offline. I run the following commands to get status' and this is what I see for the Lag switch that is offline. 

FortiGate Firmware 7.0.11

Switch Firmware 6.2.3

More info:

MCLAG config on switch that is online

edit "port50"
set speed 1000full
set vlan "_default"
set allowed-vlans "quarantine"
set untagged-vlans "quarantine"
set lldp-profile "default-auto-mclag-icl"
set export-to "root"
set mac-addr xxxxxx

MCLAG config on switch that is Offline

edit "port50"
set speed 1000full
set vlan "_default"
set allowed-vlans "quarantine"
set untagged-vlans "quarantine"
set lldp-profile "default-auto-mclag-icl"
set export-to "root"
set mac-addr ac:xxxxxxx
next

Fortilink interface config. 

config system interface
edit "fortilink"
set vdom "root"
set fortilink enable
set ip 10.255.1.1 255.255.255.0
set allowaccess ping fabric
set type aggregate
set member "port19" "port20"
set lldp-reception enable
set lldp-transmission enable
set snmp-index 34
set auto-auth-extension-device enable
set fortilink-split-interface disable
set switch-controller-nac "fortilink"
set switch-controller-dynamic "fortilink"
set swc-first-create 255

Fortilink Debug

diagnose netlink aggregate name fortilink

LACP flags: (A|P)(S|F)(A|I)(I|O)(E|D)(E|D)
(A|P) - LACP mode is Active or Passive
(S|F) - LACP speed is Slow or Fast
(A|I) - Aggregatable or Individual
(I|O) - Port In sync or Out of sync
(E|D) - Frame collection is Enabled or Disabled
(E|D) - Frame distribution is Enabled or Disabled

status: up
npu: y
flush: n
asic helper: y
oid: 91
ports: 2
link-up-delay: 50ms
min-links: 1
ha: master
distribution algorithm: L4
LACP mode: active
LACP speed: slow
LACP HA: enable
aggregator ID: 2
actor key: 17
actor MAC address: xxxxxxxxx
partner key: 17
partner MAC address: ac:xxxxxxxxx

slave: port19
index: 0
link status: up
link failure count: 0
permanent MAC addr: xxxxxxxxx
LACP state: negotiating
actor state: ASAODD
actor port number/key/priority: 1 17 255
partner state: ASAIDD
partner port number/key/priority: 1 17 255
partner system: 0 ac:xxxxxxxxx
aggregator ID: 1
speed/duplex: 1000 1
RX state: CURRENT 6
MUX state: WAITING 2

slave: port20
index: 1
link status: up
link failure count: 0
permanent MAC addr: xxxxxxxx
LACP state: established
actor state: ASAIEE
actor port number/key/priority: 2 17 255
partner state: ASAIEE
partner port number/key/priority: 1 17 255
partner system: 0 ac:xxxxxxx
aggregator ID: 2
speed/duplex: 1000 1
RX state: CURRENT 6
MUX state: COLLECTING_DISTRIBUTING 4

execute switch-controller diagnose-connection S248EFTFxxxxxxxx

Fortilink interface ... OK
fortilink enabled

DHCP server ... OK
fortilink enabled

NTP server ... OK
fortilink enabled
NTP server sync ... OK
synchronized: yes, ntpsync: enabled, server-mode: enabled

ipv4 server(ntp1.fortiguard.com) 208.91.112.61 -- reachable(0xff) S:5 T:266
server-version=4, stratum=2
reference time is e7ed2c45.766816f3 -- UTC Fri Apr 21 15:37:41 2023
clock offset is 0.010927 sec, root delay is 0.067551 sec
root dispersion is 0.010559 sec, peer dispersion is 374 msec

ipv4 server(ntp2.fortiguard.com) 208.91.112.62 -- reachable(0xef) S:5 T:272
server-version=4, stratum=2
reference time is e7ed2c45.766816f3 -- UTC Fri Apr 21 15:37:41 2023
clock offset is 0.018932 sec, root delay is 0.067551 sec
root dispersion is 0.010574 sec, peer dispersion is 222 msec

ipv4 server(ntp2.fortiguard.com) 208.91.112.60 -- reachable(0xff) S:5 T:257 selected
server-version=4, stratum=2
reference time is e7ed2c45.766816f3 -- UTC Fri Apr 21 15:37:41 2023
clock offset is 0.017357 sec, root delay is 0.067551 sec
root dispersion is 0.010544 sec, peer dispersion is 276 msec

ipv4 server(ntp1.fortiguard.com) 208.91.112.63 -- reachable(0xff) S:5 T:253
server-version=4, stratum=2
reference time is e7ed2c45.766816f3 -- UTC Fri Apr 21 15:37:41 2023
clock offset is 0.013255 sec, root delay is 0.067551 sec
root dispersion is 0.010544 sec, peer dispersion is 173 msec

HA mode ... disabled

Fortilink
Status ... SWITCH_AUTHORIZED_READY
Last keepalive ... 2 seconds ago

No CAPWAP IP address retrieved for FortiSwitch S248EFTFxxxxxxxx
CAPWAP
Remote Address : N/A
Status ... Idle

# config system ntp

 (ntp) # show
config system ntp
set ntpsync enable
set server-mode enable
set interface "fortilink"
end

 (ntp) # get
ntpsync : enable
type : fortiguard
syncinterval : 60
source-ip : 0.0.0.0
source-ip6 : ::
server-mode : enable
authentication : disable
interface : "fortilink"

Why do they lose connection to the fortigate switch management? 

[gfleming]

Can you show the topology from the Managed FortiSwitches console?

Can you show your FortiLink interface configuration?

[UnderscoresAndDashes]

FortiLink Config: I took out the ipv6section for space. 

config system interface
edit "fortilink"
set vdom "root"
set vrf 0
set fortilink enable
set switch-controller-source-ip outbound
set mode static
set dhcp-relay-interface-select-method auto
set dhcp-relay-service disable
set ip 10.255.1.1 255.255.255.0
set allowaccess ping fabric
set fail-detect disable
set pptp-client disable
set arpforward enable
set broadcast-forward disable
set bfd global
set l2forward disable
set icmp-send-redirect enable
set icmp-accept-redirect enable
set reachable-time 30000
set vlanforward disable
set stpforward disable
set ips-sniffer-mode disable
set ident-accept disable
set ipmac disable
set status up
set netbios-forward disable
set wins-ip 0.0.0.0
set type aggregate
set netflow-sampler disable
set sflow-sampler disable
set src-check enable
set sample-rate 2000
set polling-interval 20
set sample-direction both
set explicit-web-proxy disable
set explicit-ftp-proxy disable
set proxy-captive-portal disable
set tcp-mss 0
set inbandwidth 0
set outbandwidth 0
set egress-shaping-profile ''
set ingress-shaping-profile ''
set weight 0
set external disable
set trunk disable
set member "port19" "port20"
set description ''
set alias ''
set l2tp-client disable
set device-identification disable
set lldp-reception enable
set lldp-transmission enable
set lldp-network-policy ''
set estimated-upstream-bandwidth 0
set estimated-downstream-bandwidth 0
set measured-upstream-bandwidth 0
set measured-downstream-bandwidth 0
set bandwidth-measure-time 0
set monitor-bandwidth disable
set vrrp-virtual-mac disable
set role undefined
set snmp-index 34
set secondary-IP disable
set preserve-session-route disable
set auto-auth-extension-device enable
set ap-discover enable
set fortilink-neighbor-detect fortilink
set ip-managed-by-fortiipam disable
set fortilink-split-interface disable
set switch-controller-mgmt-vlan 4094
set switch-controller-igmp-snooping-proxy disable
set switch-controller-igmp-snooping-fast-leave disable
set switch-controller-nac "fortilink"
set switch-controller-dynamic "fortilink"
set switch-controller-iot-scanning disable
set swc-first-create 255
set priority 1
set dhcp-relay-request-all-server disable
set dhcp-client-identifier ''
set dhcp-renew-time 0
set idle-timeout 0
set disc-retry-timeout 1
set padt-retry-timeout 1
set dns-server-override enable
set dns-server-protocol cleartext
set mtu-override disable
set wccp disable
set drop-overlapped-fragment disable
set drop-fragment disable
set lacp-mode active
set lacp-ha-slave enable
set system-id-type auto
set lacp-speed slow
set min-links 1
set min-links-down operational
set algorithm L4
set link-up-delay 50
next
end

[gfleming]

You need to enable split-interface on the FortiLink interface:

https://docs.fortinet.com/document/fortiswitch/7.2.4/fortilink-guide/780635/switch-redundancy-with-m...

And do you have an ICL between the two MC-LAG switches?

[UnderscoresAndDashes]

It literally says "Disable the split interface in the FortiLink interface"

 

MCLAG config on switch that is Offline

edit "port50"
set speed 1000full
set vlan "_default"
set allowed-vlans "quarantine"
set untagged-vlans "quarantine"
set lldp-profile "default-auto-mclag-icl"
set export-to "root"
set mac-addr ac:xxxxxxx
next

[gfleming]

Oops my bad. Got it mixed up sorry bout that. Either way, I assumed you followed all of the instructions in that doc for setting up the MCLAG environment? i.e. mclag-stp-aware is enabled? STP is enabled on ICL trunks etc.

It's odd that you don't see the ICL in the topology view. Is it showing as up on the switch side?

[UnderscoresAndDashes]

I can't find were that would be at. It say on the global switch level, I am guessing that is: config switch-controller global ??

I don't see it in the show full, though the instructions do say it is enabled by default. 

[gfleming]

Not sure i don't have an mc-lag deployment in front of me atm.

What about the ICL link tho? Is it showing up on the switch side?

[UnderscoresAndDashes]

I think because one of the core switches are showing offline, it won't show that link because it can't get info from the other core. Not sure what to do. Everything connected to it is passing traffic, but can't explain why they are showing offline. Most of the Down stream switches are connected to that offline coreswitch so that explains why so many show offline. If I can get that one back online, then the rest would show up I bet. 

[gfleming]

So that's why i'm wondering if you see the ICL as being up from the switch side of things. If the ICL is down then that explains why the other switch is offline (to prevent split brain).

Also are your downstream switches not dual-homed to the two MC-LAG switches? They should all still be online with a single link....