FortiSwitches are status not stable appearing onlin/offline frequentl. High CPU usage on Fortilink.

[ahamza89]

I have a Fortigate 601E v6.4.9 as a switch controller, with 2 FotiSwitch 1048E as MCLAG peers and multiple 124F on distribution.

Few switches suddenly went offline including 1x 1048E.

in LOGS it shows 

NTP is fine.

[Adolfo_Z_H]

the most common reason for this issue is a network loop caused for cabling issues and misconfiguration on the MCLAG - ICL link between peers or between Tier1 and Tier2 switches.

Please open a TAC ticket if you need still support.

[ahamza89]

Thanks for your reply.. how can I figured it out. Any possibility through CLI. 

[Adolfo_Z_H]

i am afraid this is too complex to check it using forum tools. Seems your networks is some kind complex.

there is some usefull comands on FGT side

execute switch-controller diagnose-connection

execute switch-controller get-conn-status

(take note of the fortilink interface/stack name)

execute switch-controller get-physical-conn standard FortiSwitch-Stack-FortiLink

Please check with this last command the name of the fortilink stack and change if it is necessary

diagnose switch-controller switch-info mclag peer-consistency-check

on FSW side you can try to check for non planed ports on STP blocked or backup status

diagnose stp instance list

also check if your intended network topology is matching LLDP outputs.

get switch lldp neighbors-summ

this command is assuming all relevant devices have LLDP enabled. Per default it is enabled on all FSW devices, but be aware some other 3rd party devices may not have it enabled, so doble check with STP diagnose command if these ports are not blocked.

i encourage you if this is an urgent matter, get in contact with TAC support.

[ahamza89]

thanks. Will check these cmds. Everything was smooth from last 3/4 months. Don’t know suddenly it went like this. Will share the output.

[ahamza89]

Below the results from Fortigate

FortiGate-601E # execute switch-controller diagnose-connection

Fortilink interface ... OK
fortilink enabled

DHCP server ... OK
fortilink enabled

NTP server ... OK
fortilink enabled
NTP server sync ... OK
synchronized: no, ntpsync: enabled, server-mode: enabled

ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:98
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:486
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:587
no data
ipv6 server(.ntp.org) unresolved -- unreachable(0xff) S:0 T:1
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:926
no data

HA mode ... disabled

#######################################################
*******************************************************

execute switch-controller get-conn-status

only core switches are Authorized/UP all distribution switches are Authorized/Down

####################################################
***************************************************

FortiGate-601E # execute switch-controller get-physical-conn standard fortilink
This will display connectivity graph information for FortiLink from FortiGate's perspective
NOTE : If FortiSwitch is not authorized, no connectivity information will be shown
NOTE : If FortiSwitch is in idle state, no connectivity information will be shown
NOTE : If FortiSwitch ISL peer has inconsistent info, no connectivity information will be shown

FortiLink interface : fortilink

FortiGate(s)
FG6H1ETB2*******(x2) <<------------------>> FS1E48T****00213(port48)

Tier 1
FS1E48T****00213(port48) <<------------------>> FG6H1ETB*****(x2)

Tier 2+
FS1E48T*****242(port45/E48T4210****-0) <<------------------>> FS1E48T*****213(port45/E48T4210****-0)

############################################
********************************************

FortiGate-601E # diagnose switch-controller switch-info mclag peer-consistency-check
Vdom: root
Managed Switch : FS1E48T****213 0

Running diagnostic, it may take sometime...

** Comparing "switch.global.mclag-split-brain-detect" config ....OK
** Comparing "switch.global.mclag-split-brain-all-ports-down" config ....OK

mclag-trunk-name peer-config lacp-state stp-state local-ports remote-ports
__________________ ___________ __________ _________ _____________ _____________

AGG-Y NOT-FOUND UNEVEN MISMATCH port1 port2
port3 port4
port5 port6
port7 port8
port10 port11
port12 port16

E48T*****242-0* OK UP OK port45 port18 port45
G6H1ET****058 OK UP OK port48 port48

switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
switch:S124F does not support MCLAG feature!
Managed Switch : FS1E48T*****242 0

Running diagnostic, it may take sometime...

** Comparing "switch.global.mclag-split-brain-detect" config ....OK
** Comparing "switch.global.mclag-split-brain-all-ports-down" config ....OK

mclag-trunk-name peer-config lacp-state stp-state local-ports remote-ports
__________________ ___________ __________ _________ _____________ _____________

AGG-F1 NOT-FOUND UNEVEN MISMATCH port1
AGG-X NOT-FOUND UNEVEN MISMATCH port2 port3
port4 port5
port6 port7
port8 port9
port10 port12
port13 port14
port15 port16

E48T******213-0* OK UP OK port18 port45 port45
G6H1ET******058 OK UP OK port48 port48

[ahamza89]

[ahamza89]

What OS are stable for this. Currently :

Fortigate on 7.0.0

Forti SW 6.4.7/7.2.1

[sachitdas_FTNT]

Hi,

We recommend FSWs to be on latest version and all FSWs to be on same version. So, if you are having FSWs in 6.4.x, upgrade to 6.4.11.  Similarly if 7.2, then 7.2.1 on all FSWs.

When any FSW goes offline and comes back online, check the FSW logs for any log.

On FSW CLI:-

exec log filter view-lines 1000

exec log display

check for crash - diag debug crashlog read

Also, there could be a possibility where due to any one FSW in network, it could cause issue on all FSWs which causes the fortilink process to go high. So as first step, i recommend upgrading FSWs.

[Adolfo_Z_H]

we encourage to use any R marked combo mentioned on this KB

https://docs.fortinet.com/document/fortiswitch/7.2.1/fortilink-compatibility

about your issue is it very likely due your FGT lost NTP sync

FortiGate-601E # execute switch-controller diagnose-connection

Fortilink interface ... OK
fortilink enabled

DHCP server ... OK
fortilink enabled

NTP server ... OK
fortilink enabled
NTP server sync ... OK
synchronized: no, ntpsync: enabled, server-mode: enabled

ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:98
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:486
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:587
no data
ipv6 server(.ntp.org) unresolved -- unreachable(0xff) S:0 T:1
no data
ipv4 server(.ntp.org) -- unreachable(0x0) S:7 T:926
no data

HA mode ... disabled

it is critical to build management tunnels to have properly configured on FGT

an external NTP server.

as time drift increases between FGT and FSW units, management tunnels will lost and unable to come up.

use this kb to solve NTP issues on FGT side

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Troubleshoot-NTP-synchronization-issue/ta-...

this will likely solve "down" issue on access FSW.