Troubleshooting Tip: How to view Fabric switch Heart Beat on chassis 7K

[adecottignies_FTNT]

Description	This article describes how to see Fabric Heartbeat to troubleshoot the Heartbeat issue between FPM and FIM in a FortiGate 7K Chassis.
Scope	FortiGate 7000F – FortiGate 7000E.
Solution	The command diagnose load-balance status provides an output as follows:  FortiGate7K (global) # diagnose load-balance status ========================================================================== (…) Slot X Status:Dead Function:Active Link: Base: Up Fabric: Up Heartbeat: Management: Good Data: Failed Status Message:"Waiting for data heartbeat." (…) Inside a FortiGate 7K Chassis, two internal switches coexist : The Base backplane: Used for management and administration inside the Chassis. The Fabric backplane (also named Data backplane): Handle the user traffic, where the data transits. Each FPM exchanges management heartbeats with both FIM1 / FIM2 simultaneously using respectively its interface f-slot1 and f-slot2. The Master FIM will use the interface f-slotX where X is the FIM number. For example on the Master FIM f-slot3 is used to communicate with FPM3, and f-slot4 for FPM4. Packets are ethertype 0x8990 sent using an Ethernet multicast 01:80:c2:00:00:0c destination MAC. The following diagram details the Fabric Backplane: To see the fabric heartbeat, from CLI :  Fortigate-7K [FIM01] # config global Fortigate-7K [FIM01] (global) # diagnose hardware deviceinfo nic f-slot3 ========================================================================== Current slot: 1  Module SN: FIM01E3E12345678 Description             FGT-7000E Ethernet Driver Driver Name             FGT-7000E Ethernet Driver System_Device_Name      f-slot3 Current_HWaddr          02:6c:ac:11:22:33 Permanent_HWaddr        02:6c:ac:11:22:33 <----------------- (…) The MAC address of f-slot3 is 02:6c:ac:11:22:33. It is now necessary to know the MAC address used by the FPM that needs to be verified.  For example, to check the FPM3 : Fortigate-7K [FIM01] (global) # execute load-balance slot manage 3 <enter credential> Fortigate-7K [FPM03] $ config global Fortigate-7K [FPM03] (global) $ diagnose hardware deviceinfo nic f-slot1 Description             FGT-7000E Ethernet Driver Driver Name             FGT-7000E Ethernet Driver System_Device_Name      f-slot1 Current_HWaddr          02:4c:a5:99:88:77 Permanent_HWaddr        02:4c:a5:99:88:77  <----------------- (…) The MAC address of f-slot1 of FPM3 is 02:4c:a5:99:88:77. The FPM can be left with CTRL+D. The next step will be done on the master FIM, inside mgmt-vdom.  Fortigate-7K [FIM01] (global) # end Fortigate-7K [FIM01] # config vdom Fortigate-7K [FIM01] (vdom) # edit mgmt-vdom current vf=mgmt-vdom:2 Fortigate-7K [FIM01] (mgmt-vdom) # The heartbeats will be seen through a sniffer : Fortigate-7K [FIM01] (mgmt-vdom) # diagnose sniffer options filter-out-internal-pkts disable Fortigate-7K [FIM01] (mgmt-vdom) # diagnose sniffer packet f-slot3 ‘’ 6 0 l [FIM01] 2024-02-13 16:27:54.990378 f-slot3 -- Ether type 0x8990 printer hasn't been added to sniffer. 0x0000   0180 c200 000c 024c a599 8877 8990 01a7        .......L.-...... 0x0010   0000 0003 0701 0000 0000 0000 0000 0000        ................ 0x0020   0000 0000 0000 0000 0000 0000 0000 0000        ................ (…) [FIM01] 2024-02-13 16:27:55.087418 f-slot3 -- 802.1AD vlan#41 P0 0x0000   0180 c200 000c 026c ac11 2233 88a8 0029        .......l...f...) 0x0010   8990 0047 0500 0003 0700 0000 0003 0000        ...G............ 0x0020   0003 0000 0004 0000 0000 0000 0000 0000        ................ (…) To stop the sniffer use Ctrl+C. The sniffer above shows that heartbeat packets are correctly sent and received on the Master FIM.  The same will be done on the FPM. On the FPM, it is the interface f-slot1 that handles this Heartbeat traffic. Fortigate-7K [FIM01] (global) # execute load-balance slot manage 3. <enter credentials> Fortigate-7K [FPM03] $ config vdom Fortigate-7K [FPM03] (vdom) $ edit mgmt-vdom current vf=mgmt-vdom:2 Fortigate-7K [FPM03] (mgmt-vdom) $ Fortigate7k [FPM03] (mgmt-vdom) $ diagnose sniffer packet f-slot1 '' 6 0 l [FPM03] 2024-02-13 16:45:42.860050 f-slot1 -- Ether type 0x8990 printer hasn't been added to sniffer. 0x0000   0180 c200 000c 024c a599 8877 8990 01a7        .......L.-...... 0x0010   0000 0003 0701 0000 0000 0000 0000 0000        ................ 0x0020   0000 0000 0000 0000 0000 0000 0000 0000        ................ (…) [FPM03] 2024-02-13 16:45:43.238751 f-slot1 -- Ether type 0x8990 printer hasn't been added to sniffer. 0x0000   0180 c200 000c 026c ac11 2233 8990 0047        .......l...f...G 0x0010   0500 0003 0700 0000 0003 0000 0003 0000        ................ 0x0020   0004 0000 0000 0000 0000 0000 0000 0000        ................ (…) To stop the sniffer use Ctrl+C. These packets should appear both ways every second. It means that FIM1 and FPM3 are correctly exchanging the management heartbeat. Note that if the command 'execute load-balance slot manage' is not working, use the console port from SMM to reach the FPM :  Using the console ports Related documents:  Technical Note: Verification of SLBC status before an upgrade  Troubleshooting Tip: FortiGate 7000 Series blade config synchronization issues (confsync) Technical Tip: How to find the config difference between blades in 6K/7K Chassis using 'diagnose sys... Technical Tip: FortiGate-6000/7000 Chassis health check commands  Using the console ports