Troubleshooting Note : FortiGate HA synchronization messages and cluster verification steps

[rmetzger]

Description

This article describes a simple procedure to verify if FortiGate devices in an HA cluster are all synchronized.

Note that all commands are passed in global mode if VDOMs are enabled (as shown in the following examples).

The following commands are listed in this article:

• get system ha status
• diagnose sys ha showcsum
• execute ha synchronize config
• execute ha manage <id>

Reminder: The following command can be used to connect to the Slave device CLI from the Master CLI:

FGT300-5 (global) # execute ha manage <id>
....where <id> is the the subsidiary unit listed with the command "execute ha manage ?"


Step 1

At the initial HA configuration, any new device that joins a cluster in a Slave role will display the following message sequence on the console. This will indicate a successful cluster formation.

FGT300-2 login:
slave's configuration is not in sync with master's, sequence:0
slave's configuration is not in sync with master's, sequence:1
slave's configuration is not in sync with master's, sequence:2
slave's configuration is not in sync with master's, sequence:3
slave's configuration is not in sync with master's, sequence:4
slave starts to sync with master
logout all admin users
slave succeeded to sync with master

Step 2

On an operational HA cluster, the following commands will allow verification of the HA status:

2.1 : Output example from the Master

FGT300-5 (global) # get system ha status

Model: 300
Mode: a-p
Group: 30
Debug: 0
ses_pickup: disable
Master:200 FGT300-5 FG300A3906550380 0
Slave :128 FGT300-2 FG300A2904500186 1
number of vcluster: 1
vcluster 1: work 169.254.0.1
Master:0 FG300A3906550380
Slave :1 FG300A2904500186

2.2 : Output example from the Slave

FGT300-2 (global) # get system ha status

Model: 300
Mode: a-p
Group: 30
Debug: 0
ses_pickup: disable
Slave :128 FGT300-2 FG300A2904500186 1
Master:200 FGT300-5 FG300A3906550380 0
number of vcluster: 1
vcluster 1: standby 169.254.0.1
Slave :1 FG300A2904500186
Master:0 FG300A3906550380

Step 3

On an operational HA cluster, the following commands will allow verification of all devices which have got the same configuration

The following example shows a FortiGate running with multiple VDOMs, and the configuration checksum being similar on both devices for all of the VDOMs.

3.1 : Getting the HA checksums on the Master

FGT300-5 (global) # diagnose sys ha showcsum

is_manage_master()=1, is_root_master()=1
debugzone
global: e5 45 87 ff 9d 4b d5 dc 37 98 ce bd 53 c0 75 70
root: f3 a7 72 9a f8 8a 42 f3 80 77 89 a3 eb d9 09 2b
LAN: a5 f8 cf 4c 98 3b 25 b7 22 3b 17 f6 76 8e b0 3c
INTERNET: f9 32 66 b4 d6 6d 2e 0a 42 59 11 c2 4c 85 53 f8
DMZ: 30 96 97 69 ff 07 32 bd 6c 84 0c 5c 4a 13 78 92
all: 4b a1 24 73 2b 3a 86 71 a8 9a 98 22 15 1c 76 65

checksum
global: e5 45 87 ff 9d 4b d5 dc 37 98 ce bd 53 c0 75 70
root: f3 a7 72 9a f8 8a 42 f3 80 77 89 a3 eb d9 09 2b
LAN: a5 f8 cf 4c 98 3b 25 b7 22 3b 17 f6 76 8e b0 3c
INTERNET: f9 32 66 b4 d6 6d 2e 0a 42 59 11 c2 4c 85 53 f8
DMZ: 30 96 97 69 ff 07 32 bd 6c 84 0c 5c 4a 13 78 92
all: 4b a1 24 73 2b 3a 86 71 a8 9a 98 22 15 1c 76 65

3.2 : Getting the HA checksums on the Slave (and compare with the Master):

FGT300-2 (global) # diagnose sys ha showcsum

is_manage_master()=0, is_root_master()=0
debugzone
global: e5 45 87 ff 9d 4b d5 dc 37 98 ce bd 53 c0 75 70
root: f3 a7 72 9a f8 8a 42 f3 80 77 89 a3 eb d9 09 2b
LAN: a5 f8 cf 4c 98 3b 25 b7 22 3b 17 f6 76 8e b0 3c
INTERNET: f9 32 66 b4 d6 6d 2e 0a 42 59 11 c2 4c 85 53 f8
DMZ: 30 96 97 69 ff 07 32 bd 6c 84 0c 5c 4a 13 78 92
all: 4b a1 24 73 2b 3a 86 71 a8 9a 98 22 15 1c 76 65

checksum
global: e5 45 87 ff 9d 4b d5 dc 37 98 ce bd 53 c0 75 70
root: f3 a7 72 9a f8 8a 42 f3 80 77 89 a3 eb d9 09 2b
LAN: a5 f8 cf 4c 98 3b 25 b7 22 3b 17 f6 76 8e b0 3c
INTERNET: f9 32 66 b4 d6 6d 2e 0a 42 59 11 c2 4c 85 53 f8
DMZ: 30 96 97 69 ff 07 32 bd 6c 84 0c 5c 4a 13 78 92
all: 4b a1 24 73 2b 3a 86 71 a8 9a 98 22 15 1c 76 65

Any checksum difference between Master and Slave will depict a synchronization problem. Configuration synchronization can be forced with the command:

FGT300-5 (global) # execute ha synchronize config

Should any further problems be experienced, it is recommend to open a ticket with the Fortinet TAC and attach the information that has been gathered.
Scope

FortiOS 3.0

FortiOS 4.0 and above

Related Articles

Troubleshooting Note : Fortigate HA message "HA master heartbeat interface intf_name lost neighbor i...

List of most popular articles related to Troubleshooting

Connecting to an HA slave unit with the CLI command "execute ha manage" brings into the HA VDOM "vsy...