FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
bshingadia_FTNT
Description
When two FortiGate units are in an HA cluster they synchronize the configuration and act as a single unit providing redundancy when one of the units fail.

Commands are set in global mode if VDOM's are in use.

The synchronization status of the two cluster units can be verified using the following command:

FGT_1# di sys ha cluster-csum

================== FG100D3G13xxxxxx =================
 is_manage_master()=0, is_root_master()=0

debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f

checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f

================== FG100D3G12xxxxxx ==================
is_manage_master()=1, is_root_master()=1

debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: d8 f5 57 46 f0 b8 45 1e 00 be 45 92 a2 07 14 90
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b

checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: d8 f5 57 46 f0 b8 45 1e 00 be 45 92 a2 07 14 90
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b

The checksum for both of the units should match for the HA to work properly.  Hence, if the checksum does not synchronize as shown in the output above the following steps can be taken to make cluster synchronize again.

Solution
ANALYSIS

How to identify which object(s) is not synchronized.  The key is to find the first out-of-sync object and correct it.  Checksums are cumulative and so all checksums which follow the first out-of-sync object will also be incorrect.

1)       Execute the following command on the slave unit to  manually synchronize the unit :
exec ha synchronize start

starting synchronize with HA master...

Verify the cluster-csum as mentioned above to see if the cluster is synchronized.

2)       If the cluster is still not synchronized execute the following command on both master and slave units and compare the output.

 

 

FGT_1:- dignose  system ha showcsum 1

system.global: d6c216d8449d75b2cd80110fa02a84e5

system.accprofile: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.npu: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.vdom-link: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.global: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.vap: cda65c180c25050eb83398fa23ab7fd1

system.switch-interface: cda65c180c25050eb83398fa23ab7fd1

system.interface: 56f2362fd69f51a2b6fc22a008c0c755

system.password-policy: 56f2362fd69f51a2b6fc22a008c0c755

system.sms-server: 56f2362fd69f51a2b6fc22a008c0c755

system.admin: af9c2b4f63e40551e33eabd64436fb3e

system.fsso-polling: af9c2b4f63e40551e33eabd64436fb3e

system.ha: ddfeff2ae037f615fbd83110169b70d2

 

FGT_2:-diagnose system ha showsum 1

system.global: d6c216d8449d75b2cd80110fa02a84e5

system.accprofile: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.npu: 7df6f055a28e5d5216c4d2c2b3ee77d1

system.vdom-link: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.global: 7df6f055a28e5d5216c4d2c2b3ee77d1

wireless-controller.vap: cda65c180c25050eb83398fa23ab7fd1

system.switch-interface: cda65c180c25050eb83398fa23ab7fd1

system.interface: 56f2362fd69f51a2b6fc22a008c0c755

system.password-policy: 56f2362fd69f51a2b6fc22a008c0c755

system.sms-server: 56f2362fd69f51a2b6fc22a008c0c755

system.admin: 18c534445cb4e9f6fccdd0a101e31e69

system.fsso-polling: 18c534445cb4e9f6fccdd0a101e31e69

system.ha: a4de1e0fcd4add6c764ed10a61b9d022

 

 

 

 Check the configuration parameter where the checksum mismatches. For e.g. the configuration checksum is mismatched in “Admin Settings” here as shown above.

Drill down to object level:

In order to find out what exactly in the admin settings is causing the issue, you can repeat the command diagnose system ha showcsum with value 2 as shown below:-

 

FGT_1# diag sys ha showcsum 2

admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5

dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d

dashboard-tabs.2: cb109689bf46e400e01aa1778d217523

dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507

dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.51: 5263dd1339cc964e43ea5e2799e53faa

 

FGT_2# diag sys ha showcsum 2

admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5

dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d

dashboard-tabs.2: cb109689bf46e400e01aa1778d217523

dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507

dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.51: 7d9d58ccb29e38e5ff6c89483effd00ca

 

 As shown, in the snippet above, there was an additional dashboard configured in the “admin” user configuration which caused the configuration to go out-of-sync. Removing that dashboard from admin user configuration causes the HA checksum to sync again.

Alternatively, to find out the source of the mismatch in the admin section command “diagnose system ha showcsum <path.object>”  can be used. This will also show the root cause of the mismatch as shown below:-

 

FGT_1# diag sys ha showcsum system.admin

admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5

dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d

dashboard-tabs.2: cb109689bf46e400e01aa1778d217523

dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507

dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.51: 5263dd1339cc964e43ea5e2799e53faa

 

FGT_2# diag sys ha showcsum system.admin

admin.admin: vdom.root: 161f1834e4f7d7cefdc65f602f8116f5

dashboard-tabs.1: 0ce0ad276cc2a7f002ce9ac0b9ad073d

dashboard-tabs.2: cb109689bf46e400e01aa1778d217523

dashboard-tabs.3: bc78e6c29c05fb268094e3ac06a6d507

dashboard-tabs.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.1: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.2: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.3: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.4: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.21: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.31: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.41: 7d9d58ccb29e38e5ff6b29c96e650b3c

dashboard.51: 7d9d58ccb29e38e5ff6c89483effd00ca

 

 

In a multi-VDOM environment, the above commands can be executed on global level and for each VDOM. The following is the syntax for the command for individual VDOM:-

Diagnose system ha showcsum <level> <vdom>      where level can be 01-04 (the leading zero is important)

e.g., diagnose system ha showcsum 01 root

Other Tips:

Manually reconfiguring the object that is out of sync should trigger re-synchronization automatically.  If necessary, you can manually trigger the start of synchronization:

execute ha synchronize start

This should cause the units to synchronize. The CLI will show a message as shown below:-
FGT_1# execute ha synchronize start

If you find the that the object in question is in fact identical on all cluster members, it may just be the checksum which needs recalculating.

diag sys ha csum-recalculate


VERIFICATION

The checksum of the two units is synchronized as can be verified below:-

FGT_1# di sys ha cluster-csum
 
================== FG100D3G13xxxxxx ==================
 
is_manage_master()=0, is_root_master()=0
debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
 
checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
 
================== FG100D3G12xxxxxx ==================
 
is_manage_master()=1, is_root_master()=1
debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
 
checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f


Contributors