Description
This article describes how to troubleshoot a checksum mismatch in a FortiGate cluster.
These HA units must be manually synchronized by detecting mismatches and correcting them using the following steps.
Scope
FortiGate.
Solution
Step 1: Check the cluster unit's checksums and compare where the mismatch is (execute the following from a global VDOM):
diagnose sys ha checksum cluster
================== FGT1 =================
is_manage_master()=0, is_root_master()=0
debugzone
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
checksum
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Not matching the secondary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
================== FGT2 ==================
is_manage_master()=1, is_root_master()=1
debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b
checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e <- Not matching primary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b
Step 2: In the above output, there is a mismatch in the global checksum output.
Execute the following command in ALL cluster units (to connect to the subordinated units, follow the steps in Technical Tip: Managing individual cluster units with the CLI command 'execute ha manage').
diagnose sys ha checksum show <VDOM_NAME>
So in this example:
diagnose sys ha checksum show global
Compare the output to find out which part of the configuration has a mismatch. The user can use a text compare tool or a diff checker tool to compare the checksums in two different files.
From v7.0, it is possible to check which checksums are not matching within FortiGate. There are two options:
Option 1: via the HA widget in the dashboard.
Hover the mouse cursor over the member that is not in sync:
Option 2: go to System -> HA.
As above, hover the mouse cursor over the member that is not in sync:
Find the differences in the actual config files and, if possible, add the missing portions on the device(s) and check if the cluster is in sync again afterwards:
diagnose sys ha checksum cluster
================== FGT1 =================
is_manage_master()=0, is_root_master()=0
Debugzone:
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Matching secondary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
Checksum:
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
================== FGT2 ==================
is_manage_master()=1, is_root_master()=1
Debugzone:
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Matching primary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
Checksum:
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
If multi-VDOM is enabled, it is necessary to execute the commands below on the Primary FortiGate for the configuration that is specific for the VDOM, to determine in which VDOM the configuration is different. Below, the checksum is used for the section 'firewall.address' of Primary FortiGate, if the FortiGate HA cluster has, for example, 'root' and 'test' VDOM.
diagnose sys ha checksum show root | grep c8f863c0ee2b47c3c06886af311450a6 <- Where 'root' is the VDOM name.
diagnose sys ha checksum show test | grep c8f863c0ee2b47c3c06886af311450a6 <- Where 'test' is the VDOM name.
If the above commands do not give any output, then the configuration change is under 'config firewall address', in global configuration.
Use the following commands to locate the specific VDOM both on the primary and secondary, then compare the tables manually:
diagnose sys ha checksum show global
diagnose sys ha checksum show root
diagnose sys ha checksum show <VDOM NAME>
The next step is to compare the configurations between the two FortiGate Firewalls in the HA cluster and update the config if needed as well as execute the below commands:
execute ha synchronize stop
execute ha synchronize start
diagnose sys ha checksum recalculate
Note:
For the object 'vpn.certificate.ca', it is possible to check the difference in 'Certificate Bundle' versions by using the following command:
diagnose autoupdate versions | grep Certificate -A 6
In some circumstances, cached items cause HA to get out of sync. Collect the cached check as below and compare:
diagnose sys ha checksum cluster
diagnose sys ha checksum cached global
diagnose sys ha checksum cached root <----- VDOM name.
If the cached objects are mismatched, rebooting the secondary will resolve the issue.
Note: If using a FortiGate-120G with v7.2.9, a known bug (1056138) is encountered.
Which behaviour yields the following results:
get system ha status
diagnose sys ha status
chksum dump: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =====> for secondary checksum all in 00
================== FG120GXXXXXXXXXXXXXX================== <----- checksum cluster for Secondary is empty after serial number.
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
The issue is scheduled to be resolved in versions 7.2.11 and 7.6.1.
There are 2 workarounds:
- Use another port instead of a dedicated HA.
- Roll back to the previous version.
Currently, this is only presented for FortiGate-120G using v7.2.9 and v7.2.10.
Note:
v5.0 up to v6.4 are out of engineering support. So these commands might be different on higher versions. Consider upgrading the firmware level on the device to a supported version (v7.0 up to v7.6). Here, check the firmware path and compatibility depending on the hardware.
Related articles:
Technical Tip: Understanding Configuration Checksum on FortiGate
