Description
This article describes how to troubleshoot a checksum mismatch in a FortiGate cluster.
These HA units must be manually synchronized by detecting mismatches and correcting them using the following steps.
Scope
FortiGate.
Solution
Step 1: Check the cluster unit's checksums and compare where the mismatch is:
diag sys ha checksum cluster
================== FGT1 =================
is_manage_master()=0, is_root_master()=0
debugzone
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
checksum
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Not matching the secondary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
================== FGT2 ==================
is_manage_master()=1, is_root_master()=1
debugzone
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b
checksum
global: 89 f2 f0 0b e8 eb 0d ee f8 55 8b 47 27 7a 27 1e <- Not matching primary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: a7 8d cc c7 32 b5 81 a2 55 49 52 21 57 f9 3c 3b
Step 2: In the above output, there is a mismatch in the global checksum output.
Execute the following command in ALL cluster units (to connect to the subordinated units, follow the steps in this article).
diag sys ha checksum show <VDOM_NAME>
So in this example:
diag sys ha checksum show global
Next, compare the output to find out which part of the configuration has a mismatch.
From FortiOS 7.0, it is possible to check which checksums are not matching within FortiGate. There are two options:
Option 1: via the HA widget in the dashboard.
Hover the mouse cursor over the member that is not in-sync:
Option 2: go System -> HA.
As above, hover the mouse cursor over the member that is not in-sync:
Now, find the differences in the actual config files and, if possible, add the missing portions on the device(s) and check if the cluster is in sync again afterwards:
diag sys ha checksum cluster
================== FGT1 =================
is_manage_master()=0, is_root_master()=0
debugzone
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Matching secondary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
checksum
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
================== FGT2 ==================
is_manage_master()=1, is_root_master()=1
debugzone
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11 <- Matching primary.
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
checksum
global: : 79 24 76 8a a8 03 9a 81 dc c4 3c f8 96 72 59 11
root: cf 85 55 fe a7 e5 7c 6f a6 88 e5 a9 ea 26 e6 92
all: f4 62 b2 ce 81 9a c9 04 8f 67 07 ec a7 44 60 1f
If multi-VDOM is enabled, it is necessary to execute below commands on the Primary FortiGate for the configuration which is specific for the VDOM, in order to determine in which VDOM, is the configuration different. Below, the checksum is used for section 'firewall.address' of Primary FortiGate, if the FortiGate HA cluster has for example: 'root' and 'test' VDOM.
diagnose sys ha checksum show root | grep c8f863c0ee2b47c3c06886af311450a6 <- Where 'root' is the VDOM name.
diagnose sys ha checksum show test | grep c8f863c0ee2b47c3c06886af311450a6 <- Where 'test' is the VDOM name.
If the above commands do not give any output, then the configuration change is under 'config firewall address', in global configuration.
Use the following commands to locate the specific VDOM both on the primary and secondary, then compare the tables manually:
diagnose sys ha checksum show global
diagnose sys ha checksum show root
diagnose sys ha checksum show <VDOM NAME>
The next step is to compare the configurations between the two FortiGate Firewalls in the HA cluster and update the config if needed as well as execute the below commands:
execute ha synchronize stop
execute ha synchronize start
diagnose sys ha checksum recalculate
Related article:
