Description
This article explains how to allocate config disparity for HA out-of-sync and resolve the problem.
Scope
FortiGate.
Solution
FG01 # get sys ha status
HA Health Status: OK
Model: FortiGate-1500D
Mode: HA A-P
--- truncated ----
Configuration Status:
FG1K5D3I12XXXXXX(updated 2 seconds ago): in-sync
FG1K5D3I13XXXXXX(updated 2 seconds ago): out-of-sync
--- truncated ----
- Allocate in which VDOM the disparity is located:
FG01 # diag sys ha checksum cluster
================== FG1K5D3I12XXXXXX ==================
is_manage_primary()=1, is_root_primary()=1
debugzone
global: 46 aa d3 d9 7d 7f dd 84 88 d2 fb 9c 12 80 25 f0
root: ee 00 e3 4d d1 76 b1 11 52 40 e2 37 f0 d6 8b 25
all: e2 0d a3 c4 e3 d4 fa 7e e0 72 ac 7e 77 8a 9b 3d
checksum
global: 46 aa d3 d9 7d 7f dd 84 88 d2 fb 9c 12 80 25 f0
root: ee 00 e3 4d d1 76 b1 11 52 40 e2 37 f0 d6 8b 25 <----- Disparity is here.
all: e2 0d a3 c4 e3 d4 fa 7e e0 72 ac 7e 77 8a 9b 3d
================== FG1K5D3I13XXXXXX ==================
is_manage_primary()=0, is_root_primary()=0
debugzone
global: 46 aa d3 d9 7d 7f dd 84 88 d2 fb 9c 12 80 25 f0
root: 93 48 eb 9e 1b c8 88 a1 9d 89 ff c6 ff 80 3b b9
all: 8c a1 bc 71 a5 0b 64 31 1f 14 72 93 c8 d0 bf 6f
checksum
global: 46 aa d3 d9 7d 7f dd 84 88 d2 fb 9c 12 80 25 f0
root: 93 48 eb 9e 1b c8 88 a1 9d 89 ff c6 ff 80 3b b9 <----- Disparity is here.
all: 8c a1 bc 71 a5 0b 64 31 1f 14 72 93 c8 d0 bf 6f
*all: Means total checksums calculation, it is possible to ignore this and focus on VDOM 'root' or global: in case multi-vdoms are set, so there will be more VDOMs listed.
-
Perform diagnose sys ha checksum show <vdom-name>: On both firewalls and make separate files for them, to make it easier to compare.
It is possible to use Notepad++ with the 'compare' plugin, to help to find the disparity config.
Firewall 01:
FG01 # diagnose sys ha checksum show root
--- truncated ----
firewall.address: dbf29408b258a4df40a203a7c9a74b8f
firewall.multicast-address: a00a0b721b4ca3cda2759ed08a6523e1
--- truncated ----Firewall 02
FG01 # diagnose sys ha checksum show root
--- truncated ----
firewall.address: dbf29408b258a4df40a203a7c9a74b7f
firewall.multicast-address: a00a0b721b4ca3cda2759ed08a6522e1
--- truncated ---- -
Show the config for each section on both firewalls and compare the real config:
FG01 # show firewall address <----- It is easy to associate the checksum result and the real config.
config firewall address
edit "none"
set uuid c4bf90e6-c568-51ed-8ab3-94e1853db160
set subnet 0.0.0.0 255.255.255.255
--- truncated ----
FG01 # show firewall multicast-address
config firewall multicast-address
edit "all"
set start-ip 224.0.0.0
set end-ip 239.255.255.255
--- truncated ----
Now it is necessary to manually alter the config.
The easiest way to match the configuration and make the out-of-sync error message disappear is to take the configuration of that specific table (which has the mismatch) from the primary unit and paste into the secondary unit. Here is an example of how to do it:
For example if firewall.address table is out of sync as presented above, it is possible to open the CLI of the primary unit and run:
config firewall address
show full
It will show a similar output as below:
Copy these outputs from the primary unit (it is possible to copy them all together, just make sure to scroll until the end).
Paste the copied outputs in the secondary unit CLI and save the configuration by typing:
end
Wait for a couple of minutes and it should sync.
It is necessary to repeat these steps for each mismatched table.
For the example above, there are 2 mismatched tables firewall.address and firewall.multicast-address.
It is necessary to do the same step for firewall.multicast-address.
If it is possible to reboot the secondary unit it can also cause the out-of-sync error to disappear.
Important Tip:
- No matter on primary or secondary the config is alterd, the latest config will be synchronized and considered as the latest.
- This method should be taken as the last resort, when the FortiGate config cannot get in sync after some time, like 30 minutes or more. For large disparity, it will require longer for the config to get fully in sync.
- After the upgrade, in case there is this issue, it is always recommended to not immediately take this action, but to wait for like 30 minutes, to give cluster time for the synchronization process.
- Due to the huge disparity, it is almost impossible to perform this manual intervention. Check first if the sync process is working properly.
Refer to this KB article to check if the sync system works normally:
Troubleshooting Tip: How to troubleshoot HA synchronization issue using GUI
Related document:
HA active-passive cluster setup | FortiGate / FortiOS 7.2.4 (fortinet.com)
