Description
This article describes advanced troubleshooting for High Availability Cluster and collects information to deliver to Fortinet TAC for a support ticket.
Scope
FortiGate, HA.
Solution
- Obtain General HA information in the Primary unit:
get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by vcluster
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster
show full-configuration sys ha
diagnose debug crashlog read
- Recalculate HASH values for all configurations in the Primary unit:
diag sys ha checksum recalculate global
diag sys ha checksum recalculate roo
diag sys ha checksum recalculate <VDOM_name> <----- Write name of each VDOM.
- Obtain HASH values for all configurations in the Primary unit:
diag sys ha checksum show global
diag sys ha checksum show root
diag sys ha checksum show <VDOM_name> <----- Write name of each VDOM.
- Connect to the Secondary unit.
Must connect to the secondary unit using CLI, using this command:
execute ha manage
To know how to use it, check this article:
- Obtain General HA information in the Secondary unit:
get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster
diagnose debug crashlog read
- Recalculate HASH values for all configurations in the Secondary unit:
diagnose sys ha checksum recalculate global
diagnose sys ha checksum recalculate root
diagnose sys ha checksum recalculate <VDOM_name>
- Obtain HASH values for all configurations in the Secondary unit:
diagnose sys ha checksum show global
diagnose sys ha checksum show root
diagnose sys ha checksum show <VDOM_name>
- Compare all HASH obtained in point 3 and point 7 to get differences.
Use a simple text comparison tool, for example, notepad++, and get differences in configuration.
To know how to compare, add this plug-in to the notepad++.
Install Plug-in 'compare' in Notepad++.
Copy the HASH obtained in point 3, in another notepad copy the HASH obtained in point7, and perform a compare task, to obtain the differences highlighted:
Take note of differences and identify if belong to Global, Root, or any other VDOM.
in this example, there is:
The difference in GLOBAL HASH:
FGT Primary system.admin: 951a796c3d5cce457563476f9aa9b98e
FGT Secondary system.admin: 91fb884502aa610692bac371d6490456
The difference in ROOT HASH:
FGT Primary firewall.policy: a7f03dc8d82aca5281510c989723e120
FGT Secondary firewall.policy: 50318191a743e5d5884c57f407e6b66e
- Identify the exact difference.
With this information, complete and execute the next command in each FortiGate:
diagnose sys ha checksum show <value_1> <value_2>
- <value_1> can be global, root, or any VDOM_name.
- <value_2> can be system.admin or any other string obtained in the previous point.
Alternatively, it is possible to get the exact difference from the GUI if hovering over the not synchronized member (this feature is available from FortiOS 7.0):
Now, it is possible to open a support ticket and provide all information collected.
If the user is an advanced FortiGate administrator, it is possible to continue with troubleshooting as well.
- Correct the difference. In the example above, two configuration differences were found, one in administrative user 'john.connor' and sent in a Firewall Policy ID-66. Now correct differences using CLI in both FortiGate, sometimes a special character can cause this mismatch.In some particular cases, some parts of the configuration are different and cannot be changed manually using CLI, for example. UUIDs, replacement messages, default objects, ISDB objects, when this happens it is necessary to rebuild the entire HA cluster, or if there are too many differences to correct, try to rebuild completely HA with two methods:
1st. Method. Rebuild HA Cluster only by configuring HA parameters.
Prerequisites:
- Physical access to HA cluster.
- Serial Console cable.
- Identify HA-physicall port.
Procedure:
- Connect to the serial port on Slave FortiGate.
- Execute and confirm:
execute factory reset
- Access to the secondary FortiGate GUI with default values:
- Configure HA parameters, be careful if override is enabled, and be sure priority is lower than primary otherwise, there is a possibility to erase all configurations.
- Wait for cluster sync, it takes between 5-15 minutes, and the serial console in the secondary unit will show when sync finishes.
2nd Method. Rebuild HA Cluster using primary backup.
This method is described in the below KB article:
