Created on
02-09-2023
10:15 PM
Edited on
11-26-2024
06:12 AM
By
Jean-Philippe_P
Description
This article describes advanced troubleshooting for High Availability Cluster and collects information to deliver to Fortinet TAC for a support ticket.
Scope
FortiGate, HA.
Solution
- Obtain General HA information in the Primary unit:
diagnose sys ha checksum show global
diagnose sys ha checksum show root
- Recalculate HASH values for all configurations in the Primary unit:
- Obtain HASH values for all configurations in the Primary unit:
- Connect to the Secondary unit.
- Obtain General HA information in the Secondary unit:
6. Recalculate HASH values for all configurations in the Secondary unit:
- Obtain HASH values for all configurations in the Secondary unit:
- Compare all HASH obtained in point 3 and point 7 to get differences.
Take note of differences and identify if belong to Global, Root, or any other VDOM.
In this example, there are:
- The difference in GLOBAL HASH:
- The difference in ROOT HASH:
- Identify the exact difference.
- <value_1> can be global, root, or any VDOM_name.
- <value_2> can be system.admin or any other string obtained in the previous point.
Alternatively, it is possible to get the exact difference from the GUI if hovering over the not synchronized member (this feature is available from FortiOS 7.0):
Now, it is possible to open a support ticket and provide all information collected.
If the user is an advanced FortiGate administrator, it is possible to continue with troubleshooting as well.
- Correct the difference. In the example above, two configuration differences were found, one in administrative user 'john.connor' and sent in a Firewall Policy ID-66. Now correct differences using CLI in both FortiGate, sometimes a special character can cause this mismatch. In some particular cases, some parts of the configuration are different and cannot be changed manually using CLI, for example.
UUIDs, replacement messages, default objects, and ISDB objects, when this happens it is necessary to rebuild the entire HA cluster, or if there are too many differences to correct, try to rebuild completely HA with two methods:
1st. Method: Rebuild the HA Cluster only by configuring HA parameters.
- Physical access to HA cluster.
- Serial Console cable.
- Identify HA-physical port.
- Connect to the serial port(Console port) on Slave FortiGate.
- Execute and confirm:
- Access to the secondary FortiGate GUI with default values:
- Configure HA parameters, be careful if override is enabled, and be sure priority is lower than primary otherwise, there is a possibility to erase all configurations.
- Wait for cluster sync, it takes between 5-15 minutes, and the serial console in the secondary unit will show when sync finishes.
If while running HA debugs:
diag debug application hasync -1
diag debug application hatalk -1
If having a FortiGate-120G using v7.2.9, a known bug 1056138 is encountered.
Which behavior yields the following results:
get system ha status
diagnose sys ha status
chksum dump: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =====> for secondary checksum all in 00
================== FG120GXXXXXXXXXXXXXX================== <----- checksum cluster for Secondary is empty after serial number.
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
The issue is scheduled to be resolved in v7.2.11 and v7.6.1.
There are 2 workarounds:
- Use another port instead of dedicated HA.
- Rollback to the previous version.
Currently, this is only presented for FortiGate-120G using v7.2.9 and v7.2.10.