FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
jdelafuente_FTNT
Article Id 244833
Description

 

This article describes advanced troubleshooting for High Availability Cluster and collects information to deliver to Fortinet TAC for a support ticket.

 

Scope

 

FortiGate, HA.

 

Solution

 

  1. Obtain General HA information in the Primary unit:

 

get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by vcluster
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster

diagnose sys ha checksum show global

diagnose sys ha checksum show root

show full-configuration sys ha
diagnose debug crashlog read
 
  1. Recalculate HASH values for all configurations in the Primary unit:
 
diag sys ha checksum recalculate global
diag sys ha checksum recalculate root
diag sys ha checksum recalculate <VDOM_name> <----- Write name of each VDOM.
 
  1. Obtain HASH values for all configurations in the Primary unit:
 
diag sys ha checksum show global 
diag sys ha checksum show root
diag sys ha checksum show <VDOM_name> <----- Write name of each VDOM.
 
  1. Connect to the Secondary unit.
 
Must connect to the secondary unit using CLI, using this command:
 
exec ha manage <ID> <User_Name>
 
Example:
 
exec ha manage 0 admin
 
To know how to use it, check this article:
 
  1. Obtain General HA information in the Secondary unit:

 

get system status
get sys ha status
get hardware status
diagnose sys ha status
diagnose sys ha dump-by group
diagnose sys ha history read
diagnose sys ha checksum cluster
diagnose debug crashlog read
 

6. Recalculate HASH values for all configurations in the Secondary unit:

 
diagnose sys ha checksum recalculate global
diagnose sys ha checksum recalculate root
diagnose sys ha checksum recalculate <VDOM_name> 
 
  1. Obtain HASH values for all configurations in the Secondary unit:
 
diagnose sys ha checksum show global 
diagnose sys ha checksum show root 
diagnose sys ha checksum show <VDOM_name> 
 
  1. Compare all HASH obtained in point 3 and point 7 to get differences.
Use a simple text comparison tool, for example, notepad++, and get differences in configuration.
 
To know how to compare, add this plug-in to the notepad++.
 
01-notepadCompara}e.JPG
 
01-notepadCompare2-.png
 
Install Plug-in 'compare' in Notepad++.
 
Copy the HASH obtained in point 3, in another notepad copy the HASH obtained in point7, and perform a compare task, to obtain the differences highlighted:
 
Anthony_E_0-1676009249188.png

 

Anthony_E_1-1676009265385.png
 
Anthony_E_2-1676009284394.png

 

Take note of differences and identify if belong to Global, Root, or any other VDOM. 

In this example, there are:

 

  • The difference in GLOBAL HASH:
 
FGT Primary system.admin: 951a796c3d5cce457563476f9aa9b98e
FGT Secondary system.admin: 91fb884502aa610692bac371d6490456
 
  • The difference in ROOT HASH:
 
FGT Primary firewall.policy: a7f03dc8d82aca5281510c989723e120
FGT Secondary firewall.policy: 50318191a743e5d5884c57f407e6b66e
 
  1. Identify the exact difference.
With this information, complete and execute the next command in each FortiGate:
 
diagnose sys ha checksum show <value_1> <value_2>
 
  • <value_1> can be global, root, or any VDOM_name.
  • <value_2> can be system.admin or any other string obtained in the previous point.
 
connor-A.png

poli2.png

 

Alternatively, it is possible to get the exact difference from the GUI if hovering over the not synchronized member (this feature is available from FortiOS 7.0):

 

ha_difference_from_gui.png

 

Now, it is possible to open a support ticket and provide all information collected.

 

If the user is an advanced FortiGate administrator, it is possible to continue with troubleshooting as well.

 

  1. Correct the difference. In the example above, two configuration differences were found, one in administrative user 'john.connor' and sent in a Firewall Policy ID-66. Now correct differences using CLI in both FortiGate, sometimes a special character can cause this mismatch. In some particular cases, some parts of the configuration are different and cannot be changed manually using CLI, for example.

    UUIDs, replacement messages, default objects, and ISDB objects, when this happens it is necessary to rebuild the entire HA cluster, or i
    f there are too many differences to correct, try to rebuild completely HA with two methods:
 

1st. Method: Rebuild the HA Cluster only by configuring HA parameters.

 
Prerequisites:
  • Physical access to HA cluster.
  • Serial Console cable.
  • Identify HA-physical port.
 
Procedure:
  • Connect to the serial port(Console port) on Slave FortiGate.
  • Execute and confirm:

 

execute factory reset
 
  •  Access to the secondary FortiGate GUI with default values:
 
user: admin
password: blank
 
  • Configure HA parameters, be careful if override is enabled, and be sure priority is lower than primary otherwise, there is a possibility to erase all configurations.
  •  Wait for cluster sync, it takes between 5-15 minutes, and the serial console in the secondary unit will show when sync finishes.
 
HAsynsucced.JPG

2nd Method. Rebuild HA Cluster using primary backup.
 
This method is described in the below KB article:
 

If while running HA debugs:

 

diag debug application hasync -1
diag debug application hatalk -1

 

If having a FortiGate-120G using v7.2.9, a known bug 1056138 is encountered.

Which behavior yields the following results:

 

get system ha status
diagnose sys ha status

chksum dump: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =====> for secondary checksum all in 00

================== FG120GXXXXXXXXXXXXXX================== <----- checksum cluster for Secondary is empty after serial number.

hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)
hasync:WARN conn=0x2d145510 connect(169.254.0.1) failed: 113(No route to host)
hasync:WARN conn=0x2d145510 abort: rt=-1, dst=169.254.0.1, sync_type=3(fib)

The issue is scheduled to be resolved in v7.2.11 and v7.6.1.


There are 2 workarounds: 

  1. Use another port instead of dedicated HA.
  2. Rollback to the previous version.

Currently, this is only presented for FortiGate-120G using v7.2.9 and v7.2.10.