Staff

Technical Tip: Procedure for manual synchronization for HA out-of-sync issue

Forum|Forum|11 years ago
April 23, 2015
0 replies
501370 views

Description

This article describes the methods used to force the synchronization of a High Availability (HA) cluster before proceeding to rebuild the HA cluster.

Scope

FortiGate, High Availability synchronization.

Solution

For this procedure, it is recommended to have access to all units through SSH (i.e, PuTTY). Furthermore, confirm that all devices in the cluster are running the same firmware version (If VDOMs are enabled, make sure to not be in the VDOM context, and then execute the commands below):

get system status
execute ha manage <id> <username>
get system status

Note:

It is possible to connect to the other units with 'execute ha manage X <username>' where X is the member ID (Available IDs can be found by using the 'execute ha manage ?' command).

After that, a prompt requesting to insert the password for the specified user will show up:

DE_A (global) # execute ha manage 0 admin
Warning: Permanently added '169.254.0.1' (ECDSA) to the list of known hosts.
admin@169.254.0.1's password:

Also, HA-related commands in multi-VDOM environments must be run from the global VDOM.

To check the FortiGate HA status in the CLI:

get system ha status
diagnose sys ha checksum cluster

All cluster members need to have the same checksum values (compare the last digits of the 'all' checksum). The FortiGate Clustering Protocol (FGCP) uses incremental and periodic updates to make sure that all cluster units share the same configuration.

To check which part of the checksum is not matching, see the following link: Technical Tip: Troubleshooting a checksum mismatch in a FortiGate HA cluster.
Once it is identified that the specific VDOM checksum value is different, it is possible to check in which config the checksum is a mismatch in the specific VDOM: Troubleshooting Tip: How to troubleshoot HA synchronization issue using GUI.

If the checksums do not match, perform the following steps, logging ALL the output, in case it is needed to later open a Technical Support case with Fortinet:

Force the backup unit to synchronize with the primary unit.

On the backup unit:

execute ha synchronize start

Starting from FortiOS v8.0, a new GUI feature has been introduced to force HA synchronization. For details on the newly added HA GUI features, refer to Technical Tip: Additional GUI support for HA operations.

To navigate to this option, select System -> HA -> Select HA cluster member -> Select 'Diagnostics and Tools' -> Select Force resync. The Force resync option can also be used by selecting the unsynchronized unit and 'right-clicking' it.

Recalculate HA checksums:

A simple recalculation of checksums might help. Create a dummy unused object, such as a firewall address, or recalculate checksums on each unit manually.

On the primary unit:

diagnose sys ha checksum recalculate

(Check again afterwards if it is synchronized.)

On backup units:

diagnose sys ha checksum recalculate

(Check again afterwards if it is synchronized.)

Restart the synchronization process and monitor if there is an error in the debug (check both units simultaneously).

Note: The user may be logged out of the backup units during this process. This is a good sign. See this article Troubleshooting Tip: FortiGate HA synchronization messages and cluster verification steps.

On the primary unit:

execute ha synchronize stop
diagnose debug reset
diagnose debug enable
diagnose debug console timestamp enable
diagnose debug application hasync -1
diagnose debug application hatalk -1
execute ha synchronize start

On backup units:

diagnose debug reset
diagnose debug enable
execute ha synchronize stop
diagnose debug console timestamp enable
diagnose debug application hasync -1
diagnose debug application hatalk -1
execute ha synchronize start

Note: In the commands 'execute ha synchronize stop' and 'execute ha synchronize stop', the last word (start/stop) will not be autocompleted by pressing the Tab key on the keyboard. These commands must be manually typed out.

It is possible to check if the checksums match during this debug output. Disable debugging once the backup units are in sync with the primary unit, or after the capturing of logs is completed (5-6 minutes):

diagnose debug disable
diagnose debug reset

Provide the collected logs and configuration backup of each device to Fortinet Support in a support ticket.

Load the primary's configuration to the secondary directly:

If other methods of correcting a configuration difference between two devices are not effective, apply the change manually by following the instructions in Technical Tip: Correcting an out-of-sync HA cluster by modifying the primary unit configuration file and restoring it to the secondary unit. This requires physical access to the secondary device to safely isolate it from the network.

Restart the HA daemons/restart the units, one by one.

Warning: This option requires a maintenance window and might need physical access to both units, as it can affect the traffic. Killing the hatalk process will cause HA split-brain until the daemon restarts.

If there is no output generated in 'hasync debug' or 'hatalk debug', a restart of these daemons may be needed. This can be done by running the following commands on each unit at a time:

diagnose sys process pidof hasync
diagnose sys process pidof hatalk
diagnose sys kill 11 <pid#>

Note: Repeat for both noted processes. Killing the hatalk process ID will cause traffic interruption.

After these commands, the daemons normally restart with different numbers (check this via 'diagnose sys process pidof').

Another way to restart the HA daemons is by running the following commands:

fnsysctl killall hasync
fnsysctl killall hatalk

Note: Killing the hatalk process will cause traffic interruption.

In certain conditions, this does not solve the problem, or the daemons fail to restart. A hard reboot may be necessary (either execute a reboot from the console or plug/unplug the power supply).

After the reboot, check the disk status for both units (if a disk scan is needed, perform it before anything else), then check the cluster status (checksums) once again.

Manual synchronization:

In certain specific scenarios, the cluster fails to synchronize due to some elements in the configuration. To avoid rebuilding the cluster, compare the configurations and perform the changes manually:

Obtain the configurations from both units marked as Primary and Secondary/Backup. Make sure the console output is standard (no '---More---' text appears*), log the SSH output, and issue the command 'show' in both units.

Note:

To remove the paginated display:

config system console 
    set output standard
end

Note:

Do not issue 'show full-configuration' unless necessary.

Use any comparison tool available to check the two files side-by-side (i.e., Notepad++ with the 'Compare' plugin).
Certain fields can be ignored (hostname, SN, interface dedicated to management if configured, password hashes, certificates, HA priorities and override settings, and disk labels).
Perform configuration changes in CLI on backup units to reflect the configuration of the primary; if errors occur and they are explanatory, act accordingly. If it is not explanatory and the config can not be changed (added/deleted), ensure these errors are logged and presented in a TAC case.

After all of the changes outlined in the comparison are corrected, check for cluster status once again.

If all the above methods fail, a cluster rebuild may be needed.

Note 1: primary and secondary with different disk statuses.

If the primary and secondary units have different disk statuses, the cluster will fail. The following error can be seen on the console of the secondary unit:

Primary and secondary have different hdisk status. Cannot work with HA primary. Shutdown the box!

The output of the following commands needs to be collected from both cluster members:

get system status
execute disk list

If one of the cluster members shows log disk status as 'Need format' or 'Not Available', the unit needs to be disconnected from the cluster, and a disk format needs to be performed. This requires a reboot. This can be done by executing the following command:

execute formatlogdisk

A confirmation for a reboot will follow.

If the problem persists, open a ticket with Technical Support with the output of the following commands from both units in the cluster:

get system status
execute disk list

Note 2: The secondary unit is not visible in the cluster. When checking the checksums, the second unit may be missing or with incomplete output as follows:

diagnose sys ha checksum cluster
================== FGVMXXXXXXXXXX1 ==================
is_manage_primary()=1, is_root_primary()=1
debugzone
global: c5 33 93 23 26 9f 4d 79 ed 5f 29 fa 7a 8c c9 10
root: d3 b5 fc 60 f3 f0 f0 d0 ea e4 a1 7f 1d 17 05 fc
all: 04 ae 37 7e dc 84 aa a4 42 3d db 3c a2 09 b0 60

checksum
global: c5 33 93 23 26 9f 4d 79 ed 5f 29 fa 7a 8c c9 10
root: d3 b5 fc 60 f3 f0 f0 d0 ea e4 a1 7f 1d 17 05 fc
all: 04 ae 37 7e dc 84 aa a4 42 3d db 3c a2 09 b0 60

================== FGVMXXXXXXXXXX2 ==================

FortiVM1#

This happens in a situation where the hasync cannot communicate properly with the other unit.

What can be done:

Make sure the units are running the same firmware via 'get system status'.
Reboot both units one at a time, starting with the secondary.

Note 3: The secondary unit is not visible in the cluster due to a different FIPS-CC mode.

To verify, run the following on both units, and a similar error will be seen:

diagnose debug app hatalk -1

diagnose debug enable 

<hatalk> vcluster_0: ha_prio=0(primary), state/chg_time/now=2(work)/1750807157/1750812543
<hatalk:WARN> 'FG-SerialNumber' enc/auth mismatch: hdr_enc/auth=0/0, my_enc/auth=1/1
<hatalk> vcluster_0: ha_prio=0(primary), state/chg_time/now=2(work)/1750807157/1750812553
<hatalk> vcluster_0: ha_prio=0(primary), state/chg_time/now=2(work)/1750807157/1750812563
<hatalk:WARN> 'FG-SerialNumber' enc/auth mismatch: hdr_enc/auth=0/0, my_enc/auth=1/1

To disable the real-time debugging, run the following on both units:

diagnose debug disable
diagnose debug reset

Compare the fips-mode on both units under get system status.

If one unit mode differs, enable it by following this article: Technical Tip: How to enable FIPS-CC mode.

It can only be enabled through the console. The 'admin' account needs to exist, and the config will be removed after enabling FIPS-CC mode.

Restore the modified backup config for the secondary afterwards, and the secondary unit should be visible in the HA cluster.

Note 4: When the secondary device is attempting to synchronize configuration, administrators accessing it using primary CLI or secondary HA reserved management interface will be periodically logged out as shown below:

secondary's configuration is not in sync with primary's, sequence:0
secondary's configuration is not in sync with primary's, sequence:1
secondary's configuration is not in sync with primary's, sequence:2
secondary's configuration is not in sync with primary's, sequence:3
secondary's configuration is not in sync with primary's, sequence:4
secondary starts to sync with primary
logout all admin users

Note 5: In the cloud platforms below, many settings, including the IP interface for the interface, are not synced by HA, since the FortiGates may be in a different subnet:

'FGT_ARM64_AZURE'.
'FGT_ARM64_GCP'.
'FGT_VM64_ALI'.
'FGT_VM64_AZURE'.
'FGT_VM64_GCP'.
'FGT_VM64_IBM'.
'FGT_VM64_RAXONDEMAND'.
FortiGate VM08V.
FortiGate VM Azure On-Demand.

Differences in configuration intentionally not synced do not cause an out-of-sync warning. For example, changing an IPsec tunnel IP address on a FortiGate Azure VM primary is not automatically applied to the secondary cluster device and does not cause an out-of-sync warning. The workaround is to manually configure the IP addresses on both devices.

Certain additional configuration locations can be exempted from synchronization by adding them as a VDOM exception. See VDOM exceptions.

Related articles:

Sign up

Login with SSO

Login to the community

Login with SSO