Technical Tip: Standalone configuration sync function shows out-of-sync after reboot

This article is not relevant for troubleshooting High Availability configuration synchronization issues. See: How to troubleshoot HA synchronization issue using GUI' for initial troubleshooting steps in such cases.

The issue only occurs for FortiGates that have standalone configuration synchronization as well as FortiGate Session Life Support Protocol (FGSP) enabled. If a device has entries under ‘config system cluster-sync’ and also has ‘standalone-config-sync’ enabled, then it can be affected by this issue.

config system cluster-sync

edit <index>

set peerip <A.B.C.D>

next

end

config system ha

set standalone-config-sync enable

end

With the above configuration, if either device loses power or reboots, the primary FortiGate will not be able to synchronize the configuration with the secondary FortiGate when the device boots.

Data sessions will still synchronize as expected, but firewall configuration will not synchronize. The CLI command 'get system ha status' on either device will show the remote peer as out-of-sync and the peer's configuration status will not be updated.

LabFGT-B # get system ha status
Primary selected using:
HA Health Status: OK
Model: FortiGate-121G
Mode: ConfigSync
Group: 129
Debug: 0
Cluster Uptime: 0 days 3:29:58
Cluster state change time: 2024-10-09 12:21:04

<2024/10/09 12:21:04> FG121GTKBBBBBBBB is selected as the primary because its uptime is larger than peer member FG121GTKAAAAAAAA.

<2024/10/09 12:19:29> FG121GTKBBBBBBBB is selected as the primary because it's the only member in the cluster.

<2024/10/09 12:19:22> FG121GTKBBBBBBBB is selected as the primary because SET_AS_SECONDARY flag is set on peer member FG121GTKAAAAAAAA.

<2024/10/09 12:14:47> FG121GTKAAAAAAAA is selected as the primary because its override priority is larger than peer member FG121GTKBBBBBBBB.

ses_pickup: enable, ses_pickup_delay=disable
override: disable
Configuration Status:

FG121GTKBBBBBBBB(updated 1 seconds ago): in-sync
FG121GTKAAAAAAAA(updated 12311 seconds ago): out-of-sync

System Usage stats:

FG121GTKBBBBBBBB(updated 1 seconds ago):

sessions=29, average-cpu-user/nice/system/idle=0%/0%/1%/98%, memory=18%

FG121GTKAAAAAAAA(updated 12311 seconds ago):

sessions=29, average-cpu-user/nice/system/idle=0%/0%/1%/98%, memory=18%

HBDEV stats:

FG121GTKBBBBBBBB(updated 1 seconds ago):

ha: physical/100full, up, rx-bytes/packets/dropped/errors=82238837/165383/0/0, tx=83335588/161692/0/0

FG121GTKAAAAAAAA(updated 12311 seconds ago):

ha: physical/100full, up, rx-bytes/packets/dropped/errors=1956148/3996/0/0, tx=1949361/3865/0/0

Primary : LabFGT-B , FG121GTKBBBBBBBB, HA cluster index = 0
Secondary : LabFGT-A , FG121GTKAAAAAAAA, HA cluster index = 1
number of vcluster: 1
vcluster 1: work 169.254.0.1
Primary: FG121GTKBBBBBBBB, HA operating index = 0
Secondary: FG121GTKAAAAAAAA, HA operating index = 1

Attempting to connect to the other cluster device with the 'execute ha manage' command will fail with the error 'No route to host'. 

LabFGT-B # execute ha manage 1 admin
ssh: connect to host 169.254.0.2 port 22: No route to host

Warning:

Restarting hatalk on a device that does not match this configuration and is instead using active-active or active-passive High Availability mode will cause temporary split-brain conditions and network disruption.

To workaround the issue, restart the hatalk daemon on the primary device with 'fnsysctl killall hatalk'. This will not remove data sessions synced by FGSP, which uses a different daemon for its operation. It is recommended to take configuration backups from each device before restarting hatalk. 

When the device is in standalone ha mode, restarting hatalk will only affect configuration synchronization.
If the cluster has been out-of-sync for some time or has significant configuration differences between devices, it is recommended to make the change in a test window.

show full-configuration system ha | grep mode

set mode standalone  <--- if this is any value other than 'standalone', restarting hatalk is likely to cause network disruption.

fnsysctl killall hatalk

After restarting hatalk, it may take up to 15 minutes for the configuration to show as in-sync, but it should recover without further intervention.

Configuration Status:

FG121GTKBBBBBBBB(updated 2 seconds ago): in-sync
FG121GTKAAAAAAAA(updated 0 seconds ago): out-of-sync

Configuration Status:

FG121GTKBBBBBBBB(updated 2 seconds ago): in-sync
FG121GTKAAAAAAAA(updated 1 seconds ago): in-sync

If the configuration still shows out of sync after several minutes and the device has a valid support contract, open a case with FortiGate TAC for further investigation.

Restarting hatalk is just a temporary fix. It does not prevent the issue from occurring again when a device reboots. The permanent fix for the issue is scheduled for inclusion in v7.2.11, v7.4.8, and v7.6.1.

Related documents:

FGSP FortiGate Session Life Support Protocol

Standalone configuration synchronization

Managing individual cluster units with the CLI command 'execute ha manage'

How to restart/kill one or several processes on the FortiGate with CLI commands

How to troubleshoot HA synchronization issue using GUI