Troubleshooting Tip: High Availability checksum mismatch with System Event logs indicating 'external-files' are out-of-sync
Description
This article explains how to solve a checksum mismatch that leads to 'external-files' being labelled as 'out-of-sync'.
Scope
FortiGate.
Solution
Events such as the following may occasionally be found while reviewing FortiGate event logs (elogs) on a High Availability cluster:
date=2016-07-15 time=01:30:20 devname=BERCA1FW02 devid=FG100D3G13803210 logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with primary" msg="The sync status with the primary" sync_type=external-files sync_status="out-of-sync"
...date=2016-07-15 time=01:31:04 devname=BERCA1FW02 devid=FG100D3G13803210 logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with primay" msg="The sync status with the primary" sync_type=external-files sync_status="in-sync"
The above HA System Event log messages indicate that the FortiGuard signatures/engines of the Primary and Secondary FortiGate systems were desynchronized when the time was equal to 01:30:20.
This issue has several possible causes:
This issue has several possible causes:
- The Primary FortiGate has updated its FDS databases through FortiGuard updates, but the Backup unit has not yet synchronized its database and/or engine versions through the heartbeat connection.
- The periodic refresh of external threats databases (when configured).
Executing the following command on the CLI may also reveal a difference in HA checksum between the two systems:
diagnose sys ha checksum show
This event may cause further problems while operating in Active-Active HA mode since the Secondary unit is used to offload some of the UTM processing, and the most up-to-date FortiGuard information is desired on both units.
The best approach to synchronizing the two units varies depending on the following factors:
- Heartbeat interface connection reliability: in this case, the best practice is to use a direct (isolated) connection between the clusters or a dedicated switch for clusters with more than two units.
- Whether session-pickup is enabled on highly active HA clusters: When session-pickup is enabled, more traffic is transferred through the heartbeat interface. This may delay the synchronization of the new FortiGuard information depending on the heartbeat connection interface reliability and how many sessions are being synchronized. Use the following options to reduce the impact of this feature:
Use session-pickup-delay to synchronize sessions only if they remain active for more than 30 seconds.
CLI configuration:
config system ha
set session-pickup-delay enable
end
Use the session-sync-dev option to dedicate interface(s) for session synchronization.
CLI configuration for enabling port9 and port11 for session synchronization:
config system ha
set session-sync-dev port9 port11
end
In the second case mentioned above, check the configuration with the following command:
config system external-resource
show
For every entry the command returns, there is an automated action that repeats (by default) every 5 minutes and checks and updates the lists defined on the servers. This causes the normal behavior of HA in-sync and out-of-sync every 5 minutes, even if there are no updates in the lists.
Correct these timers to a more suitable value:
set refresh-rate x (Values from 1 to 43200, in minutes)
Alternatively, remove the HA emails or notifications for this type of alert.
For more details on improving and troubleshooting HA synchronization, review the FortiOS Handbook - High Availability documentation.