Description
This article explains how to solve a checksum mismatch that leads to 'external-files' being labelled as 'out-of-sync'.
Scope
FortiGate.
Solution
Events such as the following may occasionally be found while reviewing FortiGate event logs (elogs) on a High Availability cluster:
date=2016-07-15 time=01:30:20 devname=BERCA1FW02 devid=FG100D3G13803210 logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with master" msg="The sync status with the master" sync_type=external-files sync_status="out-of-sync"
...date=2016-07-15 time=01:31:04 devname=BERCA1FW02 devid=FG100D3G13803210 logid=0108037903 type=event subtype=ha level=information vd=root logdesc="Synchronization status with master" msg="The sync status with the master" sync_type=external-files sync_status="in-sync"
The above HA System Event log messages indicate that the FortiGuard signatures/engines of the Master and Slave FortiGate systems were desynced when the time was equal to 01:30:20.
This issue has a number of possible causes:
- The Primary FortiGate has updated its FDS databases through FortiGuard updates, but the Backup unit has not yet synchronized its database and/or engine versions through the heartbeat connection.
- The periodic refresh of external threats databases (when configured).
Executing the following command on CLI may also reveal a difference in HA checksum between the two systems:
# diagnose sys ha checksum show
This event may cause further problems while operating in Active-Active HA mode since the Slave unit is used to offload some of the UTM processing, and the most up-to-date FortiGuard information is desired on both units.
The best approach to synchronizing the two units varies depending on the following factors:
- Heartbeat interface connection reliability: in this case, the best practice is to use a direct (isolated) connection between the clusters or a dedicated switch for clusters with more than two units.
- Whether session-pickup is enabled on highly active HA clusters: When session-pickup is enabled, more traffic is transferred through the heartbeat interface. This may delay the synchronization of the new FortiGuard information depending on the heartbeat connection interface reliability and how many sessions are being synchronized. Use the following options to reduce the impact of this feature:
- Use session-pickup-delay in order to synchronize sessions only if they remain active for more than 30 seconds.
CLI configuration:
# config system ha
# set session-pickup-delay enable
# end
- Use the session-sync-dev option to dedicate interface(s) for session synchronization:
CLI configuration for enabling port9 and port11 for session synchronization:
# config system ha
# set session-sync-dev port9 port11
# end
In the second case mentioned above, check configuration with the following command:
# config system external-resource
# show
For every entry the command returns, there is an automated action that repeats (by default) every 5 minutes and checks and updates the lists defined on your servers. This causes the normal behavior of HA in-sync and out-of-sync every 5 minutes, even if there are no updates in the lists.
You can correct these timers to a more suitable value:
# set refresh-rate x (Values from 1 to 43200, in minutes)
Alternatively, remove the HA emails or notifications for this type of alert.
