Description
This article describes how the HA synchronization behave when changes are made either on master or slave.
Solution
Incremental synchronization.
When a user is logged into the cluster web-based manager or CLI to make configuration changes, it is actually into the primary unit.
All of the configuration changes are first made to the primary unit.
Incremental synchronization then immediately synchronizes these changes to all of the subordinate units.
When a user is logged into a subordinate unit CLI (for example using execute ha manage) all of the configuration changes made to the subordinate unit are also immediately synchronized to all cluster units, including the primary unit, using the same process.
Incremental synchronization also synchronizes other dynamic configuration information such as the DHCP server address lease database, routing table updates, IPsec SAs, MAC address tables, and so on.
See an introduction to the FGCP for more information about DHCP server address lease synchronization and Synchronizing kernel routing tables for information about routing table updates.
Whenever a change is made to a cluster unit configuration, incremental synchronization sends the same configuration change to all other cluster units over the HA heartbeat link.
A HA synchronization process running on each cluster unit receives the configuration change and applies it to the cluster unit.
The HA synchronization process makes the configuration change by entering a CLI command that appears to be entered by the administrator who made the configuration change in the first place.
Synchronization takes place silently, and no log messages are recorded about the synchronization activity.
However, log messages can be recorded by the cluster units when the synchronization process enters CLI commands.
These log messages are visible on the subordinate units if event logging is enabled, set the minimum severity level to Information and then check the event log messages written by the cluster units when a configuration change is made.
These log messages are also visible on the primary unit if configuration changes from a subordinate unit are made.
Periodic synchronization.
Incremental synchronization makes sure that as an administrator makes configuration changes, the configurations of all cluster units remain the same.
However, a number of factors can cause one or more cluster units to go out of sync with the primary unit.
For example, by adding a new unit to a functioning cluster, the configuration of this new unit will not match the configuration of the other cluster units.
It is not practical to use incremental synchronization to change the configuration of the new unit.
Periodic synchronization is a mechanism that looks for synchronization problems and fixes them.
Every minute the cluster compares the configuration file checksum of the primary unit with the configuration file checksums of each of the subordinate units.
If all subordinate unit checksums are the same as the primary unit checksum, all cluster units are considered synchronized.
If one or more of the subordinate unit checksums are not the same as the primary unit checksum, the subordinate unit configuration is considered out of sync with the primary unit.
The checksum of the out-of-sync subordinate unit is checked again every 15 seconds.
This re-checking occurs in case the configurations are out of sync because an incremental configuration sequence has not been completed.
If the checksums do not match after 5 checks the subordinate unit that is out of sync retrieves the configuration from the primary unit.
The subordinate unit then reloads its configuration and resumes operating as a subordinate unit with the same configuration as the primary unit.
The configuration of the subordinate unit is reset in this way because when a subordinate unit configuration gets out of sync with the primary unit configuration there is no efficient way to determine what the configuration differences are and to correct them.
Resetting the subordinate unit configuration becomes the most efficient way to resynchronize the subordinate unit.
Synchronization requires that all cluster units run the same FortiOS firmware build. If some cluster units are running different firmware builds, then unstable cluster operation may occur and the cluster units may not be able to synchronize correctly.
Key Considerations:
-
Firmware Uniformity: All cluster units must operate on the same FortiOS firmware build. Inconsistencies can lead to synchronization failures and erratic cluster behavior.
-
Network Health: Ensure that the HA heartbeat link is healthy and stable. Network disruptions can interfere with both incremental and periodic synchronizations.
-
Monitoring Tools: Utilize FortiGate's built-in monitoring tools to keep an eye on HA synchronization status. This can help preemptively identify and address potential issues.
-
Backup: Regularly backup configurations. While synchronization mechanisms are robust, maintaining backups ensures an additional layer of security against configuration losses.
-
Hardware Consistency: While not directly related to synchronization, it is beneficial to have consistent hardware specifications across the cluster units to avoid potential performance discrepancies.
Conclusion: Ensuring synchronization in HA setups is crucial to maintaining the availability and reliability of services. By understanding and monitoring the different synchronization mechanisms, administrators can ensure seamless operation and quick recovery from potential issues.
