Description
This article provides some simple steps to follow where an HA cluster may have to be rebuilt in order to ensure basic HA operation.
Scope
FortiGate.
Solution
In cases where Administrators wish to build a HA cluster or try to recover from a synchronization issue.
Note:
It is recommended to perform these steps locally as cabling should be disconnected and reconnected to ensure a smooth rebuild. Be sure to perform these steps during a maintenance window per best practices.
Preparing for the rebuild:
- Back up the primary unit config. If obtaining support from Fortinet, please provide the configuration so the Engineer can provide specific guidance.
- Be careful to remove the cluster's data connections first, monitored interface cables next, and HA heartbeat cables last.
- Disconnect the backup unit from the cluster. Ensure the unit is not connected to the network in any way to prevent both FortiGates from acting as primaries (split-brain).
- In the CLI, reset the backup to factory defaults: 'exec factoryreset'. It will take 5-10 minutes to reboot.
From the Primary unit config copy the HA settings. Some of the most critical parameters are:
- group-id
- group-name
- password <----- Check note below.
- unit priority
- mode
- hbdev (heartbeat interface/device)
- monitored interfaces ('monitor')
To show the settings in the CLI, run the following:
show system ha
To check the HA settings using the config file, search for 'config system ha'.
Example copied config:
config system ha
set group-id 33
set group-name "haCluster"
set mode a-p
set password ENC bi+kLsLH7Z8Gxyw4P/+5eIE2PoWs/Cp/aI+2qtJjlwdhP2ckfK4AFc45yKJTak9M7x1OsXtixBDCz70Uru/238zWqbXnobuuLIWqCM7udaGpWwgaXWoDi8rNPegNVtZ4yIbC5xA7T6ZwCje4/+SvKcMQ8R5AjiEskIZb3fNMhDVRbHiyGKsvFKsx1iy/vpP1OvFSUg==
set hbdev "wan2" 50
set session-pickup enable
set ha-mgmt-status enable
set ha-mgmt-interface "internal9"
set override enable
set priority 255
set monitor "wan1"
end
Set the following on the new unit via console:
config system global
set hostname <secondary_unit>
end
Configure the following only if there is a dedicated management interface:
config system interface
edit <mgmt-interface>
set ip <dedicated secondary_unit ip> <subnet mask>
end
Note:
Be sure to also copy the line that contains the cluster password. Paste the HA settings into a text document. If the cluster password is lost or forgotten, it can be changed on all cluster units. Change it from GUI on primary and paste the same on the above configuration in the text editor. (When the primary password changed, a few packets would be dropped or sessions would disconnected.)
Rebuilding the backup:
- If override is enabled like in the config above, it is recommended to set the priority below that of the primary unit. This will avoid any service interruptions when the backup is added to the cluster. For this example, we will set priority to 254 since the primary is 255.
- On the backup unit in the console, paste the HA settings from the text editor.
Note:
Assuming override is disabled, whichever unit has the highest uptime will become the new primary unit. Ensure the primary unit in production has a higher uptime before connecting the backup if override is disabled (which is the default). To check: 'get sys perf stat | grep Uptime'.
- Reconnect the Primary unit(s). Connect the HA heartbeat cables first, monitored interface cables next, and data connections last when re-connecting the secondary device. Note that it may reboot once when synchronizing.
- Repeat these steps for as many cluster members as are needed.
Once this is done and the cluster has been formed, the primary unit configuration will be synchronized to the backup devices. This process takes 5-20 minutes depending on the size of the configuration and how many cluster members exist. To ensure the cluster is fully synchronized, follow the checksum article referenced below.
Note:
If override needs to be enabled, make sure it is enabled individually on both Primary and Secondary units
config system ha
set override enable
end
Related articles:
