Troubleshooting Tip: Restore Cluster HA with more than 10 VDOMs

[pabarro]

Description	This article describes how an HA Cluster with more than 10 VDOMs may have out of sync statuses because there are too many checksum differences between different tables, or because a FortiGate in the cluster is behaving abnormally and is not synchronized in any way. This procedure shown is to recover a FortiGate that belongs to the HA Cluster. The procedure is based on when the override is disabled.
Scope	Any FortiGate up to 3K from v6.0 up to v7.6.
Solution	For test purposes, FortiGate-500E v7.0.15, build0566, 231024 is used. Confirm if override is disabled by checking the sample below (the command used is 'get sys ha status'): Hostname: FortiGate-A FG-SERIALXXX_A Primary Priority 200 FortiGate-500E v7.0.15,build0566,231024 (GA.M) Override disable Mode: Active - Pasive -------------------------- Hostname: FortiGate-B FG-SERIALXXX_B Secondary Priority 100 FortiGate-500E v7.0.15,build0566,231024 (GA.M) Override disable Mode: Active - Pasive Preparation: A person must be on-site to be able to connect to the devices. Make a full backup of the Primary (FortiGate-A). Download the FortiOS FortiGate-500E v7.0.15,build0566,231024 (GA.M) image (for the FortiGate-500E hardware) to do a clean install of the Secondary (FortiGate-B). VDOM license key should be available . Activity Summary: Currently, FortiGate-B has many problems. The procedure will be performed on this device. Perform a Flash Format on FortiGate-B. Load the required image FortiGate-500E v7.0.15, build0566,231024 (GA.M), load a backup, edit it, add it to the cluster, and resynchronize the HA. Technical Tip: Formatting and loading FortiGate firmware image using TFTP. Procedure: Make a validation of the number of VDON in the  Fortigate. Initialy the number of VDOM is confirmed as follow (this is part of the output for better understanding): get sys status Version: FortiGate-500E v7.0.15, build0566, 231024  v7.2.7,build1577,240131 (GA.M) Serial-Number: FG-SERIALXXX_A Hostname: Fortigate-A .................................. Virtual domains status: 20 in NAT mode, 0 in TP mode Virtual domain configuration: multiple FIPS-CC mode: disable Current HA mode: a-p, primary Make a full backup of Fortigate-A (Active FortiGate with serial terminal FG-SERIALXXX_A). Proceed to download the FortiOS required for the hardware from the support website (https://support.fortinet.com). n this case, FortiGate-500E v7.0.15, build0566,231024 (GA.M). Make sure it is for the hardware (FortiGate-500E). Save it in a folder to avoid confusion with any other version. Disconnect (all cables) from the HA cluster's FortiGate-B. (The secondary FortiGate with serial terminal FG-SERIALXXX_B). Leave only the Fortigate-A connected (Active FortiGate with serial terminal FG-SERIALXXX_A). This equipment will remain operational to avoid services being affected. See Figure 1: Note: If the cables are not labeled, proceed to identify them to avoid confusion when reconnecting them later.  Work with the FortiGate that is disconnected from the HA i.e. FortiGate-B (Hardware with serial number FG-SERIALXXX_B). Proceed to perform a flash format and load the FortiGate-500E v7.0.15,build0566,231024 (GA.M) image as indicated in the link shared above. Once step 6 is completed and with the unit up, log in to the FortiGate via GUI. Open a CLI console and type the next command: config global     execute upd-vd-license <vdom license key>             execute upd-vd-license XXXX-YYYY-ZZZZ-XXX update vdom license succeeded  (See Technical Tip: Detailed explanation on how to install VDOM license in HA environment.) Load the backup made in Step 1 to this FortiGate. Once the backup is loaded, the FortiGate will reboot. Log back in to the FortiGate and Edit the Hostname and basic HA configuration via the CLI as follows: config system global     set hostname Fortigate-B end config system ha     set priority 100  <- The value will be set to 100, originally 200. end                             <- Save the changes. Check from the CLI that the changes have been taken. show full system global | grep hostname      <- FortiGate-B should be the hostname. show full system ha | grep priority          <- 100 should be the Priority. Once these configuration settings have been made and verification has been performed, physically connect the secondary device (FortiGate-B) to the HA cluster and wait for it to sync. See Figure 2: If the units DO NOT sync, open a ticket with support and call support for immediate assistance.