Description
This article describes how to fix HA (High Availability) cluster upgrade failure, which results in each firewall in the cluster having a different OS version.
Scope
FortiGate HA Active Passive.
Solution
Uninterruptible HA cluster upgrade mode (the default) will upgrade the secondary device before the primary.
In most failure cases with this option, the secondary device is successfully upgraded to a newer version while the primary device stays on the current OS version.
This problem can occur with any OS version and device model.
Pre-checklist:
- Perform 'diagnose sys flash list' on both units:
The primary unit most likely has no newer image on its flash storage:
diagnose system flash list
Partition Image TotalSize(KB) Used(KB) Use% Active
1 FG100F-6.04-FW-build1914-211117 253920 102616 40% No
2 FG100F-7.00-FW-build0367-221005 253920 110112 43% Yes
3 ETDB-90.07704 3021708 790996 26% No
Image build at Oct 5 2022 22:02:56 for b0367
- Execute 'diagnose system ha dump-by kernel' on both firewalls:
diagnose system ha dump-by kernel
<hatalk> HA information.
<hatalk> group_id=13, nvcluster=1, mode=2, load_balance=0, schedule=3, ldb_udp=0.
<hatalk> nvcluster=1, mode=2, ses_pickup=1, delay=0, load_balance=0
schedule=3, ldb_udp=0, upgrade_mode=0. <----- Upgrade_mode should be '0'. If not, reboot the firewall to reset this to 0.
- HA session sync still takes place.
Since each unit has a different OS version, the cluster status will be out-of-sync.
However, the session sync should still be running.
If all prerequisites are met, follow the steps below. Otherwise, contact Fortinet TAC for additional help.
- Perform the cluster upgrade once again: FortiGate will automatically upgrade the device without the latest firmware.
- Manually download the firmware image from the support site.
- Reboot both HA cluster members.
- Then proceed with the upgrade using the downloaded image.
If this step did not work and it is not possible to upload the image, follow step 2:
- Failover the FortiGate: make the secondary device the new primary device and perform the cluster upgrade.
In this case, be sure that the secondary can take over the traffic in normal conditions.
- Downgrade the device to the latest version (which is usually the secondary device) and perform the normal HA upgrade procedure.
Downgrade through the CLI with the following command:
execute set-next-reboot {primary | secondary} <-- Make sure the older image is there.
See this article: Technical Tip: Selecting an alternate firmware for the next reboot for more information.
If none of the steps above work, try the following:
- Take the following steps with physical access to the device:
- Separate one of the firewalls from the HA cluster, but back up the config first.
- Turn off the firewall that will be taken out of the cluster to avoid split-brain. Or, to prevent inducing a split brain through an alternative method, disconnect from HA. To do this, follow the steps below:
- Before disconnecting the HA Cables, disconnect the traffic cables first.
- Do not remove only the HA Cables, as doing so will cause both devices to consider themselves as the master units.
- Remove all cables from the secondary device (starting with the data cables and then the HA cables). Connect to this unit directly through a PC and manually perform an upgrade.
- Once done, connect the HA cables first, allow the system to sync, and then connect the data cables.
- Technical Tip: Installing firmware from system reboot
- Technical Tip: How to verify downloaded firmware checksum
Before starting an HA upgrade, validate that both units are running compatible firmware versions and that the cluster synchronization status is healthy.
If the upgrade fails, check HA logs, diagnose sys ha status, diagnose sys ha dump-by vcluster, and clear synchronization errors before retrying.
Note:
If the problem persists after following the steps above, contact Fortinet TAC for further assistance.
Related documents:
