Troubleshooting Tip: Fix an HA (High Availability) cluster upgrade failure

[iskandar_lie]

Description

This article describes how to fix HA (High Availability) cluster upgrade failure which results to each firewall in cluster having different OS version.

Scope

FortiGate HA Active Passive.

Solution

Uninterruptible HA cluster upgrade mode (the default) will upgrade the secondary device before the primary.

In most failure cases with this option, the secondary device is successfully upgraded to a newer version while the primary device stays on the current OS version.

This problem can occur with any OS version and device model.   

Pre-checklist:

1. Perform 'diag sys flash list' on both units:

The primary unit most likely has no newer image on its flash storage:

diagnose sys flash list
Partition Image TotalSize(KB) Used(KB) Use% Active
1 FG100F-6.04-FW-build1914-211117 253920 102616 40% No
2 FG100F-7.00-FW-build0367-221005 253920 110112 43% Yes
3 ETDB-90.07704 3021708 790996 26% No
Image build at Oct 5 2022 22:02:56 for b0367

2. Execute 'diagnose sys ha dump-by kernel' on both firewalls:

diagnose sys ha dump-by kernel

<hatalk> HA information.
<hatalk> group_id=13, nvcluster=1, mode=2, load_balance=0, schedule=3, ldb_udp=0.
<hatalk> nvcluster=1, mode=2, ses_pickup=1, delay=0, load_balance=0
schedule=3, ldb_udp=0, upgrade_mode=0. <----- Upgrade_mode should be '0'. If not, reboot the firewall to reset this to 0.

3. HA session sync still takes place.

Since each unit has a different OS version, the cluster status will be out-of-sync.       

However, the session sync should still be running.

If all prerequisites are met, follow the steps below. Otherwise, contact Fortinet TAC for additional help.

1. Perform the cluster upgrade once again: FortiGate will automatically upgrade the device without the latest firmware.

• Manually download the firmware image from the support site. 
• Reboot both HA cluster members.
• Then proceed with the upgrade using the downloaded image.

If this step did not work and it is not possible to upload the image, follow step 2:

2. Failover the FortiGate: make the secondary device the new primary device and perform the cluster upgrade.

In this case, be sure that the secondary can take over the traffic in normal conditions.

3. Downgrade the device with the latest version (which is usually the secondary device) and perform the normal HA upgrade procedure.

Downgrade through the CLI with the following command:

execute set-next-reboot {primary | secondary}  <-- Make sure the older image is there.

See this article for more information.

If none of the steps above work, try the following:

4. Take the following steps with physical access to the device:

1. Separate one of the firewalls from the HA cluster, but back up the config first.
2. Turn off the firewall that will be taken out from the cluster to avoid split brain.
   Or:
   To prevent inducing a split brain through an alternative method, disconnect from HA. To do this, follow the steps below:

• Before disconnecting the HA Cables, disconnect traffic cables first.
• Do not remove only the HA Cables, as doing so will cause both devices to consider themselves as the master units.
• Remove all cables from the secondary device (starting with the data cables and then the HA cables). Connect to this unit directly through a PC and manually perform an upgrade.
• Once done, connect the HA cables first, allow the system to sync, and then connect the data cables.

3. Install the intended OS to this firewall.
4. Always verify the downloaded OS.
5. Rebuild the HA cluster.

Note:

If the problem still persists after following the steps above, contact Fortinet TAC for further assistance.

Related documents:

• Technical Tip: Manual upgrade procedure of a FortiGate HA cluster .
• Technical Tip: Restoring HA master role after a failover using 'diag sys ha reset uptime' (ha 'set o....
• Force HA failover for testing and demonstrations - FortiOS Docs.
• Technical Tip: How to use failover flag to change Active unit .