Technical Tip: HA upgrade fails when upgrading from v7.0.16 to v7.2.10 or v7.4.5 firmware on FortiGate-90G/91G and 120G/121G models

HA cluster upgrades could fail on the FortiGate-90/91G and 120/121G models due to high BIOS security level. This is related to known issue 1102588.

get system status
Version: FortiGate-120G v7.0.16,build7536,241003 (GA.M)
Security Level: High
Firmware Signature: certified
Virus-DB: 92.19222(2024-12-03 21:26)
Extended DB: 92.19222(2024-12-03 21:25)
AV AI/ML Model: 3.12007(2024-12-03 20:45)
IPS-DB: 29.00916(2024-12-05 02:16)
IPS-ETDB: 0.00000(2001-01-01 00:00)
APP-DB: 29.00916(2024-12-05 02:16)
INDUSTRIAL-DB: 6.00741(2015-12-01 02:30)
IPS Malicious URL Database: 5.00254(2024-12-06 15:16)
Serial-Number: FG120GTK24007657
BIOS version: 06000104
System Part-Number: P28808-04
Log hard disk: Not available
Hostname: DRHS-Main-120G-TOP
Private Encryption: Disable
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: a-p, primary
Cluster uptime: 2 days, 21 hours, 13 minutes, 16 seconds
Cluster state change time: 2024-12-06 14:54:20
Branch point: 0667
Release Version Information: GA
System time: Sat Dec 7 15:09:28 2024
Last reboot reason: warm reboot

During the upgrade process, FortiGate could encounter the error 'firmware failed signature validation', and the upgrade process will be aborted.

diagnose debug application hatalk 255

Debug messages will be on for 30 minutes.

diagnose debug application hasync 255

Debug messages will be on for 30 minutes.

diagnose debug en

<hasync> reap child: pid=25485, status=0

<hatalk> vcluster_0: ha_prio=1(secondary), state/chg_time/now=3(standby)/1733453659/1733879631

<hasync> reap child: pid=25486, status=0

<hatalk> vcluster_0: ha_prio=1(secondary), state/chg_time/now=3(standby)/1733453659/1733879641

<hasync:WARN> conn=0x36f2af50, peer closed the connection: dst=169.254.0.1, sync_type=18(byod)

<hatalk> vcluster_0: ha_prio=1(secondary), state/chg_time/now=3(standby)/1733453659/1733879651

<hatalk> vcluster_0: ha_prio=1(secondary), state/chg_time/now=3(standby)/1733453659/1733879661

<hatalk> parse options for 'FG120GTK24007657', packet_version=58

<hatalk> cfg_changed is set to 1: intf-changed

<hatalk> vcluster_0: vmember 'FG120GTK24007657' updated, override=0, usr_priority=200, mondev/pingsvr=0/0, uptime/reset_count=1950/0, flag=0x00000009

<hatalk> vcluster_0: reelect=1, vmember updated

<hatalk> vcluster_0: ha_prio's are not changed after HA election

<hatalk> cfg_changed is set to 0: hatalk_packet_setup_heartbeat

<hatalk> setup new heartbeat packet: hbdev='port1', packet_version=39

<hatalk> options buf is small: opt_type=41(DEVINFO), opt_sz=13806, buf_sz=1231

<hatalk> pack compressed dev_info: dev_nr=33, orig_sz=13800, z_len=253

<hatalk> heartbeat packet is set on hbdev 'port1'

<hatalk> setup new heartbeat packet: hbdev='port2', packet_version=39

<hatalk> options buf is small: opt_type=41(DEVINFO), opt_sz=13806, buf_sz=1231

<hatalk> pack compressed dev_info: dev_nr=33, orig_sz=13800, z_len=253

<hatalk> heartbeat packet is set on hbdev 'port2'

<hasync> reap child: pid=25491, status=0

<hasync:WARN> conn=0x36f2af50, peer closed the connection: dst=169.254.0.1, sync_type=5(conf)

<hasync:WARN> conn=0x36f2af50, peer closed the connection: dst=169.254.0.1, sync_type=10(cli-command)

Get image from ha primary OK.

Verifying the integrity of the firmware image...

******WARNING: This firmware failed signature validation.******

Fortinet cannot verify the authenticity of this firmware and therefore

there may be a risk that the firmware contains code unknown to Fortinet.

In short, Fortinet cannot validate the firmware and makes no warranties

or representations concerning the firmware.

Installation Aborted.

Workarounds:

There are two workarounds for this issue:

Workaround 1:

Lower the BIOS security level on each FortiGate of the HA cluster, perform the upgrades one at a time and, once the upgrade is done, ask the admin to switch the BIOS level back to high.

To change the security level:

1. Connect to the console port of the FortiGate.

2. Reboot the FortiGate (execute reboot) and enter the BIOS menu.

3. Press [I] to enter the System Information menu

4. Press [U] to enter the Set security level menu

5. Enter the required security level.

6. Continue to boot the device.

The following is a way to do this with minimal downtime for the HA cluster:

1. Change the BIOS security level first on the secondary unit (say FGT-2) by following the above steps with a reboot, this way the primary (FGT-1) continues to handle traffic without disruption.
2. After the secondary (FGT-2) comes up with new BIOS setting, do a failover (using diagnose sys ha reset-uptime on FGT-1) so that the FGT-2 becomes the new Primary and starts forwarding traffic, and FGT-1 becomes the new Secondary. Now change the BIOS security level on the new Secondary unit (FGT-1) with the same steps as in the previous section with a reboot, it should come up and stay as Secondary after the reboot.
3. Now with FGT-2 as primary and FGT-1 as Secondary, initiate the firmware upgrade from FGT-2 (by uploading image locally or using Fortiguard). This will trigger FGT-1 as the Secondary unit to load the image first and complete the upgrade, followed by FGT-2 to complete its upgrade. Once the upgrades are complete, FGT-1 will become Primary again and FGT-2 as Secondary due to the order of our reboots.
4. To increase the BIOS security level back to High from low, start with the Secondary (FGT-2) unit by rebooting it with console connection and follow the 1-6 steps in the previous section to enter the BIOS menu and increase the security level back to High (or which ever level it was before). Once it is complete, do a failover with reset-uptime command on FGT-1 so that FGT-2 now becomes Primary and is carrying traffic. Once FGT-1 becomes Secondary, repeat the steps to reboot this FortiGate and change its BIOS security level to High. Now both FGT-1 and FGT-2 have their BIOS security level back to High (initial settings).
5. As a last step, to make FGT-1 Primary again (if ha override is disabled), reboot the FGT-2 device (reset uptime might not work now since the duration would be less than 300 seconds). This will make FGT-1 the Primary again.

At the end of the above steps, verify that FGT-1 is Primary and FGT-2 is Secondary, both are upgraded and are in ha sync - using the commands "get sys status", "diagnose sys ha status" and "get sys ha status".

Workaround 2:

Another option is to break the HA Cluster briefly and upgrade the device one at a time, and then re-add the HA cluster once the upgrades are complete. More details are available in Technical Tip: How to break a HA cluster and use one of the members as standalone.

Related articles:
Troubleshooting Tip: Unable to boot the firewall or load firmware image