|
Upgrading Fortigate in an HA cluster:
- Upload the image on the Primary device.
- The primary node pushes the firmware image to the member node (Secondary device).
- Upgrade starts, the secondary device starts the upgrade first and reboots, after upgrade the secondary node sends the primary node an acknowledgment that the upgrade has been completed.
- Once the Secondary is up Primary node starts the upgrade and it reboots. When the system is rebooting, a member node assumes primary status, and the traffic fails over from the former primary node to the new primary node.
- After the upgrade process is completed, the system determines whether the original node becomes the primary node, according to the HA Override setting.
Make sure uninterrupted upgrade is enabled, check the config using 'show full system ha':
config system ha
set uninterruptible-upgrade enable
end
Follow the below guide for more info on the HA election process: Technical Tip: FortiGate HA Primary unit selection process when override is disabled vs enabled.
Below is how HA FortiGates negotiate during the upgrade and status in different stage:
Stage 1: Upgrade start.
Primary:
2023-02-22 12:33:03 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=1(ENTER), daemon_bits=0x00000000
2023-02-22 12:33:03 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=2(PRIMARY), daemon_bits=0x00000000
2023-02-22 12:33:03 Send image to HA secondary.
2023-02-22 12:33:06 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=3(SENT-IMAGE), daemon_bits=0x00000000
Secondary:
2023-02-22 12:33:04 Get image from ha primary OK.
2023-02-22 12:33:04 Verifying the integrity of the firmware image...
2023-02-22 12:33:05 Check image OK.
2023-02-22 12:33:05 Please wait for system to restart.
2023-02-22 12:33:13
Firmware upgrade in progress ...
Done.
The system is going down NOW !!
Stage 2: After the Secondary FortiGate reboot, HA will failover in order for the Primary to upgrade.
Primary:
2023-02-22 12:33:12 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=4(CHECK-SECONDARY), daemon_bits=0x00000000
2023-02-22 12:34:01 <hatalk> new member 'FGVM020000-----6' is added into group
2023-02-22 12:34:03 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=14(WAIT-DAEMONS), daemon_bits=0x00000170
2023-02-22 12:34:03 <hatalk> check_daemon_wait: ClearPass sync is done, daemon_bits=0x00000070
2023-02-22 12:34:18 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=5(SECONDARY-UP), daemon_bits=0x00000000
2023-02-22 12:34:18 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=6(RELEASE-UPGRADE-FLAG), daemon_bits=0x00000000
2023-02-22 12:34:19 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=7(RELEASE-DONE), daemon_bits=0x00000000
2023-02-22 12:34:19 Wait for first secondary to become new primary
2023-02-22 12:34:20 <hatalk> entering hatalk_upgrade_timer_func: uprade_state=8(CHECK-STATE), daemon_bits=0x00000000
2023-02-22 12:34:20 <hatalk> vcluster_0: state changed, 2(work)->3(standby)
2023-02-22 12:34:20 <hatalk> [hatalk_work_as_slave:391] vcluster_0: work_as_secondary
2023-02-22 12:34:21 <hatalk> leaving hatalk_upgrade_timer_func: uprade_state=9(BECOME-SECONDARY), daemon_bits=0x00000000
Secondary:
2023-02-22 12:34:21 <hatalk> vcluster_0: set flag HATALK_VMEMBER_F_CLAIM_AS_PRIMARY
2023-02-22 12:34:21 <hatalk> cfg_changed is set to 1: hatalk_work_as_master
2023-02-22 12:34:21 <hatalk> vcluster_0: update state_change_time, work-as-primary
Primary:
2023-02-22 12:34:30
Firmware upgrade in progress ...
Done.
The system is going down NOW !!
Stage 3: The Primary reboots, if failover is enabled or failover is disabled but uptime is still within the margin time, the Primary will take over again.
Secondary:
2023-02-22 12:35:17 <hatalk> vcluster_0: update state_change_time, work-as-secondary
2023-02-22 12:35:17 <hatalk> vcluster_0: unset flag HATALK_VMEMBER_F_CLAIM_AS_PRIMARY
On secondary:
149 # get sys ha status
Primary selected using:
HA Health Status: OK
Model: FortiGate-VM64-KVM
Mode: HA A-P
Group: 11
Debug: 0
Cluster Uptime: 0 days 0:10:20
Cluster state change time: 2023-02-22 12:35:17
<2023/02/22 12:35:17> FGVM020000-----2 is selected as the primary because its override priority is larger than peer member FGVM020000-----6.
<2023/02/22 12:34:29> FGVM020000-----6 is selected as the primary because it's the only member in the cluster.
<2023/02/22 12:34:21> FGVM020000-----6 is selected as the primary because UPGRADE_SECONDARY flag is set on peer member FGVM020000-----2.
<2023/02/22 12:34:00> FGVM020000-----2 is selected as the primary because UPGRADE_PRIMARY flag is unset on peer member FGVM020000-----6.
|
