Troubleshooting Tip: Upgrade option missing on the secondary GUI in FGCP MVC upgrade mode
Description
This article describes how to upgrade an individual secondary member of an FGCP cluster from the CLI when the cluster is in multi-version cluster (MVC) upgrade mode and the upgrade option is missing from the secondary's GUI.
When 'upgrade-mode' is set to either 'local-only' or 'secondary-only', a secondary is upgraded by uploading the image on that secondary itself. The Administration Guide allows this from the secondary's CLI console or from its GUI over the HA management interface.
On FortiOS 7.6, the 'System -> Firmware & Registartion' page of the secondary's GUI shows no upgrade button in either MVC mode. This is bug ID 1226122, resolved in FortiOS 8.0.1 and not fixed in the 7.6 branch at the time of writing. Until a fixed release is installed, the secondary must be upgraded from its CLI.
Scope
FortiGate FGCP HA clusters (two or more members) running FortiOS 7.6.x on the secondary, with 'upgrade-mode' set to either 'local-only' or 'secondary-only'. Verified on a three-member FortiGate-VM64-KVM A-P cluster with the secondary on 7.6.7. Resolved in FortiOS 8.0.1 (bug ID 1226122).
Solution
Upload the image from the secondary's CLI with 'execute restore image'. Only that secondary reboots, the primary and any other secondary keep passing traffic.
Symptom:
On the secondary's GUI, reached over its HA management interface, 'System -> Firmware & Registration' shows the cluster in MVC upgrade mode but offers no upgrade button. The image cannot be uploaded from the GUI.
Confirm the mode from the secondary's CLI:
show system ha | grep upgrade-mode
set upgrade-mode secondary-only
Workaround:
Step 1. On the secondary to be upgraded, set the upgrade mode to 'local-only'. The warning is expected:
config system ha
set upgrade-mode local-only
end
Warning: Setting upgrade-mode to be local-only or secondary-only might cause that HA members are running different image versions after upgrade!
Step 2. Place the image on a TFTP server reachable from the secondary unit's HA management interface, then run the following from the secondary unit's CLI:
execute restore image tftp <image_file.out> <tftp_server_ip>
The FTP and USB variants of 'execute restore image' can be used instead of TFTP.
Step 3. Answer the prompts. While members run different builds, the cluster reports out of sync; this is expected in MVC mode because configuration is not synchronized between members on different builds.
This operation will replace the current firmware version and reboot the system!
Do you want to continue? (y/n)y
The HA cluster this device belongs to is out of sync. Proceeding with the upgrade may result in the cluster continuing to remain out of sync, and may lead to a split brain condition
Do you want to continue? (y/n)y
...
Get image from tftp server OK.
Verifying the signature of the firmware image.
Image verification OK!
...
Please wait for system to restart.
Firmware upgrade in progress ...
Done.
The system is going down NOW !!
If FortiOS warns that the path is not a recommended upgrade path, check the Upgrade Path Tool before answering 'y'.
Step 4. After the reboot, the secondary rejoins and resynchronizes with the primary. The console shows the following:
secondary's external files are not in sync with the primary's, sequence:1. (type CERT_LOCAL)
...
secondary succeeded to sync external files with primary
secondary's configuration is not in sync with the primary's, sequence:0
...
secondary starts to sync with primary
logout all admin users
Step 5. Verify the version and the cluster state:
get system status | grep Version
get system ha status
In the FortiGate GUI, 'System -> HA' lists every member as Synchronized once all run the same build.
Step 6. When every member runs the target build, set 'upgrade-mode' back to the cluster's normal value (default 'uninterruptible'). The 'local-only' and 'secondary-only' modes are intended only for temporary MVC operation.
Related documents:
FortiOS 7.6 Administration Guide: MVC Feature
FortiOS 8.0.1 Release Notes: Resolved issues
Technical Tip: FortiGate HA upgrade procedure and the status during the upgrade
