Description
This article describes how to troubleshoot common issues when upgrading FortiGates (standalone or HA clusters) using FortiManager. It covers error patterns, process stages, debug steps, and configuration suggestions to improve upgrade reliability.
Scope
FortiManager, FortiGate.
Solution
Prerequisite:
The FortiGate requires a valid upgrade license (FMWR license) to proceed. Without it, the upgrade process will time out based on the timeout value configured on FortiManager. In certain cases, the FortiGate may not have internet access to verify the license validity directly. In such scenarios, it needs to use FortiManager (acting as a FortiGuard server) to retrieve the license information.
To verify the FortiGate license status from both FortiManager and FortiGate, use the CLI commands listed below. The following table summarizes the commands, their purposes, and relevant notes:
| Platform | Command | Purpose | Notes |
| FortiManager | diagnose fmupdate fds-dump subs | Displays FortiGate subscription/license info | Run from FortiManager CLI |
| FortiGate | diagnose autoupdate versions | Shows license status and FortiOS identification | Run from FortiGate CLI: Look under 'Device and FortiOS Identifications' section |
The same information on the GUI:
Understanding the Upgrade Flow:
The tables below outline the various steps FortiManager takes during the FortiGate upgrade process. These steps can be found under System Settings -> Task Monitor or via CLI debug using the command: 'diagnose fwmanager fwm-log dump'.
- FortiManager and FortiGuard:
|
Stage |
Description |
|
Upgrade start |
Initiates firmware upgrade from FortiManager |
|
Wait for the image ready |
Firmware image preparation in FortiManager |
|
Image ready |
The image is ready to be transferred |
- FortiManager to FortiGate Communication:
|
Stage |
Description |
|
FGT_start_tunnel |
Establishes communication tunnel |
|
FGT_check_disk |
Verifies FortiGate disk space |
|
FGT_send_image |
Firmware sent to FortiGate |
|
FGT_sleep |
Wait between actions |
|
FGT_check_status |
Polls FortiGate for status, version |
|
FGT_image_upgrade |
Triggers the upgrade |
|
Image upgrade done |
FortiGate confirms completion |
- Post-upgrade by FortiManager:
|
Stage |
Description |
|
Retrieve configuration |
Downloads updated FortiGate config |
|
Retrieve support data |
Collects diagnostic data |
|
Revision diff |
Compares changes pre/post-upgrade |
|
Health check |
Validates device health |
|
Upgrade done successfully |
Marks task as completed |
Common reasons for upgrade fFailures:
|
Category |
Cause |
Recommended Action |
|
Corrupted image |
Image file is incomplete or invalid (ex: The new image does not have a valid RSA signature.) |
Validate image checksum (MD5/SHA256) |
|
Compatibility mismatch |
FortiManager/ADOM not compatible with FortiGate target firmware |
Check compatibility matrix |
|
Connectivity issues |
FortiGate unreachable or unstable during upgrade, FortiGuard server not reachable |
Ensure stable links, check FortiGuard access (diag fmupdate view-linkd-log fds) |
|
Resource limitations |
FortiManager low on memory, CPU, or disk |
Monitor FortiManager resource usage |
|
Config conflict |
Pending changes or mismatches |
Commit changes before upgrade |
|
Process crash |
Example: dmserver segfault during upgrade |
Inspect crashlogs on FortiManager |
|
Known firmware bug |
Upgrade fails due to FortiOS issue |
Confirm if resolved in newer FortiOS |
|
No upgrade license |
FortiGate firmware upgrade not licensed |
Verify Firmware upgrade licensing on FortiGate unit(s) |
Typical error codes and fixes:
|
Error Code / Message |
State Code |
Likely Cause |
Recommended Fix |
|
upgrade image FAILED, r = -4 |
State=5 (error) |
Timeout during upgrade |
Increase check-status-timeout to 3600 |
|
desc=FGT_check_status failed: timeout |
State=5 (error) |
Status polling timed out |
Same as above |
|
closed:update taskline failed |
State=5 (error) |
Task prematurely closed |
Check logs, ensure no reboot issue |
|
General Timeouts |
N/A |
Connection lost or slow |
Extend timeouts, check reachability |
Default FortiManager firmware settings compared to recommended configurations for preventing upgrade issues:
- Default firmware configuration on FortiManager:
- Fine-tuned firmware configuration on FortiManager:
Important:
Most issues found originate from FortiGates. If a task fails at or after the step 'FGT_send_image src=FortiManager' in the task monitor, the issue is not with FortiManager. In this case, open a FortiGate support ticket using the FortiGate's serial number.
Key CLI for Troubleshooting:
On FortiManager:
diagnose debug reset
diagnose debug console timestamp enable
diagnose debug application fgfmsd 255
diagnose debug application depmanager 255
diagnose fwmanager fwm-log dump
On FortiGate:
The debug commands below do not display output during the upgrade process. To monitor progress, use the console directly (below capture):
diagnose debug reset
diagnose debug console timestamp enable
diagnose debug application fgfmd 255
diagnose debug cli 8
Note:
In an HA configuration scenario, the secondary node of the cluster is upgraded before the primary node.
Troubleshooting Timeout Issues:
- health-check-timeout: Default is 600s (10 min). If the upgrade task stops around 9m50s, this likely caused the failure (retreivehaconffail). Set to 1200 or more for HA upgrades:
config fmupdate fwm-setting
config upgrade-timeout
set health-check-timeout 1200
end
end
Advanced Log Analysis (Examples):
FGT_check_status loops with no upgrade progress:
The upgrade process begins, and the FortiGuard image downloads successfully. However, the FGT_VM64 image upgrade gets stuck at the step 'FGT_send_image done: src=FortiManager' for an extended period, eventually failing with the error 'FGT_check_status failed: timeout.'
As a result, the FortiGate remains on the old firmware version (e.g., build 3462 instead of the expected 3510), while FortiManager continuously polls without detecting a version change, causing the task to stall at 63% (see screen capture).
This issue occurs when FortiManager follows the recommended upgrade path, and 'Let Device Download Firmware From FortiGuard' is disabled (Upgrade in multiple steps).
Root cause:
A FortiOS issue has been identified and resolved in v7.6.3. However, for versions v7.6.1 and v7.6.2, a workaround is available: enable the 'Let Device Download Firmware From FortiGuard' option on the FortiManager side.
After 1 hour and 15 minutes, the task times out:
Additional Tips:
Disable disk check (known to cause timeouts in some environments):
config fmupdate fwm-setting
set auto-scan-fgt-disk disable
set check-fgt-disk disable
end
Ensure HA clusters are synced and secondary upgraded first.
Upload firmware manually if there is a bug/network connectivity.
Related documents:
- Technical Tip: How to upgrade FortiGate using FortiManager
- fwm-setting - FortiManager 7.6.3 CLI reference
- Troubleshooting Tip: FortiGate failed to upgrade using the Firmware Template
- Technical Tip: How to download and import firmware images into FortiManager
- Technical Tip: How to perform scheduled upgrade for FortiGates using FortiManager
-
Technical Tip: Understanding FortiGate upgrade process when using FortiManager
