Technical Tip: How to upgrade and troubleshoot a FortiManager/FortiAnalyzer HA cluster

Description

This article provides a step-by-step guide on upgrading the firmware of a FortiManager or FortiAnalyzer cluster while ensuring system stability, security, and minimal disruptions. Additionally, it includes troubleshooting steps in case issues arise during the upgrade.

Scope

FortiManager, FortiAnalyzer.

Solution

This solution: 

• Applies to FortiManager and FortiAnalyzer High Availability (HA) clusters.

• Relevant for administrators responsible for system maintenance and firmware upgrades.

• Covers both GUI-based and CLI-based upgrade procedures.

Prerequisites:

• Make sure to get administrator access to the FortiManager/FortiAnalyzer.

• Review the firmware release notes for new features, fixes, and known issues.

• Back up all configurations and data before proceeding.

• Plan a maintenance window, as services may be temporarily disrupted during the upgrade.

• If FortiAnalyzer Features are enabled on FortiManager, they will be automatically deactivated during an upgrade to FortiManager v7.0.0 or later.

Steps to upgrade the HA cluster:

1. Pre-Upgrade Preparation.

• Backup Configuration and Data: Before performing the upgrade, back up all configurations and data to avoid potential loss.

• Check Compatibility and Release Notes: Ensure the new firmware version is compatible with the current setup and review the firmware release notes for changes and potential issues.

• Schedule the Upgrade: Since the upgrade process does not cause a prolonged disconnection, it can be performed outside a scheduled maintenance window if necessary. Disable FortiAnalyzer Features (if applicable). If HA is enabled on FortiManager, FortiAnalyzer Features must be disabled before proceeding.

2. Cluster Upgrade Procedure.

Upgrade the cluster via the GUI:

1. Upgrade the secondary FortiAnalyzer first (under the Dashboard -> System Information widget).

2. Wait for the secondary to come back online and confirm it has joined the HA cluster.

3. Verify HA synchronization/status.

4. Then upgrade the primary FortiAnalyzer.

Note: For a FortiAnalyzer HA cluster, the secondary unit must be upgraded first, followed by the primary unit. Do not follow the FortiManager HA upgrade sequence, as the upgrade procedure differs between FortiManager and FortiAnalyzer. 

• Upgrade the Cluster via CLI via FTP server: Use the following command to upgrade firmware stored on an FTP or TFTP server using the following CLI command:

execute restore image {ftp | tftp} <file path to server> <IP of server> <username on server> <password>

3. Post-upgrade verification:

• Check Cluster Status: Verify that all units are synchronized and running the new firmware version.

• In the GUI, navigate to System Settings -> HA.

• In the CLI, enter: 

diagnose ha stats

Cluster-ID : 1
Debug : off
File-Quota : 4096
HA Health Status : OK
HA Role : Primary
FMG-HA Status : Synchronized
State Model : FortiManager-VM64
HB-Interval : 10
HB-Lost-Threshold : 30
HA Primary Uptime : Fri Feb 14 06:33:35 2025
HA Primary state change timestamp : 
Primary : FMG-VM64, FMG-VM0000000013,
System Usage stats :
FMG-VM0000000013(updated 0 seconds ago): average-cpu-user/nice/system/idle=0.05%/0.00%/0.00%/99.95%, memory=14.56

• The following are other useful troubleshooting commands that can be used on both the Primary and Secondary units of FortiManager or FortiAnalyzer if an issue is observed in the HA-Status:

diagnose debug reset
diagnose debug application ha 255
diagnose debug enable

• A system reboot may be required to apply the change.

4. Monitoring and Troubleshooting.

• Checking Logs in Real-Time.

During the upgrade, it is recommended to use the console to monitor log output in real time.

Example Log Output with Warnings and Errors:

If any errors arise, investigate and resolve them before continuing with normal operations.

• Checking FortiManager Event Logs:

  • After the upgrade, check the FortiManager Event Logs for recorded messages under System settings -> Event logs.

  • If errors are found, apply necessary fixes before putting the system into full production.

5. Additional Considerations.

• SQL Database Rebuild (KB article link in the 'Related documents' section):

  • Upgrading may trigger an automatic SQL database rebuild.

  • During this process, new logs will be stored and visible, but older logs may be temporarily unavailable.

  • The rebuild duration depends on database size.

Related documents:

Upgrade Guide FortiManager 7.6.2 (p22-23)

Enable or disable FortiAnalyzer features

Technical Tip: FortiAnalyzer SQL database delete and rebuild