Technical Tip: Manual upgrade procedure of a FortiGate HA cluster

Description

This article describes how, when operating in an HA cluster, FortiGates can be upgraded automatically with the HA option 'uninterruptable-upgrade' enabled by default.  The advantages of the uninterruptable upgrade process are:

• Allows the Administrator to upgrade all devices of a cluster in a single operation (from the GUI, select Dashboard -> Status ->  Firmware Version -> Upgrade).

• It upgrades (all) secondary unit(s) before upgrading the primary unit, making the necessary failover for a minimum downtime.

However, on some occasions, the Administrator may want to control the full upgrade process by upgrading the devices one by one. One reason for choosing this option could be to facilitate a rollback since one FortiGate will still be running the former firmware and configuration.

Scope

FortiGate.

Solution

Expectations, requirements:

The following procedure is proposed if a manual upgrade is desired, with the following assumptions:

• Two devices (FGT-1 and FGT-2) form the cluster. Otherwise, repeat the operations on each FortiGate.

• FGT-1 is the primary unit.

• Link monitor is enabled (not mandatory).

• Mngt1 is used a dedicated management (not mandatory).

• HA heartbeat ports are dedicated (not mandatory, but the procedures assume this).

Configuration:

Step 1: Isolate and prepare FGT-2 (FGT-2 is HA secondary).

1. Disconnect all physical network ports from FGT-2, which means all ports except Mngt1 (if applicable) and HA ports. At this moment, FGT-2 is no longer eligible as the primary unit (if port monitoring is enabled), and is isolated from the network; FGT-1 handles traffic as normal. Note that instead of disconnecting physically the cables, another option can be to disable the ports from the L2 switch to which the FortiGate is attached. Disconnect now also the HA port(s). At this point, FGT-2 is now totally isolated; FGT-1 handles traffic as normal.

2. Proceed to the upgrade of FGT-2 via Mngt1 or any other means to get IP connectivity.

3. Once FGT-2 is rebooted with the new firmware, make all necessary verifications. For example, save the configuration of FGT-2 and make a diff with FGT-1. This will make it possible to tell what the differences are between the two versions (for example, some default settings may have changed).

4. If the cluster contains more than two devices, repeat only steps 1.1 and 1.2 for all remaining FortiGates (FGT-3, FGT-4...).

Step 2: Swap FGT-1 and FGT-2:

1. Disconnect all cables from FGT1, including HA cables, but not mngt1. Note that instead of disconnecting cables, another option can be to disable the ports from the L2 switch to which the FortiGate is attached.

2. As quickly as possible, connect all appropriate cables from FGT-2 (or re-enable the L2 switch ports). At that point, traffic will be impacted but should recover quickly (this will depend on the applications, but most of the common traffic, such as Web browsing, SMTP, and VoIP(RTP), should recover quickly). Check any restrictions beforehand if required. Note that with this procedure, sessions are not synced across FGT-1 and FGT-2.

3. A minor impact on traffic is expected.

4. Make all necessary sanity checks and service verification.

Step 3: After a probation period, FGT-1 can be upgraded and re-enter the cluster.

1. Once all services protected by the FortiGate have been verified and after a probation period left to the discretion of the administrator, proceed to the upgrade of FGT-1 via mngt1 or any other means to get IP connectivity.

2. Once FGT-1 is rebooted with the new firmware, make all necessary verifications. For example, save the configuration of FGT-2 and make a diff with FGT-1. There should be no difference. Another option is to compare the HA checksums, which should now be the same on both devices (with the CLI command 'diagnose sys ha checksum cluster').

3. Optional steps if FGT-2 must stay primary:

• On FGT-1, reduce HA priority to 10 less than FGT-2 (for example: if FGT-2 HA priority is 100, set FGT-1 to 90).
  Make sure the HA override is disabled on both devices.

• Reconnect only the HA ports of FGT-1 (since the network monitored ports are still down on FGT-1, it cannot become the primary unit).

• Verify that the cluster is up and that both configurations are in sync by checking the checksum on both devices (should be similar to the checksum seen in step 3.2).

4.  Reconnect now all relevant ports of FGT-1 similarly to FGT-2 (or enable the L2 switch ports). At that point, FGT-1 should stay secondary or become the primary, depending on the steps above and the appropriate requirements for the setup.

5. If the cluster contains more than 2 devices, repeat step 3 for all remaining FortiGates.

Step 4: Test FGT-1 with a fail-over (if FGT-1 is still the HA secondary).

This can be achieved by either:

• Disconnecting a monitored port of FGT-1.

• Using the CLI command 'diagnose sys ha reset-uptime' on FGT-2.

Note:

• Starting from FortiOS v7.6.1, setting an HA password is required when building a new cluster. However, if a cluster is upgraded from versions 7.0.x, 7.2.x, or 7.4.x without an HA password, the system will bypass this requirement during the upgrade. Keep in mind that any later changes to the 'config system ha' configuration will trigger the password check, making it necessary to configure the HA password on all members of the cluster.

• Although this article is written from an upgrade perspective, the same manual procedure applies equally to a firmware downgrade of an HA cluster. However, the uninterruptable- upgrade HA option (and its automatic secondary-first, failover-to-primary behavior) is only supported for upgrades, not downgrades - a downgrade must always be performed manually using the steps above.