Technical Note : Manual upgrade procedure of a FortiGate HA cluster

[rmetzger]

Purpose

When operating in an HA cluster, FortiGate devices can be upgraded automatically with the HA option "uninterruptable-upgrade" which is enabled by default.  The advantages of the uninterruptable upgrade process are:

- Allows the Administrator to upgrade all devices of a cluster in a single operation (from the GUI, click Dashboard --> Status -->  Firmware Version --> upgrade).
- It upgrades (all) Slave(s) unit(s) before upgrading the Master, making the necessary failover for a minimum downtime.

In some occasions however, the Administrator may want to keep a control of the full upgrade process by upgrading the devices one by one. One reason for choosing this option could be to facilitate a roll-back since one FortiGate will still be running the former firmware and configuration.

Scope

All FortiGate and FortiOS

Diagram


Expectations, Requirements

The following procedure is proposed if a manual upgrade is desired with the following assumptions:

- Two devices (FGT-1 and FGT-2) are forming the cluster. Otherwise repeat the operations on each FortiGate.
- FGT-1 is Master.
- Link monitor is enabled (not mandatory).
- Mngt1 is used a dedicated management (not mandatory).
- HA heartbeat ports are dedicated (not mandatory, but the procedures assumes this).

Configuration

Step 1 : Isolate and prepare FGT-2 (FGT-2 is HA Slave)

1.1 - Disconnect all physical network ports from FGT-2 , this means all ports except Mngt1 (if applicable) and HA ports. At this moment, FGT-2 is no longer eligible as Master (if port monitoring is enabled), and isolated from the network ; FGT-1 handles traffic as normal. Note that instead of disconnecting physically the cables, another option can be to disable the ports from the L2 switch to which the FortiGate is attached.

1.2 - Disconnect now also the HA port(s). At this point, FGT-2 is now totally isolated; FGT-1 handles traffic as normal.

1.3 - Proceed to the upgrade of FGT-2 via mngt1 or any other means to get IP connectivity.

1.4 - Once FGT-2 is rebooted with the new firmware, make all necessary verifications. For example, save the configuration of FGT-2 and make a diff with FGT-1. This will tell what are the differences between the two versions (for example, some default settings can have changed).

1.5 - If the cluster contains more than two devices, repeat only step 1.1 and step 1.2 for all remaining FortiGates (FGT-3, FGT-4...)

Step 2 : Swap FGT-1 and FGT-2

2.1 - Disconnect all cables from FGT1 including HA cables but not mngt1. Note that instead of disconnecting cables, another option can be to disable the ports from the L2 switch to which the FortiGate is attached.

2.2 - As quickly as possible, connect all appropriate cables from FGT-2 (or re-enabled the L2 switch ports). At that point, traffic will be impacted but should recover quickly (this will depend on the applications, but most of the common traffic such as WEB browsing, SMTP, VoIP(RTP), should recover quickly).  Check any restrictions beforehand if required. Note that with this procedure, sessions are not synced across FGT-1 and FGT-2, hence a minor impact on traffic is expected.

2.3 - Make all necessary sanity checks and service verification.

Step 3 : After a probation period, FGT-1 can be upgraded and re-enter the cluster

3.1 - Once all services protected by the FortiGate have been verified and after a probation period left to the discretion of the administrator, proceed to the upgrade of FGT-1 via mngt1 or any other means to get IP connectivity.

3.2 - Once FGT-1 is rebooted with the new firmware, make all necessary verification . For example, save the configuration of FGT-2 and make a diff with FGT-1. There should be no difference. Another option is to compare the HA checksums which should now be the same on both devices (CLI command "diagnose sys ha showcsum").

3.3 - Optional steps if FGT-2 must stay Master:

3.3.1 - On FGT-1 reduce HA priority to 10 less than FGT-2 (for example: if FGT-2 HA priority is 100, set FGT-1 to 90).

3.3.2 - Make sure HA override is disabled on both devices.

3.3.3 - Reconnect only HA ports of FGT-1 (since network monitored ports are still down on FGT-1, it cannot become Master).

3.3.4 - Verify that the cluster is up and that both configurations are in sync by checking the checksum on both devices (should be similar to the checksum seen in step 3.2).

3.4 - Reconnect now all relevant ports of FGT-1 similarly to FGT-2 (or enable the L2 switch ports). At that point, FGT-1 should stay Slave or become Master, depending on the steps above and your requirements.

3.5 : If the cluster contains more than 2 devices, repeat step 3 for all remaining FortiGates.

Step4 : Test FGT-1 with a fail-over (if FGT-1 is still HA Slave)

This can be achieved by either:

- Disconnecting a monitored port of FGT-1.

- The CLI command "diagnose sys ha reset-uptime" passed on FGT-2.