First firmware upgrade on an HA cluster and I screwed it up.

[jefry876]

We have 2 fortigate 100EF in an HA cluster that needed to be upgraded (were on 7.0.2) and my boss asked if I wanted to give it a shot. He said it was straightforward and that it does the failover itself and all that. Upgrade path was to be .2 -> .5 -> .7 Well I think I got ahead of myself and somehow set the secondary (f2) to upgrade to .7 while the primary (f1) was rebooting to move to .5. This also briefly took down our sites since both FW were down at the same time (we're an e-commerce company).

So now f1 is on 7.0.5 and is the active primary and f2 is on 7.0.7 but f2 is out of sync and f1 still needs to come up to .7... I'm not a real sysadmin or network guy we don't have one. I tried a couple cli commands I saw online. Recalculate, and there was some force sync command too. Neither helped. I also tried the one to force HA failover to make f2 the primary but that didn't either (my boss thought this might help but I guess I wasn't surprised because the HA cluster is basically just 1 box right now I think).

Part of me thinks if I just do the upgrade to .7 on f1 that maybe they'll sync back up and all will be well. Would just have to eat a site outage again for a few minutes while it reboots? Not ideal I know. Another idea the team had was drop f2 out of the cluster but I think without sending someone to the sever farm to be ready to unplug stuff that we would run into network collisions? Any help or thoughts you guys have is appreciated.

[scan888]

Hello,

First it would be a good idea to contact a Fortinet Partner with knowhow with that products for the next time, especially if it's for your main business . There are a reason, why network specialist are existing. ;)

First important thing to know is, that the cluster are in sync only when both Firewalls has the same firmware version. If you have a firmware mismatch the cluster come never in sync.

In your case, and with the minimal information about your infrastructur, I suggest the following steps:

- Create a backup from the firewall with the correct configuration (always, create a backup before you upgrade)

- Shutdown the firewall with the Version 7.0.7

- Upgrade the firewall with the Version 7.0.5 to 7.0.7 -> Wait until upgrade is success

- Start the firewall from the first step again

- Check if both firewalls see each other (System-HA in the Menu)

- Wait 1-2 hours for sync between both members

There is no way without downtime, in my point of view.

If the cluster still "out-of-sync" after a couple of hours, you can start again with your troubleshooting.

[AlexC-FTNT]

You'll learn a lot from this one! :)
If one is in .5 the other in .7, then either disconnect .5 (and upgrade alone), or simply try to start the upgrade again.

[Debbie_FTNT]

Hey jefry,

as your FortiGates are physical appliances, you can take advantage of a fairly easy roll-back feature:
- (hardware) FortiGates usually have two disk partitions, the active one with current firmware, and a secondary with the previous firmware.

- on your FortiGate in 7.0.7, you can roll back to 7.0.5

- you can verify that they are in sync, and then upgrade

You can also, as scan and Alex suggested, upgrade your 7.0.5 unit to 7.0.7 and then sync.

To rollback, you can refer to this KB: https://community.fortinet.com/t5/FortiGate/Technical-Tip-Selecting-an-alternate-firmware-for-the-ne...