Skip to main content
jefry876
jefry876
New Member
October 10, 2022
Question

First firmware upgrade on an HA cluster and I screwed it up.

  • October 10, 2022
  • 3 replies
  • 7325 views

We have 2 fortigate 100EF in an HA cluster that needed to be upgraded (were on 7.0.2) and my boss asked if I wanted to give it a shot. He said it was straightforward and that it does the failover itself and all that. Upgrade path was to be .2 -> .5 -> .7 Well I think I got ahead of myself and somehow set the secondary (f2) to upgrade to .7 while the primary (f1) was rebooting to move to .5. This also briefly took down our sites since both FW were down at the same time (we're an e-commerce company).

So now f1 is on 7.0.5 and is the active primary and f2 is on 7.0.7 but f2 is out of sync and f1 still needs to come up to .7... I'm not a real sysadmin or network guy we don't have one. I tried a couple cli commands I saw online. Recalculate, and there was some force sync command too. Neither helped. I also tried the one to force HA failover to make f2 the primary but that didn't either (my boss thought this might help but I guess I wasn't surprised because the HA cluster is basically just 1 box right now I think).

Part of me thinks if I just do the upgrade to .7 on f1 that maybe they'll sync back up and all will be well. Would just have to eat a site outage again for a few minutes while it reboots? Not ideal I know. Another idea the team had was drop f2 out of the cluster but I think without sending someone to the sever farm to be ready to unplug stuff that we would run into network collisions? Any help or thoughts you guys have is appreciated.

3 replies

scan888
scan888
New Member
October 10, 2022

Hello,

 

First it would be a good idea to contact a Fortinet Partner with knowhow with that products for the next time, especially if it's for your main business . There are a reason, why network specialist are existing. ;)

First important thing to know is, that the cluster are in sync only when both Firewalls has the same firmware version. If you have a firmware mismatch the cluster come never in sync.

 

In your case, and with the minimal information about your infrastructur, I suggest the following steps:

- Create a backup from the firewall with the correct configuration (always, create a backup before you upgrade)

- Shutdown the firewall with the Version 7.0.7

- Upgrade the firewall with the Version 7.0.5 to 7.0.7 -> Wait until upgrade is success

- Start the firewall from the first step again

- Check if both firewalls see each other (System-HA in the Menu)

- Wait 1-2 hours for sync between both members

 

There is no way without downtime, in my point of view.

 

If the cluster still "out-of-sync" after a couple of hours, you can start again with your troubleshooting.

AlexC-FTNT
Staff
AlexC-FTNT
Staff
October 10, 2022

You'll learn a lot from this one! :)
If one is in .5 the other in .7, then either disconnect .5 (and upgrade alone), or simply try to start the upgrade again.

Debbie_FTNT
Staff & Editor
Debbie_FTNT
Staff & Editor
October 11, 2022

Hey jefry,

 

as your FortiGates are physical appliances, you can take advantage of a fairly easy roll-back feature:
- (hardware) FortiGates usually have two disk partitions, the active one with current firmware, and a secondary with the previous firmware.

- on your FortiGate in 7.0.7, you can roll back to 7.0.5

- you can verify that they are in sync, and then upgrade

 

You can also, as scan and Alex suggested, upgrade your 7.0.5 unit to 7.0.7 and then sync.

To rollback, you can refer to this KB: https://community.fortinet.com/t5/FortiGate/Technical-Tip-Selecting-an-alternate-firmware-for-the-next/ta-p/191425

Terms & ConditionsAccessibility statement