Problems Updating Fortigate Cluster

[RolandBaumgaertner72]

Hello,

we never had a problem with an update of a FG Cluster but I think there is always a first time :(

This time the FG81E Cluster just took so long (on the web GUI validating Image for 30mins) that we restarted the Slave unit after the process. Than we connected via serial and we could see that the slave was updated to 7.0.12 and the Master had a time out for restarting the slave unit.

Now the problem is that the Master has 6.4.11 and the Slave 7.0.12 and they are out of sync. 

What is the easiest now to fix the HA Sync? Do we have to disconect the HA Slave, downgrade (or exec factory reset) to same OS 6.4.11 and start from zero the HA config?

Would we expect some donwtime when we connect again the slave unit?

What is the best way?

Thanks!

[RolandBaumgaertner72]

Actual Checksum:

FG_Mazarredo_Master $ diag sys ha checksum cluster

================== FGT81ETK18006771 ==================

is_manage_primary()=1, is_root_primary()=1
debugzone
global: 24 e9 27 f9 df f1 86 b8 ac d4 59 9e 5d 77 ec 14
root: 80 3e b0 05 8f 4e cb 34 4f 88 2f d3 bc 2d 1c 84
all: cb a4 16 ec 5f bd 93 c9 be 3a fb 3e a8 d5 7c 59

checksum
global: 24 e9 27 f9 df f1 86 b8 ac d4 59 9e 5d 77 ec 14
root: 80 3e b0 05 8f 4e cb 34 4f 88 2f d3 bc 2d 1c 84
all: cb a4 16 ec 5f bd 93 c9 be 3a fb 3e a8 d5 7c 59

================== FGT81ETK19000587 ==================

is_manage_primary()=0, is_root_primary()=0
debugzone
global: 52 5f 26 81 12 67 3b f9 02 2f 51 56 fb c1 03 1c
root: 07 7f ae 50 68 15 c2 07 9e 0c 95 d1 f2 58 37 77
all: 3e d9 72 e3 b9 d5 4e ba 21 fc 02 54 a4 e2 8e a3

checksum
global: 52 5f 26 81 12 67 3b f9 02 2f 51 56 fb c1 03 1c
root: 07 7f ae 50 68 15 c2 07 9e 0c 95 d1 f2 58 37 77
all: 3e d9 72 e3 b9 d5 4e ba 21 fc 02 54 a4 e2 8e a3

FG_Mazarredo_Master $

[ede_pfau]

been there, done that...I've had the exact same problem when upgrading from v6.4.11 to v7.0.11.

Status:

one FGT is already on v7, one remains on v6.

Solution:

Fail the cluster over so that the unit with v6 becomes primary.

Then start the upgrade process.

Now, WHILE the image is uploading, at 40-50% done, REBOOT the secondary unit.

The primary will upgrade and re-sync into the cluster.

Totally intuitive, right? Got this from FTNT support. And I hope it'll fix your problem.

[pavankr5]

Hello @RolandBaumgaertner72 ,

Before proceeding with any changes, ensure you have a backup of your current configurations on both the Master and Slave units. This will allow you to revert to a known working state if needed.
Start the firmware upgrade process on the Slave unit (running 7.0.12) following the recommended upgrade procedure outlined standalone units.

After the Slave unit (running 7.0.12) has been successfully upgraded and verified, initiate a failover so that the Slave becomes the primary unit in the HA cluster.

Now that the former Master unit (which is now the Slave unit) is no longer serving as the active unit in the HA cluster, proceed with upgrading its firmware.

After the former Master unit has been upgraded and is functioning correctly, perform a failback to restore the original Master unit as the active unit in the HA cluster. Monitor the HA synchronization process to ensure that both units are running the same firmware version and operating in sync.

Thanks,

Pavan

[RolandBaumgaertner72]

Hi,

first thanks!

I got like 2 completely different ideas. 

Again, the situation is that the slave YES managed to upgrade to 7.0.12 and the Master tried to reboot the slave and did nothing and has the 6.4.11 right now. They are out of sync with different checksums, so there is basically no cluster anymore.

So one option is to remove the SLAVE from the cluster to have the Master as Stand Alone and upgrade the Master to 7.0.12 (to have both running on the same OS). Connect the SLAVE and hope that it will Sync, right? That means we would have some down time for the Master reboot.

The other option (Pavan) I dont really see. We have configs from the Master and we could get one from the Slave. The SLAVE already has the 7.0.12 so you mean like downgrading before to the same OS as the Master? (downgrade to 6.4.11). 

We will solve this on monday. Since we would like to do again another FG Cluster on monday I would like to do the best and maybe fastets option....and I will NEVER again do that without serial connection but I am kind of annoyed since we updated like 50 FGs theses days and none of them needed so much time. Also this was not the most important site and I thought with the FG81E it would be fast and easy.

Thanks!