Skip to main content
Virgule59
Virgule59
New Member
February 24, 2020
Question

Upgrade Firmware on HA cluster issue on fail over

  • February 24, 2020
  • 2 replies
  • 11068 views

Hi Guys,

 

I've upgrade my cluster of 2 Fortigates 100E from 5.4.2 to 6.0.9 and follow upgrade path to do this.

 

Cluster Ha are configure like this

 

4 VDOM + Root Vdom

2 vclusters

 

Root Vdom and 2 Vdom are on vcluster 0

2 others vdom are on vcluster1

 

Primary Master is master on vcluster0 and slave on vcluster1

Secondary Slave is master on vlcuster1 and slave on vcluster0

 

Pimary master and secondary slave are on 2 differents datacenters distants from less than 62 Miles and ping less or equal to 1 ms.

HA1 and HA2 are on differents vlan

 

As i'm in maintenance Windows, before i'll launch first upgrade i'll reboot the cluster, wait for cluster checksum are Ok, and let's go for upgrade.

 

If i understand the KB on upgrade the cluster

[ol]
  • Connect on master, backup config, launch update update automatically do this operation : [/ol][ul]
  • load and upgrade firmware on secondary slave 
  • reboot slave
  • slave become master
  • load and upgrade  firmware primary master
  • reboot master
  • Primary master become master, slave go back to secondary slave (depend on priority, ages...)[/ul]

     

    The first upgrade and other take place like that,

     

    1) load and upgrade slave

    [ul]
  • Lost secondary site, no ping on management interface, no ping on device[/ul]

    2) slave become master

    [ul]
  • ping secondary site ok, ping primary ok[/ul]

    3) master upgrade

    [ul]
  • lost ping on master and slave no site where available[/ul]

    4) Master become master

    [ul]
  • Site 1 are available and site 2 are not available
  • site 2 become available after 4-5 minutes[/ul]

    5) let's take a look a checksum before upgrade a new firmware,

     

    get sys ha status

     

    not the same, try a calculate

     

    diagnose sys ha checksum recalculate

     

    Ok cluster are in sync.

     

    [size="2"]Humm, cluster have to warm up , let's go for the round two....[/size]

     

    Same issue... round 3 same issue, round 4 same issue ...[size="2"]After 3 hours of interruptible upgrade  i'm on 6.0.9. OK guy let's try a ha failover, connect on slave cli exec reboot, lost one ping... VPN are UP, device and equipement ping are OK. all seems good on two site [/size][size="2"]connect on master cli exec reboot lost one ping... VPN are UP, device and equipement ping are OK. all seems good on two site.[/size]

     

    ha uninterruptible-upgrade are enable (default).

     

    What i forgot to do before the upgrade ?

     

    best regards

     

      • 2 replies

      lobstercreed
      lobstercreed
      New Member
      February 24, 2020

      I'd be curious to know if you find out.  This sounds like normal behavior to me, but maybe I misunderstood something.  I assume these are active-active clusters?

      Virgule59
      Virgule59Author
      New Member
      February 25, 2020

      Hi,

       

      Do you think that failover doesn't work on firmware upgrade ? This is not what i'v found on this forum and not what's fortinet KB say : Upgrade firmware.

       

      Cluster HA are in Active-Passive mode BUT:.

       

      Vcluster1 are master on the ACTIVE and Vcluster2 are master on the PASSIVE. In this type of configuration slave are not really in passive mode because it's master on the vcluster2 and slave on vcluster1.

       

      And has I said if i reboot a member of the cluster the failover mechanism work as expected.

       

      For me this sound strange

       

      As noted in the KB i understand now why i've lost the passive firewall, because of the override. but it's doesn' tell why i've lost the cluster during upgrade.

       

       

      Regards

      lobstercreed
      lobstercreed
      New Member
      February 25, 2020

      I think I must have misunderstood.  I thought you were saying that you had only 1 ping loss during the failover (which is basically hitless) and when you said one or the other was down I thought you meant during the upgrade process (rebooting the hardware, switching HA).  If you're getting a different result during upgrade than during a "normal" reboot other than maybe longer boot time, I don't know why.

      KDalbjerg
      KDalbjerg
      New Member
      December 17, 2021

      Did you ever found out why this happen?

      Terms & ConditionsAccessibility statement