Lost connection to Slave after Update

[RolandBaumgaertner72]

Hello,

this morning we decided to update a FG80F cluster from 7.4.4 to 7.4.5.

Stupid from my side, I did not check if the HA was synced and I did not notice anything wrong. I uploaded 7.4.5 and I lost access to the FG so that the Slave did not enter in the game.

After restart of the master we still had no sign of the slave. Since this is a remote side I asked a person to remove all cables from the slave unit and try to connect via port 1 LAN but there is no ping. Than we restarted the slave thinking that it might start as master but still no access. This unit for sure did not update to 7.4.5 and being with another OS it will not work in the HA anymore.

Any ideas to solve this? At the end there is just the option to access via console? Only change to reset and start HA over?

Thanks!

[vbandha]

Hello @RolandBaumgaertner72 

If you are having issue with HA out sync, please run these commands on both primary and secondary:

diagnose sys ha checksum recalculate
execute ha synchronize start

You can follow this document to get cli access of secondary from primary:

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-access-secondary-unit-of-HA-cluster...

Regards,

Varun

[RolandBaumgaertner72]

Hi,

now I am waiting for like 10mins and there is still no sync. The checksums are not the same:

================== FGT80FTK2101XXXX ==================

is_manage_primary()=1, is_root_primary()=1
debugzone
global: 9a b2 7d 9b 7e f5 a2 07 0b cb 7b 66 52 72 f2 43
root: 4d 88 11 6e e1 77 54 1b 2d b0 d0 3d cd 8d 55 92
all: ae b2 b9 01 a4 41 e1 f7 2d a3 71 c4 25 0e 04 57

checksum
global: 9a b2 7d 9b 7e f5 a2 07 0b cb 7b 66 52 72 f2 43
root: 4d 88 11 6e e1 77 54 1b 2d b0 d0 3d cd 8d 55 92
all: ae b2 b9 01 a4 41 e1 f7 2d a3 71 c4 25 0e 04 57

================== FGT80FTK2101XXXX ==================

is_manage_primary()=0, is_root_primary()=0
debugzone
global: ba 22 ad a4 95 67 26 bc fa f8 c8 a3 99 20 b9 b3
root: 12 fb e9 97 35 41 e4 a8 e7 1a 83 df 93 f2 5e 5b
all: 36 a3 a8 e6 9d ed 6b 49 fa e3 6f 03 05 7e 1d b6

checksum
global: ba 22 ad a4 95 67 26 bc fa f8 c8 a3 99 20 b9 b3
root: 12 fb e9 97 35 41 e4 a8 e7 1a 83 df 93 f2 5e 5b
all: 36 a3 a8 e6 9d ed 6b 49 fa e3 6f 03 05 7e 1d b6

MasterXXXX #

What can I do, wait longer? One Table System Custom Language is not sync

Wait or what can I do?

[AEK]

You can run this debug commands.

diag debug app hasync 255
diag debug app hatalk -1
diag debug app hasync -1
diag debug enable
execute ha synchronize start

Troubleshooting guide:
https://community.fortinet.com/t5/FortiGate/Troubleshooting-Tip-How-to-troubleshoot-HA-synchronizati...

[RolandBaumgaertner72]

Hi,

no sync so far. Thats bad, because we jsut can connect via remote, the cluster is far :(

I checked the config files, the master unit:

config system custom-language
    edit "en"
    next
    edit "fr"
    next
    edit "sp"
    next
    edit "pg"
    next
    edit "x-sjis"
    next
    edit "big5"
    next
    edit "GB2312"
    next
    edit "euc-kr"
    next
end

Slave unit:

config system custom-language
    edit "en"
        set filename "en"
        set comments ''
    next
    edit "fr"
        set filename "fr"
        set comments ''
    next
    edit "sp"
        set filename "sp"
        set comments ''
    next
    edit "pg"
        set filename "pg"
        set comments ''
    next
    edit "x-sjis"
        set filename "x-sjis"
        set comments ''
    next
    edit "big5"
        set filename "big5"
        set comments ''
    next
    edit "GB2312"
        set filename "GB2312"
        set comments ''
    next
    edit "euc-kr"
        set filename "euc-kr"
        set comments ''
    next
end

So now I am not sure if the failover works and I can remove master unit to change....

Any workaround which would be less risky?

Thanks

[AEK]

Hello

Start by opening a ticket.

We can also help if share the debug output I mentioned above, and the HA config from both nodes.

[arahman]

Hi, i would suggest to try rebooting both the foregates, secondary first and then the primary. and also try the manual synch by synching the checksum of the FortiGate 

diagnose sys ha checksum recalculate

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Procedure-for-HA-manual-synchronization/ta...

[Shashwati]

Hello, 

Please refer to the document for manual HA synchronization 

https://community.fortinet.com/t5/FortiGate/Technical-Tip-Procedure-for-HA-manual-synchronization/ta...

[pmudgal]

Hello @RolandBaumgaertner72 ,

Thank you for reaching out to fortinet support.

In order to bring your device up and back in HA Sync, first take the node2 out of HA cluster along with LAN and wan access, then management access of the Node2

Upload the correct firmware on the Node2 and then connect back the Node2 device in HA cluster.

If still you are facing any issue i would suggest to open a TAC case.

Regards,

Piyush

[RolandBaumgaertner72]

Hello Piyush,

what do you mean upload firmware, both FG80F are on 7.4.5. We had on the slave this known bug updating from 7.4.3 to 7.4.4 some months ago, at the end node 2 never got updated and we didnt see any failures in HA. Than after updating 7.4.5 when we dondt expect any failures not knowing that node 2 was not even on 7.4.4 we lost connection to node 2. I reseted node2 and updated to 7.4.5, than it was shown in the HA again but with this one table missing.

Why is the config different when both nodes are on 7.4.5? 

On Node 1:

config system custom-language
    edit "en"
    next
    edit "fr"
    next

On Node 2:

config system custom-language
    edit "en"
        set filename "en"
        set comments ''
    next
    edit "fr"
        set filename "fr"
        set comments ''

Makes no sende since Node2 was reseted and got the config from Node1.

Now I am not even sure that failover is working and since this cluster is far away I am not risking booting the Node1.

Any other workaround?

Thanks  

[AEK]

Hi Roland

If this is the only difference between the two node configs (config system custom-language), can you just make them similar? i.e.: copy this portion from one node to the other.

On the other hand, on menu System > HA, when you hover the secondary node you should see which config sections are not synced. Can you share the screenshot?