Description
This article describes the debugs that should be collected when troubleshooting HA issues.
Scope
FortiGate.
Solution
- The following debugs should be collected for any HA-related issues:
diagnose debug enable
diagnose debug console timestamp enable
diagnose debug application hatalk -1 <----- HA formation issues
diagnose debug application hasync -1 <----- HA Sync issues.
diagnose debug disable <----- Command to disable the debug.
If no output is generated in the hasync or hatalk debugs, restarting the daemons may be necessary. This can be done by running the commands below on each unit.
To determine the process IDs running for hasync and hatalk:
diagnose system process pidof hasync
diagnose system process pidof hatalk
To restart the process:
diagnose system kill 11 <process_id>
Alternatively, to kill all simultaneously running HA processes, execute the following with super admin rights:
fnsysctl killall hatalk
fnsysctl killall hasync
- Run the following on both Primary/Secondary units and collect the info:
get system performance status
get system status
get system ha status
diagnose system ha status
diagnose system ha history read
diagnose debug crashlog read
diagnose system ha checksum show
diagnose system ha dump 5
diagnose system ha dump-by group
execute ha synchronize start <----- Start on backup unit first.
diagnose system ha checksum recalculate
To access the secondary device in the CLI, run the following:
execute ha manage <Index-ID> <Admin-Username>
See this article: Technical Tip: How to access the secondary unit from the primary with the 'execute ha manage' comman....
- Packet captures for seeing communication between HA ports:
diagnose hardware device nic <heartbeat interface>
diagnose sniffer packet port_ha "" 4 0 l <----- port_ha should be the heartbeat interface.
To capture only the heartbeat packets:
diagnose sniffer packet any 'ether proto 0x8890' 4 0 l
There are three EtherType configurations in HA
- ha-eth-type : 8890 - Standard NAT/Route mode heartbeat.
- hc-eth-type : 8892 - Transparent mode heartbeat or session sync.
- l2ep-eth-type : 8893 - HA configuration synchronization (Layer 2 Endpoint).
Configuration parameters can be verified with the following commands
get sys ha
show full sys ha | grep -f eth
- Collect the FortiGate's HA and System Event logs for both units downloaded from the GUI/FortiAnalyzer or syslog (remote) server.
Related articles:
Technical Tip: Procedure for HA manual synchronization
Technical Tip: Rebuilding an HA cluster
