When a FortiGate is running in HA FGCP mode, Active-Passive or Active-Active, HA Log messages may appear indicating there was a 'Heartbeat packet lost':
date=2023-05-30 time=18:23:19 eventtime=1685463799973776644 tz="+0200" logid="0108037910" type="event" subtype="ha" level="critical" vd="root" logdesc="Heartbeat packet lost" msg="Heartbeat packet lost" ha_role="primary" devintfname="port9"
 This log message means that the HA Peer did not receive the HA Heartbeat packet within the HA Hold-down timer.
For example, if the Heartbeat packets are not received within 1.2 seconds (1.2 seconds is the default value - a calculation is shown below). For more information, refer to this document:Â HA heartbeat interface.
hb-interval * hb-interval-in-milliseconds * hb-lost-threshold = 2 * 100 millisecond * 6 = 1.2 seconds (default values)
This article provides the troubleshooting tips required to narrow down the cause of this issue.
Troubleshooting steps:
Run 'get system ha status' to see if there are dropped packets or errors on the heartbeat interface:
get system ha status | grep "ha1"
ha1: physical/10000auto, up, rx-bytes/packets/dropped/errors=8013531615/5586193/486/0, tx=256040866/180012/0/0
ha1: physical/10000auto, up, rx-bytes/packets/dropped/errors=8005393949/5575798/549/0, tx=261722674/182376/0/0
 Run the command below to see if there are dropped packets in the physical interface counters:
diagnose hardware device info nic <heartbeat port number>
diagnose hardware deviceinfo nic ha1 | grep drop
rx_dropped 4333
tx_dropped 0
port.rx_dropped 0
port.tx_dropped_link_down 0
 Check for high CPU usage due to user-level processes. The HA Heartbeat packet is generated by the 'hatalk' process. Identify this by running the hatalk debug.
diagnose debug application hatalk -1
diagnose debug application hasync -1
diagnose debug enable
To disable the debug after collecting the output:
diagnose debug disable
diagnose debug reset
<hatalk> cfg_changed is set to 0: hatalk_packet_setup_heartbeat
<hatalk> setup new heartbeat packet: hbdev='port22', packet_version=473
<hatalk> options buf is small: opt_type=41(DEVINFO), opt_sz=13806, buf_sz=8726
<hatalk> pack compressed dev_info: dev_nr=37, orig_sz=13800, z_len=244
The output of 'diagnose sys top' can be used to verify whether there is high CPU usage with the 'hatalk' process.
Having high CPU usage in the hasync process should not directly affect the heartbeat packet processing. However, if it is high and if both processes are running on the same CPU core, it could have an indirect effect.
Similarly, if there is constant high CPU usage due to other processes, that needs to be addressed as well.
 Check for high CPU usage due to a system or kernel process such as irq or softirq. If there is high CPU usage on a specific CPU core or a group of CPU cores, it will have a direct effect on the HA heartbeat packet processing. This will make it necessary to identify and fix such high CPU issues.
Verify these issues with the following commands:
get system performance status
CPU states: 0% user 99% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU0 states: 0% user 99% system 0% nice 1% idle 0% iowait 0% irq 0% softirq
CPU1 states: 0% user 98% system 0% nice 2% idle 0% iowait 0% irq 0% softirq
 diagnose sys mpstat 1
diagnose sys profile ?
start  start kernel profiling data
stop   copy kernel profiling data
show   show kernel profiling result
sysmap  show kernel sysmap
cpumask profile which CPUs [Take 0-10 arg(s)]
step   set profile step
module  show kernel moduleÂ
fnsysctl cat /proc/interrupts
fnsysctl cat /proc/softirqs
From the interrupts counter, find the CPU core number that receives the heartbeat packet and check if there is a short spike in CPU for those Cores.
For example, the output of the command 'fnsysctl cat /proc/interrupts' shows that the TxRx Queues of ha1 heartbeat interface [i40e-ha1-TxRx-31, i40e-ha1-TxRx-33] are mapped to CPU core CPU75 and CPU77, respectively.
 Â Similarly, it is necessary to find the core numbers for other host queues of the heartbeat interface and verify if there are CPU spikes on these cores. Take action as necessary if these are present. Â Collect heartbeat packet captures during the 'heartbeat packet loss' issue from both the primary and secondary units, then use them to verify whether the heartbeat packets sent from the primary are received on the secondary and vice versa.
Packet capture commands:
HA primary:
diagnose sniffer packet any 'ether proto 0x8890' 4 0 l | grep ha1
2023-06-05 16:52:15.630003 ha1 out Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.698791 ha1 in Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.740012 ha1 out Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.798792 ha1 in Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.840003 ha1 out Ether type 0x8890 printer hasn't been added to sniffer.
HA secondary:
diagnose sniffer packet any 'ether proto 0x8890' 4 0 l | grep ha1
23-06-05 16:52:15.822283 ha1 out Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.863515 ha1 in Ether type 0x8890 printer hasn't been added to sniffer.
2023-06-05 16:52:15.932283 ha1 out Ether type 0x8890 printer hasn't been added to sniffer.
 If there is a difference in the number of packets sent and received in the previous step, it is important to double-check the connectivity between the FortiGate HA Members. If the connection goes through a switch and VLANs are used, use dedicated VLANs for heartbeat traffic so that it does not interfere with other data traffic. Error counters may increase at the interface level if the physical connection between HA interfaces is suboptimal. If error counters are rising rapidly, it is advisable to replace the HA cable and re-check the counters.
The following command can be used to find the counters for errors, so run it multiple times:
fnsysctl ifconfig <ha interface name>Â
Note:
fnsysctl commands require super admin permissions.Â
diagnose hardware device info nic <ha interface name>
The command also displays interface statistics. In multi-VDOM environments, run this command from the global VDOM.
In units that have a dedicated HA interface (for example, HA1 and HA2 ports), it is recommended to use these ports because they are directly connected to the CPU of the FortiGate and not connected through an NPU or ISF.
If a network port is used for exchanging heartbeat packets (i.e. port1-port32), it should be chosen carefully by looking at the internal hardware schematics of that particular FortiGate model. For example, the internal schematics of FortiGate 3600E differ from those of FortiGate 3700D.
If the chosen heartbeat port shares the same internal path as a heavily used network interface, it could lead to sub-optimal packet processing.
Additionally, it is possible to increase the heartbeat timers to increase the fault tolerance.
Additionally, it is possible to improve the HA heartbeat traffic and performance of the HA cluster by selecting one or more ports to use for a synchronizing session, as explained in Technical Tip: HA session-sync-dev configuration.
By default, session synchronization activity takes place over the HA heartbeat link using TCP/703 and UDP/703.
If there is a large number of session synchronizations, this can cause network congestion and impact the HA cluster communication.
Collect the HA Event logs, System Event logs, and Traffic logs during the time of the issue and share them with TAC.
Collect HA history logs, HA status, link monitor, and other details of the interface. Follow the steps outlined in Technical Tip: Troubleshooting unexpected High Availability (HA) failover.
 In some network environments, increasing the values of 'hb-interval' and 'hb-lost-threshold' to a higher value may resolve the problem.  Example:
config system ha
  set hb-interval 10 (default =2)
  set hb-lost-threshold 40 (default=20)
end
Adding a backup heartbeat interface to an HA cluster is highly recommended. In the event of heartbeat packet loss, the backup interface helps maintain network stability by preventing an HA split-brain state. The backup heartbeat is limited in terms of synchronization features, as its goal is primarily to maintain member placement and avoid both cluster members declaring themselves as primary during disruption of regular heartbeat traffic.
config system ha
  set backup-hbdev <interface list>
end
The backup-hbdev is used only when the secondary unit fails to detect heartbeats from the primary unit. This functionality is available starting from FortiOS 7.6.0. Refer to the document below for more information on implementation and limitations.
Related documents:
| 
