Description
This article describes best practices for Heartbeat interfaces in FGCP high availability.
The FGCP requires Heartbeat links to maintain communication and synchronize sessions, configurations, IPsec SAs, kernel routes, etc.
Scope
FortiGate.
Solution
Fortinet suggests the following practices related to heartbeat interfaces:
Configure at least two heartbeat interfaces and set these interfaces to have different priorities. For example:
config system ha
set hbdev "ha" 150 "mgmt" 100
end
- For clusters of two FortiGate units, as much as possible, heartbeat interfaces have to be directly connected using patch cables (without involving other network equipment such as switches).
If switches have to be used, do not use them for other network traffic that can flood the switches and cause heartbeat delays.
- If it is not possible to use a dedicated switch, the use of a dedicated VLAN can help limit the broadcast domain to protect the heartbeat traffic and the bandwidth it creates.
- For clusters of three or four FortiGates, use switches to connect heartbeat interfaces. The corresponding heartbeat interface of each FortiGate in the cluster has to be connected to the same switch. To improve redundancy, use a different switch for each heartbeat interface. In that way, if the switch connecting one of the heartbeat interfaces fails or is unplugged, heartbeat traffic can continue on the other heartbeat interfaces and switch.
- Isolate heartbeat interfaces from user networks. Heartbeat packets contain sensitive cluster configuration information and can consume a considerable amount of network bandwidth. If the cluster consists of two FortiGates, connect the heartbeat interfaces directly using a crossover cable or a regular Ethernet cable. For clusters with more than two units, connect heartbeat interfaces to a separate switch that is not connected to any network.
- Use a dedicated HA port to isolate HA traffic from data traffic, bypassing ISF/NP6/NP7 for better performance and stability.
- When using a dedicated VLAN for heartbeat (HB) packets in an HA cluster, avoid sharing the same VLAN with heartbeat traffic from other clusters, as this may cause packet drops on the logical ha_port. No drops will be observed on the physical ports; however, the drop counter will increment for traffic received on the ha_port, where the HA logic (HA group ID, cluster ID) is validated. Any unmatched traffic, such as packets originating from a different cluster, will be dropped by the receiving cluster, resulting in unnecessary CPU and memory utilization to process unwanted packets. This can be avoided by using a separate VLAN for each cluster.
diagnose netlink device list | grep '\(port_ha\|port3\):'
port_ha: 323775 1614 0 942 0 0 0 0 1062146 1776 0 0 0 0 0 0 <----- Drops incrementing on ha_port for received traffic.
port3: 15016948 35255 0 0 0 0 0 0 6275284 13684 0 0 0 0 0 <----- No drop counter on physical ports.
- If heartbeat traffic cannot be isolated from user networks, enable heartbeat message encryption and authentication to protect cluster information. Both are disabled by default.
config system ha
set encryption {enable | disable} <----- Enable/disable heartbeat message encryption.
set authentication {enable | disable} <----- Enable/disable heartbeat message authentication.
end
- Configure and connect redundant heartbeat interfaces so that if one heartbeat interface fails or becomes disconnected, HA heartbeat traffic can continue to be transmitted using the backup heartbeat interface.
If heartbeat communication fails, all cluster members will think they are the primary unit, resulting in multiple units on the network with the same IP addresses and MAC addresses (condition referred to as Split Brain), and communication will be disrupted until heartbeat communication can be reestablished.
What split-brain is, how it occurs, and how to prevent and analyze it: Technical Tip: High availability split brain.
- Do not monitor dedicated heartbeat interfaces; monitor those interfaces whose failure should trigger a unit failover.
- Where possible, at least one heartbeat interface should not be connected to an NP4 or NP6 processor to avoid NP4 or NP6-related problems from affecting heartbeat traffic.
- Where possible, the heartbeat interfaces should not be connected to an NP4 or NP6 processor that is also processing network traffic.
- Where possible, each heartbeat interface should be connected to a different NP4 or NP6 processor.
- Any FortiGate interface can be used as a heartbeat interface, including 10/100/1000Base-T, SFP, QSFP fiber, and copper, and so on. If two or more interfaces are set up as heartbeat interfaces, each interface can be a different type and speed.
- Heartbeat communications can only be enabled for physical interfaces. They cannot be enabled for VLAN sub-interfaces, IPsec VPN interfaces, redundant interfaces, or 802.3ad aggregate interfaces. These types of interfaces do not appear in the heartbeat interface list.
Note:
Do not use a FortiGate switch port for the HA heartbeat traffic. If no HA interface is available, convert a switch port to an individual interface.
Related documents:
FGCP high availability
HA heartbeat
Technical Tip: Changing the HA heartbeat timers to prevent false fail over
