In the default configuration of 'config system ha' of an FGCP FortiGate cluster, these are the default values of heartbeat parameters, mostly using the on-premises aggressive hardware defaults:
config system ha
set hb-interval 2
set hb-interval-in-milliseconds 100ms
set hb-lost-threshold 20
set hello-holddown 20
…
Refer to this article: Technical Tip: Changing the HA heartbeat timers to prevent false failover for details on each parameter and allowed ranges.
This set of settings may cause instability, false-positive failovers, and can trigger warnings or error messages in 'diagnose sys ha history read' messages, for instance, when there is a peak of traffic/CPU processing or, out of FortiGate's responsibility, when Azure performs underlying hypervisor maintenance.
A workaround or an adaptation is to lower the sensitivity of the HA heartbeat detection to reduce the false alarms:
set hb-interval 2
set hb-lost-threshold 20
With a 100ms multiplier = 200 milliseconds.
The result: 200ms (hb-interval) X 20 = 4 000ms (4 seconds).
In this configuration, the cluster will declare the primary unit dead if it misses heartbeats for just 4 seconds. But in Azure, a micro-freeze on the underlying compute host or storage array during an Azure infrastructure update can easily pause a VM for 5 to 15 seconds.
With a 4-second aggressive window, the FortiGate may switch back and forth unnecessarily.
If repeated failovers occur in the Azure environments of the company, it is then recommended to first increase these timers/settings to make it less sensitive.
Moreover, the following parameter specifies the number of seconds that a cluster unit waits before changing from 'hello' state to 'work' state:
set hello-holddown 20 #(20 seconds)
In Azure, when a FortiGate in the cluster boots up or recovers, 20 seconds is often not enough time for the Azure Virtual Network interfaces to fully initialize and bind routing. The node might join the cluster before it is fully in synch with the other units and ready to process traffic properly.
Finally, the administrator can slowly increase the values of these parameters during the test to reach these recommended values:
config system ha
set hb-interval 10 # Increase to 1 second per packet (10 * 100ms)
set hb-lost-threshold 60 # Allow up to 60 missed packets
set hello-holddown 30 # Increase to 30 seconds to allow Azure unit to settle
end
This would give the primary firewall unit a 60-second buffer to survive temporary Azure backend maintenance blips without triggering unnecessary cluster emergency messages or failover.
|