| Description |
This article describes how to configure and troubleshoot a FortiManager High-availability (HA) cluster in Manual and VRRP mode. |
| Scope | FortiManager, HA. |
| Solution |
Notes on HA modes for FortiManager: Must be the same between the Primary and all other nodes of the cluster:
Note: Starting from 7.4.x and forward, can deploy through different types of machines. For example, the Primary device uses KVM, and the secondary uses VMware ESXi. The steps to configure HA are unchanged.
Note: Virtual IP should be the same in both Primary and Secondary devices. (VRRP mode).
Sample Diagram with Port & IP Configuration:
FortiManager HA settings: Below are the HA settings of the FortiManager HA cluster and its meanings:
Failover Mode: <Manual> or <VRRP> (VRRP or automatic HA failover mode will be covered later in this document).
Operation Mode: <Primary> or <Secondary>.
Cluster Settings: Peer IP: <Secondary FortiManager IP address>. Peer SN: <FMGVMXXXXXX> (Secondary FortiManager Serial Number).
The below HA settings must be the same on the Primary & Secondary nodes:
Cluster ID: Any number (1-255) can be given. Group Password: <password> Can give any password. File quota: 4096. Heartbeat Interval: <Interval_Integer>.
The time the primary unit waits between sending heartbeat packets is in seconds. The heartbeat interval is also the amount of time that the backup unit waits before expecting to receive a heartbeat packet from the primary unit. The default heartbeat interval is 5 seconds. The heartbeat interval range is 1 to 255 seconds. Users cannot configure the heartbeat interval on the backup units.
Failover Threshold: <Interval_Integer>. The number of heartbeat intervals that one of the cluster units waits to receive HA heartbeat packets from other cluster units before assuming that the other cluster units have failed.
The default failover threshold is 3. The failover threshold range is 1 to 255. Users cannot configure the failover threshold of the backup units.
These below settings can only be configured when the Failover Mode is VRRP:
VIP: <Virtual IP address>. This setting can only be configured when the Failover Mode is VRRP. (Make sure this IP is not used in the network).
VRRP Interface: <port>. Priority: <1-253>. Set the priority for this device between 1 (lowest) and 253 (highest). The device with a higher priority will operate as the primary unit when possible.
Unicast. Optionally, toggle this setting ON to use Unicast for the VRRP message.
Monitored IP. Configure the monitored IP and interface. Users can add additional monitored IPs by selecting the add icon. (not mandatory)
Configuration FortiManager HA cluster (Manual mode). On FortiManager-Primary device:
System Settings -> HA -> Operation mode select 'Manual’ -> ‘Primary’.
Configure the following details:
Failover Mode: Manual. Operation Mode: Primary. Peer IP and Peer SN: Peer IP: x.x.x.17. Peer SN: FMGVMTMxxxxxx8.
The below HA settings must be the same on the Primary and Secondary nodes:
Cluster-ID: 21. Group Password: Fortinet@Test. File quota: 4096. Heartbeat Interval: 10. Failover Threshold: 30.
On FortiManager-Secondary device: System Settings -> HA -> Operation mode select 'Manual’ -> ‘Secondary’.
Configure the following details:
Failover Mode: Manual. Operation Mode: Secondary. Peer IP and Peer SN: Peer IP: x.x.x.19. Peer SN: FMGVMTMxxxxxx4.
The below HA settings must be the same on the Primary & Secondary nodes:
Cluster ID: 21. Group Password: Fortinet@Test. File quota: 4096. Heartbeat Interval: 10. Failover Threshold: 30.
After configuring the Primary & Secondary nodes of the FortiManager HA cluster green arrows should appear on GUI (Synchronization status).
On FortiManager Primary node:
On FortiManager Secondary node:
FortiManager-HA automatic failover – VRRP mode.
VRRP (Virtual Router Redundancy Protocol) is a protocol intended to increase the availability of the default gateway for hosts on the same network. The goal is to define the default gateway for network hosts as a virtual IP address referencing a group of routers.
Not only a unique IP address will be declared on each machine (Primary and Secondary IP addresses), but also a virtual IP address shared between each FortiManager (VIP) node. This address aims to ensure the VRRP availability.
Source address: The primary IP address of the interface the packet is being sent from. Destination IP address: 224.0.0.18 (Multicast IP address).
This is a link-local scope multicast address. Routers should not forward a datagram with this destination address regardless of its ttl. TTL: 255. Protocol: The IP protocol number assigned by the IANA for VRRP is 112 (decimal). MAC Address in the following format: 00-00-5E-00-01-{VRID}.
Into the below packet capture, the multicast IP address and its mac address in the following format can be viewed: 00-00-5E-00-01 {VRID}: 01:00:5e:00:12.
How does VRRP work: The FortiManager that gets the highest priority is elected as the Primary.
The end user only knows the VIP.
When a node of the FortiManager cluster becomes down, a gratuitous ARP (preload the ARP tables of all other local hosts) request is sent by the FortiManager backup to get the Virtual IP address. The High-Availability principle then is respected.
At that moment, the active FortiManager node gets the Primary role. However, when the FortiManager node becomes again available it takes the Secondary role of the cluster even though the ID cluster is higher because the VRRP protocol considers the older age value.
Rebooting a FortiManager unit updates the HA roles (Primary/Secondary).
To use automatic failover for FortiManager-HA:
On Primary FortiManager:
On Secondary FortiManager:
To configure automatic failover in the FortiManager CLI:
config system ha set failover-mode vrrp set mode primary config monitored-ips edit 1 set interface <string> set ip <string> next end config peer edit <peer_id_int> set ip <peer_ipv4_address> set serial-number <string> next end set priority <integer> set vip <string> set vrrp-interface <string> end
For example:
config system ha set failover-mode vrrp set mode primary config monitored-ips edit 1 set interface "port2" set ip "192.168.48.63" next end config peer edit 1 set ip 10.3.106.64 set serial-number "FMG-VM0A1700xxxx" next end set priority 200 set vip "10.3.106.253" set vrrp-interface "port1" end
config system ha set failover-mode vrrp set mode secondary config monitored-ips edit <id> set interface <string> set ip <string> next end config peer edit <peer_id_int> set ip <peer_ipv4_address> set serial-number <string> next end set priority <Integer_value> set vip <string> set vrrp-interface <string> end
For example:
config system ha set failover-mode vrrp set mode secondary config monitored-ips edit 1 set interface "port2" set ip "192.168.48.64" next end config peer edit 1 set ip 10.3.106.63 set serial-number "FMG-VM0A1600xxxx" next end set priority 1 set vip "10.3.106.253" set vrrp-interface "port1" end
Troubleshooting commands:
On Primary-FortiManager:
With these 2 commands, troubleshoot the status and the configured values for FortiManager-HA:
diag ha stats
get system ha
On Secondary-FortiManager:
With these 2 commands, troubleshoot the status and the configured values for FortiManager-HA.
diag ha stats
get system ha
The following CLI commands help to troubleshoot FortiManager HA cluster issues:
diagnose debug application ha 255 diagnose debug enable
The debug output above can also be downloaded in a txt file:
For example:
The debugging results during the HA cluster build operation can also be seen through CLI. The below capture shows the negotiation of the FortiManager HA cluster from the primary node. The same operation can also be performed on the secondary node.
VRRP/Manual synchronization debug results Below are examples of output results of the HA cluster synchronization in VRRP mode on FortiManager. These debug are similar for HA configured in ‘manual mode’. VRRP debugs on Primary node:
VRRP debugs on Secondary node:
Once the HA cluster is built and synchronized, keepalive messages are sent between both nodes on the port chosen for this purpose:
On FortiGate central management, two Serial Numbers are configured (Both SNs belong to each FortiManager node of the cluster). Both HA manual and VRRP modes use two Serial numbers.
If a node of the cluster becomes unavailable, the other SN will be automatically used.
Any Serial number will be removed from the configuration of the FortiGate central management in that case.
Related documents: FortiManager-HA automatic failover enhancement Technical Tip: Things to check before FortiManager HA failover Technical Tip: High level steps in performing FortiManager HA manual failover test Technical Tip: FortiManager Geo-redundant HA with VRRP Firmware Upgrade Behavior |
