FortiGate
FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
nithincs
Staff & Editor
Staff & Editor
Article Id 339687

 

Description This article describes the type of logs generated during the HA event.
Scope FortiGate.
Solution

To understand the type of logs generated during the HA event,  consider the below scenario:

 

HA setup.JPG

 


Consider that the FortiGate is in a HA cluster and already in sync.

 

  • Primary : Fortinet1 , FGVMxxxxxTCTFB4, HA cluster index = 1.
  • Secondary : Fortinet2 , FGVMxxxxxEOUS79, HA cluster index = 0.

 

Fortinet1 HA setting:

 

config system ha
    set hbdev "port2" 256
    set monitor "port1" "port3" "port4"
end

 

Fortinet2 HA setting:

 

config system ha
    set hbdev "port2" 128
    set monitor "port1" "port3" "port4"
end

 

With the above setting, Fortinet1 has higher uptime with override disabled, it was selected as HA Primary.

 

Fortinet1 # get sys ha status
HA Health Status: OK
Model: FortiGate-VM64-KVM
Mode: HA A-P
Group: 10
Debug: 0
Cluster Uptime: 0 days 2:29:49
Cluster state change time: 2024-09-06 12:27:27
Primary selected using:
<2024/09/06 12:27:27> vcluster-1: FGVMxxxxxTCTFB4 is selected as the primary because its uptime is larger than peer member FGVMxxxxxEOUS79.
ses_pickup: disable
override: disable
Configuration Status:
FGVMxxxxxTCTFB4(updated 1 seconds ago): in-sync
FGVMxxxxxEOUS79(updated 2 seconds ago): in-sync
System Usage stats:
FGVMxxxxxTCTFB4(updated 1 seconds ago):
sessions=23, average-cpu-user/nice/system/idle=0%/0%/0%/98%, memory=42%
FGVMxxxxxEOUS79(updated 2 seconds ago):
sessions=0, average-cpu-user/nice/system/idle=0%/0%/0%/98%, memory=41%
HBDEV stats:
FGVMxxxxxTCTFB4(updated 1 seconds ago):
port2: physical/10000full, up, rx-bytes/packets/dropped/errors=23349420/53131/0/0, tx=22469350/62655/0/0
FGVMxxxxxEOUS79(updated 2 seconds ago):
port2: physical/10000full, up, rx-bytes/packets/dropped/errors=3925661/8387/0/0, tx=2755973/7991/0/0
MONDEV stats:
FGVMxxxxxTCTFB4(updated 1 seconds ago):
port1: physical/10000full, up, rx-bytes/packets/dropped/errors=5533100/15858/0/0, tx=4865692/8593/0/0
port3: physical/10000full, up, rx-bytes/packets/dropped/errors=369299/5243/0/0, tx=0/0/0/0
port4: physical/10000full, up, rx-bytes/packets/dropped/errors=369299/5243/0/0, tx=0/0/0/0
FGVMxxxxxEOUS79(updated 2 seconds ago):
port1: physical/10000full, up, rx-bytes/packets/dropped/errors=49650/705/0/0, tx=0/0/0/0
port3: physical/10000full, up, rx-bytes/packets/dropped/errors=47665/680/0/0, tx=0/0/0/0
port4: physical/10000full, up, rx-bytes/packets/dropped/errors=47665/680/0/0, tx=0/0/0/0
PINGSVR stats:
FGVMxxxxxTCTFB4(updated 1 seconds ago):
FGVMxxxxxEOUS79(updated 2 seconds ago):
Primary : Fortinet1 , FGVMxxxxxTCTFB4, HA cluster index = 1
Secondary : Fortinet2 , FGVMxxxxxEOUS79, HA cluster index = 0
number of vcluster: 1
vcluster 1: work 169.254.0.2
Primary: FGVMxxxxxTCTFB4, HA operating index = 0
Secondary: FGVMxxxxxEOUS79, HA operating index = 1

 

In the event of Fortinet1 gets restarted/monitored interface goes down/pingserver-monitor-interface fails, HA event events in the FortiGate will be visible.

 

==========================================

 

If Fortinet1 (primary) gets restarted, Fortinet2 will take over as primary. Below information will be visible in the output of 'get sys ha status' in Fortinet2 which got selected as primary:

 

-------------------------------------------------------

Primary selected using:
<2024/09/06 21:28:50> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because its uptime is larger than peer member FGVMxxxxxTCTFB4.
<2024/09/06 21:27:58> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because it's the only member in the cluster.
<2024/09/06 21:06:08> vcluster-1: FGVMxxxxxTCTFB4 is selected as the primary because its uptime is larger than peer member FGVMxxxxxEOUS79.

-------------------------------------------------------

 

At <2024/09/06 21:06:08> FGVMxxxxxTCTFB4 (Fortinet1) was selected as master due to higher uptime then FGVMxxxxxEOUS79 (Fortinet2).

At <2024/09/06 21:27:58> FGVMxxxxxEOUS79 (Fortinet2) lost HA heartbeat communication with FGVMxxxxxTCTFB4 (Fortinet1) and hence consider itself as Primary.

At <2024/09/06 21:28:50> FGVMxxxxxEOUS79 (Fortinet2) HA heartbeat communication re-established with FGVMxxxxxTCTFB4 and since its uptime is higher, its got selected as Primary.

 

 FGVMxxxxxTCTFB4 uptime is lesser then FGVMxxxxxEOUS79 (Fortinet2) after HA cluster formation confirm it was restarted.

 

HA event logs of FGVMxxxxxEOUS79 (Fortinet2):

 

date=2024-09-06 time=21:28:52 logid="0108037892" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="work" vcluster_member=0 hostname="Fortinet2" sn="FGVMxxxxxEOUS79"
date=2024-09-06 time=21:28:49 logid="0108037894" logdesc="Virtual cluster member joined" msg="Virtual cluster detected member join" vcluster=1 ha_group=10 sn="FGVMxxxxxTCTFB4"
date=2024-09-06 time=21:28:49 logid="0108037899" logdesc="HA device interface peer information" msg="HA device(interface) peerinfo" ha_role="primary" devintfname="port2"
date=2024-09-06 time=21:28:00 logid="0108037892" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="work" vcluster_member=0 hostname="Fortinet2" sn="FGVMxxxxxEOUS79"
date=2024-09-06 time=21:27:57 logid="0108037893" logdesc="Virtual cluster member dead" msg="Virtual cluster detected member dead" vcluster=1 ha_group=10 sn="FGVMxxxxxTCTFB4"
date=2024-09-06 time=21:27:57 logid="0108037910" logdesc="Heartbeat packet lost" msg="Heartbeat packet lost" ha_role="primary" devintfname="port2"

 

==========================================
 
Logs when HA uptime reset is done in the Primary device. The fortiGate is in a HA cluster and already in sync.
 
  • Primary: Fortinet2 , FGVMxxxxxEOUS79, HA cluster index = 0.
  • Secondary : Fortinet1 , FGVMxxxxxTCTFB4, HA cluster index = 1.
 
Executing 'dia sys ha reset-uptime' in Primary Fortinet2  will force HA election and Fortinet1 gets elected as Primary due to higher uptime.
In such case, the below log in the newly selected primary device will be visible:
 
Primary selected using:
 
    <2024/09/06 22:29:00> vcluster-1: FGVMxxxxxTCTFB4 is selected as the primary because its uptime is larger than peer member FGVMxxxxxEOUS79.
    <2024/09/06 21:28:49> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because its uptime is larger than peer member FGVMxxxxxTCTFB4.
 
<2024/09/06 22:29:00>FGVMxxxxxTCTFB4 (Fortinet1) became Primary since its uptime was higher then FGVMxxxxxEOUS79 (Fortinet2).
 
date=2024-09-06 time=22:29:02 logid="0108037892" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="work" vcluster_member=0
date=2024-09-06time=21:30:40logid="0108037903" logdesc="Synchronization status with primary" msg="The sync status with theprimary" sync_type="external-files"sync_status="in-sync"
 
Only the change of HA_role change log is visible in Primary device.
FortiGate in which uptime is reset, below logs with details of the admin account and user interface used to reset will be visible.
 
date=2024-09-06 time=22:29:02 logid="0108037892" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="standby" vcluster_member=1 hostname="Fortinet2" sn="FGVMxxxxxEOUS79"
 
date=2024-09-06 time=22:29:00 logid="0108035014"logdesc="HA reset uptime" user="admin" ui="jsconsole(192.168.181.1)" msg="Reset HA uptime"
 
==========================================
 
Logs, when HA monitored interface, goes down. In the current setup, port1, port3, and port4 are monitored interfaces.
HA failover is triggered if the physical link of the monitored interface goes down.
 
Setup.
Primary: Fortinet1 , FGVMxxxxxTCTFB4, HA cluster index = 1.
Secondary: Fortinet2, FGVMxxxxxEOUS79, HA cluster index = 0.
 
config system ha
    set monitor "port1" "port3" "port4"
end
 
Port4 of the Fortigate1 went down triggering HA Primary failover to Fortinet2(FGVMxxxxxEOUS79).
 
Primary selected using:
 
  •  <2024/09/06 22:48:03> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because the value 0 of link-failure + pingsvr-failure is less than peer member FGVMxxxxxTCTFB4.
  • <2024/09/06 22:29:00> vcluster-1: FGVMxxxxxTCTFB4 is selected as the primary because its uptime is larger than peer member FGVMxxxxxEOUS79.
  • <2024/09/06 21:28:50> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because its uptime is larger than peer member FGVMxxxxxTCTFB4.
  • <2024/09/06 21:27:58> vcluster-1: FGVMxxxxxEOUS79 is selected as the primary because it is the only member in the cluster.
 
MONDEV stats:
 
    FGVMxxxxxEOUS79(updated 1 seconds ago):
        port1: physical/10000full, up, rx-bytes/packets/dropped/errors=1928314/7960/0/0, tx=998731/3349/0/0
        port3: physical/10000full, up, rx-bytes/packets/dropped/errors=252416/3594/0/0, tx=0/0/0/0
        port4: physical/10000full, up, rx-bytes/packets/dropped/errors=252416/3594/0/0, tx=0/0/0/0
    FGVMxxxxxTCTFB4(updated 0 seconds ago):
        port1: physical/10000full, up, rx-bytes/packets/dropped/errors=1520687/6031/0/0, tx=594962/2213/0/0
        port3: physical/10000full, up, rx-bytes/packets/dropped/errors=196920/2802/0/0, tx=0/0/0/0
        port4: physical/00, down, rx-bytes/packets/dropped/errors=196560/2796/0/0, tx=0/0/0/0
 
Event logs of Fortinet2:
 
date=2024-09-06 time=22:48:05 logid="0108037892" type="event" subtype="ha" level="notice" vd="root" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="work" vcluster_member=0 hostname="Fortinet2" sn="FGVMxxxxxEOUS79"
 
Event logs of Fortinet1:
 
date=2024-09-06 time=22:48:05 logid="0108037892" logdesc="Virtual cluster member state moved" msg="Virtual cluster's member state moved" ha_role="primary" vcluster=1 vcluster_state="standby" vcluster_member=1 hostname="Fortinet1" sn="FGVMxxxxxTCTFB4"
date=2024-09-06 time=22:48:03 logid="0108037898" logdesc="HA device interface failed" msg="HA device(interface) fail" ha_role="primary" devintfname="port4"
Contributors