Technical Tip: How to verify HA cluster members individual uptime

[ESCHAN_FTNT]

Description
This article describes how to verify HA cluster members individual uptime.
 
From the command 'get system ha status'  FGVM04TM19-----3 is being selected as Master due to largest value of uptime:

Master (global) # get system ha status
HA Health Status: OK
Model: FortiGate-VM64
Mode: HA A-P
Group: 100
Debug: 0
Cluster Uptime: 0 days 0:39:28
Cluster state change time: 2020-03-30 17:18:09
Master selected using:
    <2020/03/30 17:18:09> FGVM04TM19-----3 is selected as the master because it has the largest value of uptime.
    <2020/03/30 17:15:46> FGVM04TM19-----4 is selected as the master because it has the largest value of uptime.
    <2020/03/30 16:39:47> FGVM04TM19-----3 is selected as the master because it has the largest value of override priority.
    <2020/03/30 16:39:00> FGVM04TM19-----3 is selected as the master because it's the only member in the cluster.
ses_pickup: enable, ses_pickup_delay=disable
override: enable
Configuration Status:
    FGVM04TM19-----3(updated 4 seconds ago): in-sync
    FGVM04TM19-----4(updated 2 seconds ago): in-sync
System Usage stats:
    FGVM04TM19-----3(updated 4 seconds ago):
        sessions=25, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=63%
    FGVM04TM19-----4(updated 2 seconds ago):
        sessions=16, average-cpu-user/nice/system/idle=0%/0%/0%/99%, memory=61%
HBDEV stats:
    FGVM04TM19-----3(updated 4 seconds ago):
        port10: physical/10000full, up, rx-bytes/packets/dropped/errors=8409460/31503/0/0, tx=35385838/37462/0/0
    FGVM04TM19-----4(updated 2 seconds ago):
        port10: physical/10000full, up, rx-bytes/packets/dropped/errors=33122327/35969/0/0, tx=8409529/31458/0/0
MONDEV stats:
    FGVM04TM19-----3(updated 4 seconds ago):
        port3: physical/10000full, up, rx-bytes/packets/dropped/errors=10471996/40295/0/0, tx=840/14/0/0
    FGVM04TM19-----4(updated 2 seconds ago):
        port3: physical/10000full, up, rx-bytes/packets/dropped/errors=4344523/15513/0/0, tx=360/6/0/0
Master: Master          , FGVM04TM19-----3, cluster index = 1
Slave : Slave           , FGVM04TM19-----4, cluster index = 0
number of vcluster: 1
vcluster 1: work 169.254.0.2
Master: FGVM04TM19-----3, operating cluster index = 0
Slave : FGVM04TM19-----4, operating cluster index = 1

However, from 'get system performance status',  FGVM04TM19-----3 and FGVM04TM19----4 have identical uptime of 44 minutes, how do the HA cluster actually select FGVM04TM19-----3 due to having higher uptime?

Master (global) # get system performance status | grep Uptime
Uptime: 0 days,  0 hours,  44 minutes
Slave (global) # get system performance status | grep Uptime
Uptime: 0 days,  0 hours,  44 minutes

Solution
Use the command '# diagnose sys ha dump-by group' to verify HA member individual uptime:

Master (global) # diagnose sys ha dump-by group
            HA information.
group-id=100, group-name='fortigate'

gmember_nr=2
'FGVM04TM19-----3': ha_ip_idx=1, hb_packet_version=6, last_hb_jiffies=0, linkfails=0, weight/o=0/0
'FGVM04TM19-----4': ha_ip_idx=0, hb_packet_version=8, last_hb_jiffies=236404, linkfails=0, weight/o=0/0
        hbdev_nr=1: port10(mac=000c..05, last_hb_jiffies=236404, hb_lost=0),

vcluster_nr=1
vcluster_0: start_time=1585558721(2020-03-30 16:58:41), state/o/chg_time=2(work)/3(standby)/1585559889(2020-03-30 17:18:09)
        pingsvr_flip_timeout/expire=3600s/3572s
        mondev: port3(prio=50,is_aggr=0,status=1)
        'FGVM04TM19-----3': ha_prio/o=0/1, link_failure=0, pingsvr_failure=0, flag=0x00000001, uptime/reset_cnt=1167/1
        'FGVM04TM19-----4': ha_prio/o=1/0, link_failure=0, pingsvr_failure=0, flag=0x00000000, uptime/reset_cnt=0/1       

Referring to the 'start_time', FGVM04TM19-----3 uptime is from 2020-03-30 16:58:41. Check on FGVM04TM19-----4:

Slave (global) # diagnose sys ha dump-by group
            HA information.
group-id=100, group-name='fortigate'

gmember_nr=2
'FGVM04TM19-----3': ha_ip_idx=1, hb_packet_version=6, last_hb_jiffies=236702, linkfails=0, weight/o=0/0
        hbdev_nr=1: port10(mac=000c..4f, last_hb_jiffies=236702, hb_lost=0),
'FGVM04TM19-----4': ha_ip_idx=0, hb_packet_version=8, last_hb_jiffies=0, linkfails=0, weight/o=0/0

vcluster_nr=1
vcluster_0: start_time=1585559888(2020-03-30 17:18:08), state/o/chg_time=3(standby)/2(work)/1585559889(2020-03-30 17:18:09)
        pingsvr_flip_timeout/expire=3600s/3541s
        mondev: port3(prio=50,is_aggr=0,status=1)
        'FGVM04TM19-----3': ha_prio/o=0/0, link_failure=0, pingsvr_failure=0, flag=0x00000001, uptime/reset_cnt=1167/1
        'FGVM04TM19-----4': ha_prio/o=1/1, link_failure=0, pingsvr_failure=0, flag=0x00000000, uptime/reset_cnt=0/1       

The uptime for FGVM04TM19------4 is actually from 2020-03-30 17:18:08. Therefore FGVM04TM19-----3 is being selected as HA master due to higher uptime.

The time difference between both units is actually 1167 seconds.
Which is higher than the default 'ha-uptime-diff-margin' of 300 seconds, therfore FGVM04TM19-----4 will be selected as Master when override is not enabled.