Troubleshooting Tip: FortiGate behavior when executing the command 'diagnose sys ha reset-uptime-primary-only' on HA units

[ojacinto]

Description	This article describes what happens when executing the command 'diagnose sys ha reset-uptime-primary-only' while configuring 'set override-disable'.
Scope	FortiGate v7.4.0 and later.
Solution	In the current example, HA (A-P mode) is configured between FortiGate-VMs FGVM04-HA01, FGVM04TM24003359 <----- Primary unit. FGVM04-HA02, FGVM04TM24000439 <----- Secondary unit. config system ha     set group-id 16     set group-name "HA-TACMEX"     set mode a-p     set password ENC 94odNCJomUZjyyazc     set hbdev "port10" 0      set session-pickup enable     set override disable     set monitor "port1" "port2" end From v7.4.0, the command 'diagnose sys ha reset-uptime-primary-only' has been added to the HA diagnostic commands so the options to reset the HA unit uptime are: reset-uptime                 <----- Reset HA up time. reset-uptime-primary-only    <----- Reset HA up time (can take effect on the primary unit only). 'reset-uptime' will reset the HA unit uptime no matter to what HA unit the command is applied. 'reset-uptime-primary-only' will take only effect if it is applied on the primary unit. Verification: FGVM04-HA01 (global) # get system  ha status  HA Health Status:      WARNING: FGVM04TM24003359 has mondev down;      WARNING: FGVM04TM24000439 has mondev down;  Model: FortiGate-VM64 Mode: HA A-P Group Name: HA-TACMEX Group ID: 16 Debug: 0 Cluster Uptime: 0 days 21h:31m:28s Cluster state change time: 2024-07-17 14:14:26 Primary selected using:     <2024/07/17 14:14:26> vcluster-1: FGVM04TM24003359 is selected as the primary because its uptime is larger than peer member FGVM04TM24000439.     <2024/07/17 14:06:04> vcluster-1: FGVM04TM24000439 is selected as the primary because its override priority is larger than peer member FGVM04TM24003359.     <2024/07/17 14:03:38> vcluster-1: FGVM04TM24003359 is selected as the primary because it's the only member in the cluster.     <2024/07/17 14:03:29> vcluster-1: FGVM04TM24003359 is selected as the primary because UPGRADE_SECONDARY flag is set on peer member FGVM04TM24000439. ses_pickup: enable, ses_pickup_delay=disable override: disable ... Primary: FGVM04TM24003359, HA operating index = 0 Secondary: FGVM04TM24000439, HA operating index = 1 FGVM04-HA01 (global) # diagnose  sys ha dump-by group              HA information. group-id=16, group-name='HA-TACMEX' has_no_aes128_gcm_sha256_member=0 gmember_nr=2 'FGVM04TM24000439': ha_ip_idx=1, hb_packet_version=9, last_hb_jiffies=224359, linkfails=2, weight/o=0/0, support_aes128_gcm_sha256=1         hbdev_nr=1: port10(mac=000c..3c, last_hb_jiffies=224359, hb_lost=0),  'FGVM04TM24003359': ha_ip_idx=0, hb_packet_version=14, last_hb_jiffies=0, linkfails=0, weight/o=0/0, support_aes128_gcm_sha256=1 vcluster_nr=1 vcluster-1: start_time=1721246536(2024-07-17 14:02:16), state/o/chg_time=2(work)/3(standby)/1721247266(2024-07-17 14:14:26)         pingsvr_flip_timeout/expire=3600s/2085s         mondev: port1(prio=50,is_aggr=0,status=1) port2(prio=50,is_aggr=0,status=0)          'FGVM04TM24000439': ha_prio/o=1/1, link_failure=50, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/3         'FGVM04TM24003359': ha_prio/o=0/0, link_failure=50, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=910/1   < --- Primary unit uptime/reset information Login to the Slave unit FGVM04-HA02 and execute the command 'reset-uptime-primary-only': FGVM04-HA01 (global) # execute ha manage 1 admin admin@169.254.0.2's password:  FGVM04-HA02 # config  global   FGVM04-HA02 (global) # diagnose  sys ha reset-uptime-primary-only  FGVM04-HA02 (global) #  FGVM04-HA02 (global) # get system ha status  HA Health Status:      WARNING: FGVM04TM24000439 has mondev down;      WARNING: FGVM04TM24003359 has mondev down;  Model: FortiGate-VM64 Mode: HA A-P Group Name: HA-TACMEX Group ID: 16 Debug: 0 Cluster Uptime: 0 days 21h:35m:20s Cluster state change time: 2024-07-17 14:14:26 Primary selected using: <2024/07/17 14:14:26> vcluster-1: FGVM04TM24003359 is selected as the primary because its uptime is larger than peer member FGVM04TM24000439.  <----- There was not any HA failover.     <2024/07/17 14:06:03> vcluster-1: FGVM04TM24000439 is selected as the primary because its override priority is larger than peer member FGVM04TM24003359.     <2024/07/17 14:06:03> vcluster-1: FGVM04TM24000439 is selected as the primary because it's the only member in the cluster. ses_pickup: enable, ses_pickup_delay=disable override: disable ... FGVM04-HA02 (global) # diagnose  sys  ha dump-by group              HA information. group-id=16, group-name='HA-TACMEX' has_no_aes128_gcm_sha256_member=0 gmember_nr=2 'FGVM04TM24000439': ha_ip_idx=1, hb_packet_version=9, last_hb_jiffies=0, linkfails=0, weight/o=0/0, support_aes128_gcm_sha256=1 'FGVM04TM24003359': ha_ip_idx=0, hb_packet_version=14, last_hb_jiffies=218240, linkfails=2, weight/o=0/0, support_aes128_gcm_sha256=1         hbdev_nr=1: port10(mac=000c..04, last_hb_jiffies=218240, hb_lost=0),  vcluster_nr=1 vcluster-1: start_time=1721247447(2024-07-17 14:17:27), state/o/chg_time=3(standby)/2(work)/1721247266(2024-07-17 14:14:26)         pingsvr_flip_timeout/expire=3600s/1923s         mondev: port1(prio=50,is_aggr=0,status=1) port2(prio=50,is_aggr=0,status=0)          'FGVM04TM24000439': ha_prio/o=1/1, link_failure=50, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/3 'FGVM04TM24003359': ha_prio/o=0/0, link_failure=50, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=910/1<----- Master unit Uptime/reset count did not change. If the same command is applied on the primary unit, a failover will be performed to the new primary unit FGVM04-HA02: FGVM04-HA01 (global) # fnsysctl date Wed Jul 17 14:45:48 CST 2024 FGVM04-HA01 (global) # diagnose  sys  ha reset-uptime-primary-only   FGVM04-HA02 (global) # get  system  ha status  HA Health Status:      WARNING: FGVM04TM24000439 has mondev down;      WARNING: FGVM04TM24003359 has mondev down;  Model: FortiGate-VM64 Mode: HA A-P Group Name: HA-TACMEX Group ID: 16 Debug: 0 Cluster Uptime: 0 days 21h:39m:34s Cluster state change time: 2024-07-17 14:45:53 Primary selected using: <2024/07/17 14:45:53> vcluster-1: FGVM04TM24000439 is selected as the primary because its uptime is larger than peer member FGVM04TM24003359.    <-----     <2024/07/17 14:14:26> vcluster-1: FGVM04TM24003359 is selected as the primary because its uptime is larger than peer member FGVM04TM24000439.     <2024/07/17 14:06:03> vcluster-1: FGVM04TM24000439 is selected as the primary because its override priority is larger than peer member FGVM04TM24003359.     <2024/07/17 14:06:03> vcluster-1: FGVM04TM24000439 is selected as the primary because it's the only member in the cluster. ses_pickup: enable, ses_pickup_delay=disable override: disable FGVM04-HA02 (global) # diagnose  sys  ha dump-by group              HA information. group-id=16, group-name='HA-TACMEX' has_no_aes128_gcm_sha256_member=0 gmember_nr=2 'FGVM04TM24000439': ha_ip_idx=1, hb_packet_version=10, last_hb_jiffies=0, linkfails=0, weight/o=0/0, support_aes128_gcm_sha256=1 'FGVM04TM24003359': ha_ip_idx=0, hb_packet_version=17, last_hb_jiffies=246440, linkfails=2, weight/o=0/0, support_aes128_gcm_sha256=1         hbdev_nr=1: port10(mac=000c..04, last_hb_jiffies=246440, hb_lost=0),  vcluster_nr=1 vcluster-1: start_time=1721247447(2024-07-17 14:17:27), state/o/chg_time=2(work)/3(standby)/1721249153(2024-07-17 14:45:53)         pingsvr_flip_timeout/expire=3600s/3528s         mondev: port1(prio=50,is_aggr=0,status=1) port2(prio=50,is_aggr=0,status=0)          'FGVM04TM24000439': ha_prio/o=0/0, link_failure=50, pingsvr_failure=0, flag=0x00000001, mem_failover=0, uptime/reset_cnt=1706/3 'FGVM04TM24003359': ha_prio/o=1/1, link_failure=50, pingsvr_failure=0, flag=0x00000000, mem_failover=0, uptime/reset_cnt=0/2 <----- Old primary unit (reset count increaed by 1).