Technical Tip: How to set HA ping server threshold

subramanis · ‎05-25-2021

Description

The link monitor is used monitor the network units which is not directly connected to the cluster and can use this link-monitor for HA failover if it fails.

The pingserver-failover-threshold value has to be configured appropriately in the HA settings to cause the failover.

Solution

Link monitor configuration.

Primary # show system link-monitor
# config system link-monitor
    edit "L_M_Port1"
        set srcintf "port1"
        set server "8.8.8.8" "8.8.4.4" "1.1.1.1"
        set ha-priority 5
    next
end

By default, the link-monitor ha-priority is set to 1.
set ha-priority 1 <-----

HA configuration.

Primary (ha) # show
# config system ha
    set group-name "Test"
    set mode a-p
    set password ENC pCgOG8Rmx8BTh+UijsHiNqR6rgFltf6umrsbKI9EY+
    set hbdev "ha1" 0
    set ha-mgmt-status enable
    # config ha-mgmt-interfaces
        edit 1
            set interface "mgmt"
            set gateway 10.109.63.254
        next
    end
    set override disable
    set pingserver-monitor-interface "port1"
   set pingserver-failover-threshold 10
end

By default, the HA pingserver-failover-threshold is set to 0.
set pingserver-failover-threshold 0 <-----

Link monitor status is alive.

Primary # diagnose sys link-monitor status
Link Monitor: L_M_Port1, Status: alive, Server num(3), Flags=0x1 init, Create time: Sun May 23 10:45:46 2021
Source interface: port1 (11)
Interval: 500 ms
Peer: 8.8.8.8(8.8.8.8)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.8.8/32, gwy(192.168.1.2)
        protocol: ping, state: alive   <-----
        Latency(Min/Max/Avg): 1.921/2.103/1.973 ms
        Jitter(Min/Max/Avg): 0.003/0.142/0.034
        Packet lost: 0.000%
        Number of out-of-sequence packets: 0
                  Fail Times(0/5)
                  Packet sent: 4749, received: 4749, Sequence(sent/rcvd/exp): 4750/4750/4751
Peer: 8.8.4.4(8.8.4.4)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.4.4/32, gwy(192.168.1.2)
        protocol: ping, state: alive    <-----
        Latency(Min/Max/Avg): 1.953/2.014/1.987 ms
        Jitter(Min/Max/Avg): 0.000/0.061/0.015
        Packet lost: 0.000%
        Number of out-of-sequence packets: 0
                  Fail Times(0/5)
                  Packet sent: 4749, received: 4749, Sequence(sent/rcvd/exp): 4750/4750/4751
Peer: 1.1.1.1(1.1.1.1)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->1.1.1.1/32, gwy(192.168.1.2)
        protocol: ping, state: alive    <-----
        Latency(Min/Max/Avg): 1.927/2.061/1.963 ms
        Jitter(Min/Max/Avg): 0.000/0.123/0.035
        Packet lost: 0.000%
        Number of out-of-sequence packets: 0
                  Fail Times(0/5)
                  Packet sent: 4749, received: 4749, Sequence(sent/rcvd/exp): 4750/4750/4751

HA Status

Master: Primary , FG100E4Q16-----1, HA cluster index = 1
Slave : Secondary , FG100E4Q16-----4, HA cluster index = 0

Link monitor status is dead.

Primary # diagnose sys link-monitor status
Link Monitor: L_M_Port1, Status: die, Server num(3), Flags=0x9 init, Create time: Sun May 23 10:45:46 2021
Source interface: port1 (11)
Interval: 500 ms
Peer: 8.8.8.8(8.8.8.8)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.8.8/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 100.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(3/5)
                  Packet sent: 5862, received: 5563, Sequence(sent/rcvd/exp): 5863/5564/5565
Peer: 8.8.4.4(8.8.4.4)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.4.4/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 100.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(3/5)
                  Packet sent: 5862, received: 5563, Sequence(sent/rcvd/exp): 5863/5564/5565
Peer: 1.1.1.1(1.1.1.1)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->1.1.1.1/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 100.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(3/5)
                  Packet sent: 5862, received: 5563, Sequence(sent/rcvd/exp): 5863/5564/5565

HA Status.

Master: Primary , FG100E4Q16-----1, HA cluster index = 1 <----- Failover did not happen.
Slave : Secondary , FG100E4Q16-----4, HA cluster index = 0

If pingserver-failover-threshold 10 is configured, the total link monitor HA priority will be 5 when three of the servers failed in the link monitor, which is lower than the failover threshold (10) so a failover will not occur.
The total link monitor HA priority should be equal/Higher than the failover threshold, causing a failover.

To change the pingserver-failover-threshold to 5 or ha-priority to 10 to immediate failover when all the three servers fail.

Primary # show system link-monitor
# config system link-monitor
    edit "L_M_Port1"
        set srcintf "port1"
        set server "8.8.8.8" "8.8.4.4" "1.1.1.1"
        set ha-priority 5
    next
end

Primary # show system ha
# config system ha
    set group-name "Test"
    set mode a-p
    set password ENC pCgOG8Rmx8BTh+UijsHiNqR6rgFltf6umrsbKI9EY+    set hbdev "ha1" 0
    set ha-mgmt-status enable
    config ha-mgmt-interfaces
        edit 1
            set interface "mgmt"
            set gateway 10.109.63.254
        next
    end
    set override disable
    set pingserver-monitor-interface "port1"
    set pingserver-failover-threshold 5 <----- pingserver-failover-threshold value changed to 5.
end

The total link monitor HA priority (5) is equal to failover threshold (5), causing a failover.

Primary # diagnose sys link-monitor status
Link Monitor: L_M_Port1, Status: die, Server num(3), Flags=0x9 init, Create time: Sun May 23 10:45:46 2021
Source interface: port1 (11)
Interval: 500 ms
Peer: 8.8.8.8(8.8.8.8)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.8.8/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 7.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(2/5)
                  Packet sent: 7042, received: 5737, Sequence(sent/rcvd/exp): 7043/7035/7036
Peer: 8.8.4.4(8.8.4.4)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.4.4/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 7.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(2/5)
                  Packet sent: 7042, received: 5737, Sequence(sent/rcvd/exp): 7043/7035/7036
Peer: 1.1.1.1(1.1.1.1)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->1.1.1.1/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 6.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(2/5)
                  Packet sent: 7042, received: 5737, Sequence(sent/rcvd/exp): 7043/7035/7036

HA Status.

Slave : Primary , FG100E4Q16-----1, HA cluster index = 1 <----- HA failover to Secondary.
Master: Secondary , FG100E4Q16-----4, HA cluster index = 0

When there are multiple servers configured on the link monitor and the link monitor only fails when no responses are received from all of the servers.
3 servers have been configured on the link monitor and two of them failed but the failover did not happen, still, the Primary unit is master.

Primary # diagnose sys link-monitor status
Link Monitor: L_M_Port1, Status: alive, Server num(3), Flags=0x1 init, Create time: Sun May 23 10:45:46 2021
Source interface: port1 (11)
Interval: 500 ms
Peer: 8.8.8.8(8.8.8.8)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.8.8/32, gwy(192.168.1.2)
        protocol: ping, state: alive <-----
        Latency(Min/Max/Avg): 1.912/2.056/1.963 ms
        Jitter(Min/Max/Avg): 0.002/0.112/0.032
        Packet lost: 0.000%
        Number of out-of-sequence packets: 0
                  Fail Times(0/5)
                  Packet sent: 35034, received: 20086, Sequence(sent/rcvd/exp): 35035/35035/35036
Peer: 8.8.4.4(8.8.4.4)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->8.8.4.4/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 100.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(0/5)
                  Packet sent: 35034, received: 19969, Sequence(sent/rcvd/exp): 35035/34919/34920
Peer: 1.1.1.1(1.1.1.1)
        Source IP(192.168.1.1)
        Route: 192.168.1.1->1.1.1.1/32, gwy(192.168.1.2)
        protocol: ping, state: die <-----
        Packet lost: 100.000%
        Number of out-of-sequence packets: 0
                  Recovery times(0/5) Fail Times(0/5)
                  Packet sent: 35034, received: 19968, Sequence(sent/rcvd/exp): 35035/34919/34920

HA Status.

Master: Primary , FG100E4Q16-----1, HA cluster index = 1 <-----

Slave : Secondary , FG100E4Q16-----4, HA cluster index = 0

Link Monitor on multiple interfaces.

The link monitor has configured on two different interface.

Primary # show system link-monitor
# config system link-monitor
    edit "L_M_Port1"
        set srcintf "port1" <-----
        set server "8.8.8.8"
        set ha-priority 5
    next
    edit "L_M_Wan1"
        set srcintf "wan1" <-----
        set server "8.8.4.4"
        set ha-priority 5
    next
end

--> pingserver-failover-threshold 5 : The failover will occur If the link monitor fails on any one of the interface which is equal to 5 .
--> pingserver-failover-threshold 10 : The failover will not occur If the link monitor fails on any one of the interface which is not equal to 5 (the link monitor should faile on both interface to cause the failover).

Related Articles

Technical Tip: Combining Remote Link Monitoring with FGCP cluster High Availability

HA Remote IP Monitoring