| Solution |
Consider the link-monitor configuration excerpt below:
config system link-monitor
edit "1"
set srcintf "port2"
set server "8.8.8.8"
set route "10.128.0.0/16"
next
end
To verify active Routing table has port2 as its primary WAN interface:
get router info routing-table all
Routing table for VRF=0
S 0.0.0.0/0 [10/0] via 10.9.15.254, port2, [1/0]
S 10.128.0.0/16 [10/0] via 10.9.15.254, port2, [1/0]
C 172.16.16.0/24 is directly connected, port1
C 192.168.30.0/24 is directly connected, port3
The link-monitor keeps track of the interface’s health by continuously pinging the configured server and will remove configured routes if enough consecutive ping responses fail to arrive.
To verify the health of the interface:
diagnose sys link-monitor status
Link Monitor: 1, Status: alive, Server num(1), Flags=0x1 init, Create time: Sun Jul 4 16:20:25 2021
Source interface: port2 (3)
Interval: 500 ms
Peer: 10.109.21.50(10.109.21.50)
Source IP(10.109.16.223)
Route: 10.109.16.223->10.109.21.50/32, gwy(10.109.16.223)
protocol: ping, state: alive
Latency(Min/Max/Avg): 0.211/0.585/0.362 ms
Jitter(Min/Max/Avg): 0.006/0.298/0.098
Packet lost: 0.000%
Number of out-of-sequence packets: 0
Fail Times(0/5)
Packet sent: 1472, received: 1334, Sequence(sent/rcvd/exp): 1473/1473/1474
The diagnostic example above shows that the status of the link-monitor is 'alive' and associated routes for port2 are therefore still in the active routing table.
To troubleshoot or debug the FortiGate link-monitor functionality, a real-time debug can be run using the commands below:
diagnose debug application link-monitor -1
diagnose debug enable
To stop the debug, use the command given below:
diagnose debug disable
diagnose debug reset
The working debug output for an 'alive' link-monitor would resemble the excerpt below:
lnkmtd::lnkmt_add_monitor(2535): ---> create monitor "1"
lnkmtd::lnkmt_addr_mode_route4_print(118): ---> the route (10.9.11.225->8.8.8.8/32, vrf=0, oif=port2, gw=10.9.15.254) is added. <- This route is specifically for the link-monitor server target.
sys_ha_comm_init()-772
ha_timers_init()-741
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> added route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16) <- Route is added to routing table.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=2, icmp id=2598, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0) <<-Ping sent
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd <<-Received reply
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728415276s 568996us, revd time 1728415276s 572165us
lnkmtd::monitor_proute_cmdb_set(1121): ---> policy routes or internet service routes related to the monitor(1) may be added <- Routes kept as the link-monitor is alive.
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> added route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16) <- Routes kept as the link-monitor is alive.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=3, icmp id=2598, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0) <<-Ping sent
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd <- Received reply.
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728415277s 70689us, revd time 1728415277s 73878us
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=4, icmp id=2598, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)<<-Ping sent
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd <- Received reply.
In the above debug logs, the routes are added as per the configuration. The link-monitor continuously pings the configured server from port2. When the ping probes receive replies, the routes are maintained.
The debugs for a dead link-monitor would resemble the excerpt below:
lnkmtd::lnkmt_add_monitor(2535): ---> create monitor "1"
lnkmtd::lnkmt_addr_mode_route4_print(118): ---> the route (10.9.11.225->8.8.8.8/32, vrf=0, oif=port2, gw=0.0.0.0) is added.
sys_ha_comm_init()-772
ha_timers_init()-741
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> added route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=2, icmp id=2574, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0) <- Fail count 1.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=3, icmp id=2574, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(1) ) <- Fail count 2.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=4, icmp id=2574, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(2) ) <- Fail count 3.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=5, icmp id=2574, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(3) ) <- Fail count 4.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=6, icmp id=2574, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(4) ) <- Fail count 5.
lnkmtd::monitor_ppeer_fail(1847): ---> 1(8.8.8.8 ping) is dead. <- After 5 failed pings, the interface is declared as dead.
lnkmtd::monitor_proute_cmdb_set(1121): ---> policy routes or internet service routes related to the monitor(1) may be removed <- After the declaration the route is removed.
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> removed route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16) <<- After the declaration the route is removed
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=7, icmp id=2575, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=8, icmp id=2575, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(1)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=9, icmp id=2575, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(2)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=10, icmp id=2575, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(3)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=11, icmp id=2575, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(4)
lnkmtd::monitor_ppeer_fail(1847): ---> 1(8.8.8.8 ping) is dead.
The default consecutive failed ping count on a FortiGate is 5 pings sent every 500ms. If five pings fail to receive a response, the link-monitor declares the interface as dead and removes associated routes. Even when a link is declared 'dead', the link-monitor will continuously send pings to the configured server and insert the routes back into the routing-table once responses are received again.
The debugs for a dead link-monitor to be alive would resemble the excerpt below:
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54446, icmp id=13011, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54447, icmp id=13011, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(1)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54448, icmp id=13011, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(2)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54449, icmp id=13011, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(3)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54450, icmp id=13011, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(4)
lnkmtd::monitor_ppeer_fail(1847): ---> 1(8.8.8.8 ping) is dead.
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54451, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54452, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(1)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54453, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(2)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54454, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(3)
lnkmtd::lnkmt_addr_mode_route4_print(118): ---> the route (10.9.11.225->8.8.8.8/32, vrf=0, oif=port2, gw=0.0.0.0) is removed.
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> added route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16)
lnkmtd::lnkmt_addr_mode_route4_print(118): ---> the route (10.9.11.225->8.8.8.8/32, vrf=0, oif=port2, gw=10.9.15.254) is added.
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): ---> removed route vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16)
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54455, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(4) <- This was the last ping from ping count.
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd <- And for it received ping response.
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728605611s 880165us, revd time 1728605611s 890962us
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54456, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728605612s 373514us, revd time 1728605612s 376739us
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54457, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728605612s 879680us, revd time 1728605612s 882855us
lnkmtd::ping_send_msg(435): ---> ping 8.8.8.8 seq_no=54458, icmp id=13012, send 20 bytes
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728605613s 371320us, revd time 1728605613s 374482us
lnkmtd::monitor_proto_peer_send_request(638): ---> 1(8.8.8.8:ping) send probe packet, fail count(0)
lnkmtd::ping_do_addr_up(136): ---> 1->8.8.8.8(8.8.8.8), rcvd
lnkmtd::monitor_peer_recv(2172): ---> 1 send time 1728605613s 877463us, revd time 1728605613s 880587
lnkmtd::monitor_proute_cmdb_set(1121): ---> policy routes or internet service routes related to the monitor(1) may be added <- After 5 successful ping the status is up.
lnkmtd::lnkmt_addr_mode_do_downgateway4(373): vd(root), oif=port2(4) gateway(0.0.0.0) for subnet(10.128.0.0/16) <- Route is now added.
Related article:
Technical Tip: Link monitor
|
