Description
This article describes link health monitoring which measures the health of links by sending probing signals to a server and measuring the link quality based on latency, jitter, and packet loss.
Scope
FortiGate.
Solution
Link-monitor can be configured for status checks. There is no option to configure link-monitor on the GUI and it can be configured in CLI only. The CLI commands below can be used to configure link-monitor.
config system link-monitor
edit "1"
set addr-mode Address mode (IPv4 or IPv6).
set srcintf Interface that receives the traffic to be monitored.
set server-config Mode of server configuration.
set server-type Server type (static or dynamic).
set server IP address of the server(s) to be monitored.
set protocol <ping | tcp-echo | udp-echo | http | https | twamp>
set gateway-ip Gateway IP address used to probe the server.
set route Subnet to monitor.
set source-ip Source IP address used in packet to the server”
set interval Detection interval in milliseconds (20 - 3600 * 1000 msec, default = 500).
set probe-timeout Time to wait before a probe packet is considered lost (20 - 5000 msec, default = 500).
set fail-weight Threshold weight to trigger link failure alert
set failtime Number of retry attempts before the server is considered down (1 - 3600, default = 5).
set recoverytime Number of successful responses received before server is considered recovered (1 - 3600, default = 5).
set probe-count Number of most recent probes that should be used to calculate latency and jitter (5 - 30, default = 30).
set ha-priority HA election priority (1 - 50).
*set update-cascade-interface Enable/disable update cascade interface, default: enable”
[* It is advised to keep disabled as it may cause the production environment down , Make sure it's working before enabling it]
**set update-static-route Enable/disable updating the static route, default: enable”
[** It is advised to keep disabled as it may cause the production environment down , Make sure it's working before enabling it]
set update-policy-route Enable/disable updating the policy route.
set status Enable/disable this link monitor, default: enable.
set diffservcode Differentiated services code point (DSCP) in the IP header of the probe packet.
set class-id Traffic class ID.
set service-detection Only use monitor to read quality values. If enabled, static routes and cascade interfaces will not be updated.
next
end set server-type Server type (static or dynamic).
Note:
Some options are only available in later FortiOS versions.
- The following options are available in v7.0.0 and above:
set route Subnet to monitor.
set diffservcode Differentiated services code point (DSCP) in the IP header of the probe set packet.
set class-id Traffic class ID.
set service-detection Only use monitor to read quality values. If enabled, static routes and cascade interfaces will not be updated.
- The following options are available in v7.0.1 and above:
set server-config Mode of server configuration (default or individual).
set update-policy-route Enable/disable updating the policy route.
- The following options are available in v7.2.0 and above:
set server-type Server type (static or dynamic).
Consider a simple setup where FortiGate is probing the server 10.109.21.50 via the wan1 interface.
config system link-monitor
edit "1"
set srcintf "wan1"
set server "10.109.21.50" <- The server that is probed via WAN1 interface.
next
end
As can be seen in the output below, the status is active, which means FortiGate can reach the server having the IP address 10.109.21.50.
FGT # diagnose sys link-monitor status
Link Monitor: 1, Status: alive, Server num(1), Flags=0x1 init, Create time: Sun Jul 4 16:20:25 2021
Source interface: wan1 (3)
Interval: 500 ms
Peer: 10.109.21.50(10.109.21.50)
Source IP(10.109.16.223)
Route: 10.109.16.223->10.109.21.50/32, gwy(10.109.16.223)
protocol: ping, state: alive
Latency(Min/Max/Avg): 0.211/0.585/0.362 ms
Jitter(Min/Max/Avg): 0.006/0.298/0.098
Packet lost: 0.000%
Number of out-of-sequence packets: 0
Fail Times(0/5)
Packet sent: 1472, received: 1334, Sequence(sent/rcvd/exp): 1473/1473/1474
FGT # get router info routing-table all
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 10.109.31.254, wan1
C 10.109.16.0/20 is directly connected, wan1
When WAN1 goes down or the ping server is not reachable, the default route is removed from the routing table.
FGT # diagnose sys link-monitor status
Link Monitor: 1, Status: die, Server num(1), Flags=0x9 init, Create time: Sun Jul 4 16:20:25 2021
Source interface: wan1 (3)
Interval: 500 ms
Peer: 10.109.21.50(10.109.21.50)
Source IP(10.109.16.223)
Route: 10.109.16.223->10.109.21.50/32, gwy(10.109.16.223)
protocol: ping, state: die
Packet lost: 5.000%
Number of out-of-sequence packets: 0
Recovery times(0/5) Fail Times(1/5)
Packet sent: 2128, received: 1983, Sequence(sent/rcvd/exp): 2129/2122/2123
As visible in the output below, the default route is removed from the routing table due to link monitor failure.
FGT # get router info routing-table all
Routing table for VRF=0
C 10.109.16.0/20 is directly connected, wan1
When the ping server is reachable and the link monitor is restored, the default route is installed again. Link-Monitor will remove only the Static and Policy Route, not the Directly Connected Route. To prevent link-monitor from removing the default route, the following command can be used.
config router static
edit "1"
set link-monitor-exempt enable <- The default is 'disable'.
next
end
Starting v7.0.1, it was added the option to disable updating policy routes when the link health monitor fails:
config system link-monitor
edit "1"
set update-policy-route disable
next
end
Note:
Configure the gateway IP address under the policy route and link-monitor for this feature to work.
When update-policy-route disable is configured, the policy-based route is retained even if the link health monitor detects a failure. To use this feature, it must be enabled to ensure that the associated policy route is disabled when necessary.
It is possible to add multiple servers to avoid false positives caused by monitoring a single IP address.
config system link-monitor
edit "1"
set server "8.8.8.8" "4.2.2.2" "8.8.4.4"
next
Reachability over the link is regarded as satisfactory even if only one probe server is in an 'alive' state when many probe servers are set up as described.
Link Monitor can also be configured to trigger an HA failover when Technical Tip: Combining Remote Link Monitoring with FGCP cluster High Availability, allowing the cluster to fail over based on actual path availability rather than relying solely on Technical Tip: HA failover failure cause by link failure under HA config, which works on Layer 1 status.
The logs can be viewed on FortiGate under Log & Report -> Events -> System Events.
date=2021-07-04 time=16:22:06 eventtime=1625408526938249768 tz="+0200" logid="0100022922" type="event" subtype="system" level="notice" vd="root" logdesc="Link monitor status" name="1" interface="wan1" probeproto="ping" msg="Link Monitor changed state from die to alive, protocol: ping."
date=2021-07-04 time=16:21:41 eventtime=1625408501933624821 tz="+0200" logid="0100022922" type="event" subtype="system" level="warning" vd="root" logdesc="Link monitor status" name="1" interface="wan1" probeproto="ping" msg="Link Monitor changed state from alive to die, protocol: ping."
date=2021-07-04 time=16:20:25 eventtime=1625408425881086208 tz="+0200" logid="0100022922" type="event" subtype="system" level="notice" vd="root" logdesc="Link monitor status" name="1" interface="wan1" probeproto="ping" msg="Link Monitor initial state is alive, protocol: ping"
When a server is not reachable, the logs do not show the server name or IP. To check which server is down, it is required to run the CLI command 'diagnose sys link-monitor status'.
date=2021-07-04 time=16:22:06 eventtime=1736278922124255400 tz="-0500" logid="0100022922" type="event" subtype="system" level="notice" vd="root" logdesc="Link monitor status" name="1" msg="Link monitor state is changed from 0 to 1, please check if this triggers HA failover."
- 0 means up.
- 1 means down.
Here is another article that provides an example where link-monitor is used for a redundant Internet connection with load-balancing: Technical Tip: Redundant Internet connection without load-balancing.
Note:
When configuring the link monitor, it is essential to have local access to the firewall. Incorrect configurations may lead to network inaccessibility.
An email alert notification can be configured whenever there is a change in the link monitor status. This can be configured by following the steps outlined in Technical Tip: E-mail alert when WAN interface went down.
Troubleshooting:
To troubleshoot or debug the FortiGate link-monitor functionality, a real-time debug can be run using the commands below:
diagnose debug application link-monitor -1
diagnose debug enable
To stop the debug, use the command given below:
diagnose debug disable
diagnose debug reset
Related documents:
Combining Remote Link-Monitoring with FGCP cluster
Technical Tip: Command to get the status of link-monitor settings configured
Technical Tip: How to set HA ping server threshold
Technical Tip: Link-Monitor Behavior when Internal LAN IP is use as Source IP
