Description
This article discusses ‘link-cost-threshold’, an attribute available when using SD-WAN Best Quality Rules.
Scope
FortiGate, SD-WAN
Solution
As a primer, FortiGate SD-WAN allows administrators to define Rules that can be used to select which outgoing interface(s) should be used when forwarding user traffic. There are four modes available for these SD-WAN Rules
- Manual (default)
- Best Quality
- Lowest Cost (SLA)
- Maximize Bandwidth (SLA)
- Note that Maximize Bandwidth has been removed as a dedicated mode as of FortiOS 7.4.1 and later. Instead, Manual and Lowest Cost (SLA) rules may optionally enable load-balancing functionality (see: FortiOS 7.4 New Features - Using load balancing in a manual SD-WAN rule without configuring an SLA t...)
In the above picture, the ‘Best Quality’ strategy has been selected. This mode instructs the FortiGate to select the best outgoing interface based on the specified SLA metric (for example, the above picture shows that Latency is the measured metric, so the interface with the best/lowest latency wins).
Notably, the order of the interfaces in the SD-WAN Best Quality rule matters. The first interface in the list (port1 in this example) is treated as the 'preferred' interface and receives an 'advantage', whereas all other interfaces are non-preferred. This advantage is controlled by an option named link-cost-threshold, which is only available for SD-WAN Rules of type Best Quality (mode priority) or the legacy CLI-only type Automatic (mode auto):
config system sdwan
config service
edit <id>
set mode [ priority | auto ]
set link-cost-threshold <0 - 10000000, default = 10>
next
end
Note:
In v6.4.0 and earlier, SDWAN was configured under config system virtual-wan-link.
How link-cost-threshold affects the SDWAN outgoing interface selection:
As noted above, Best Quality rules will take the measured metric defined by the Quality criteria (aka link-cost-factor, such as latency, jitter, packet-loss, etc.,) and use that to rank interfaces from best to worst. For the preferred interface (i.e. the first interface in the list), there will be two versions of this measured metric:
- There is the real metric, which will be displayed in the SD-WAN GUI as well as in the output of diagnose sys sdwan service <id> in the CLI. In the above example, this would be 100ms for port1.
- Then, there is the adjusted metric. This metric is calculated using the following equation and will be used by SDWAN as the metric for the preferred interface instead of the real metric:
A = R / (1 + L/100), where:
- A is the adjusted metric,
- R is the real metric
- L is the link-cost-threshold as a percent (e.g. L = 10)
Note regarding which interfaces receive link-cost-threshold advantage:
In earlier FortiOS (6.x), only the first interface in the list gains an advantage based on link-cost-threshold. All other interfaces (2nd, 3rd, etc.) are equally treated as non-preferred and will not have any advantages over each other.
In later FortiOS, all interfaces gain the advantage, but only over the lower-preference members. For example, if the SD-WAN rule contained port1, port2, port3, and port4, and if the link-cost-threshold was 10, then the following must be taken into consideration:
- First, it is important to define certain roles, which are Highest Priority Member (HtPM), Higher Priority Member (HrPM) and Lower Priority Member (LPM). these roles are set based on the initial configured order of interfaces in an SD-WAN rule.
- The HtPM role belongs to the first interface (i.e. the most preferred) configured on the SD-WAN rule. In this case it is port1, a
- The HrPM role is any interface that is higher/more-preferred compared to another.
- For example, port2 is an HrPM relative to port3, and port3 is an HrPM relative to port4.
- The LPM role is any interface that is lower/less-preferred compared to another. Note that interfaces may have dual-roles.
- For example, port3 is an LPM relative to port2, and yet it is also an HrPM relative to port4. On the other hand, port4 is an LPM relative to both port2 and port3.
- The Highest Priority Member (HtPM, port1 in this case) will always be assessed by it's Adjusted metric, no matter where it is currently positioned in the SD-WAN rule.
- For example, if port1 had a Real metric of 100ms of latency (Adjusted metric = ~91ms), then it will always become the outgoing interface if the other interfaces have Real metrics greater than ~91ms.
- If a Lower Priority Member (LPM) wants to be moved above a Higher Priority Member (HrPM), then the LPM must beat the HrPM's Adjusted metric.
- For example, if port2's Real metric is 200ms of latency (Adjusted metric = ~180ms) then port3's latency must be lower than 180ms for it to be moved above port2 in the SD-WAN rule.
- However, if an HrPM wants to return to its original position above an LPM, it does not use the Adjusted metric; instead, the members are compared using their Real metric.
- In the same example as above, port3 (LPM) has a Real metric of 175ms of latency, and port2 has a Real metric of 200ms. The port2 interface would NOT use an Adjusted metric here, and so its Real metric must become lower than 175ms for it to beat port3 and retake its original position in the SD-WAN rule.
Examples using Latency:
In the earlier screenshot's example #1, port1 is the preferred interface with a real latency metric of 100ms, and port2 is a non-preferred interface with a metric of 96ms. When adjusted using a default link-cost-threshold of 10, port1's adjusted metric is ~90.9ms, and so it is selected as the outgoing interface for the SD-WAN rule:
A = 100 / (1 + 10/100)
A = 100 / (1 + 0.1)
A = 100 / 1.1
A = ~90.909 (which is lower/better than port2's 96ms of latency, so port1 is the preferred outgoing interface)
In example 2, port2's latency is now 88ms, which is lower/better than port1's 90.9ms adjusted latency, and so port2 is now chosen as the outgoing interface for the SD-WAN rule. Increasing the link-cost-threshold advantage to a larger value can allow port1 to become the chosen outgoing interface again, if desired:
config system sdwan
config service
edit 1
set link-cost-threshold 20
next
end
A = 100 / (1 + 20/100)
A = 100 / (1 + 0.2)
A = 100 / 1.2
A= ~83.333 (which is now lower/better than port2's 88ms of latency, so port1 is once again the preferred outgoing interface).
Note regarding the Bandwidth metric specifically:
The above calculation is relevant for latency, jitter, packet-loss, and custom profiles. When adjusting Bandwidth metrics with link-cost-threshold, a different calculation is used:
ABWD = LPM AvBW * L/100, and
AvBWD = LPM AvBW - HPM AvBW, where
- AdBWD = Adjusted Bandwidth Difference
- AvBWD = Average Bandwidth Difference
- LPM AvBW = Average Bandwidth of the Lower Priority Member
- HPM AvBW = Average Bandwidth of the Higher Priority Member
- L = link-cost-threshold
The above formula calculates the Adjusted Bandwidth Difference (AWBD, which is a percentage of the LPM's Average Bandwidth) and compares it against the Average Bandwidth Difference (which is the difference between the average bandwidth of the Higher Priority Member and Lower Priority Member). The following is the expected result for this comparison:
- If AvBWD > ABWD, then the LPM is placed above the HPM in the SD-WAN rule (i.e. LPM is more preferred because the LPM's average bandwidth exceeds the HPM's average bandwidth plus the adjusted bandwidth difference).
- If AvBWD =< ABWD, then the LPM is placed below the HPM in the SD-WAN rule (i.e. the HPM's average bandwidth + the adjusted bandwidth difference is greater than the LPM average bandwidth).
- The above will also be true if the LPM AvBW =< HPM AvBW (i.e. the LPM only takes over if it has a significantly higher average bandwidth compared to the HPM).
Example using Bandwidth:
If port2 (HPM) has an average bandwidth of 500Mbps, port3 (LPM) has an average bandwidth of 300Mbps, and the link-cost-factor is 10, then:
AWBD = LPM AvBW * L/100
AWBD = 300 * 10/100
AWBD = 300 * 0.1
AWBD = 30
AvBWD = LPM AvBW - HPM AvBW
AvBWD = 300 - 500
AvBWD = -200
Per the above comparison, AvBWD =< ABWD (-200 < 30), so port3 (LPM) remains below port2 in the SD-WAN rule list.
Note regarding SD-WAN CLI metrics:
The output of diagnose sys sdwan service <id> (aka diagnose sys virtual-wan-link service <id> in FortiOS 6.4.0 and earlier) will show the real/non-adjusted metric for each interface as well as the currently-configured link-cost-threshold:
FortiGate # diagnose sys sdwan service
Service(1): Address Mode(IPV4) flags=0x0
TOS(0x0/0x0), Protocol(0: 1->65535), Mode(auto), link-cost-factor(latency), link-cost-threshold(10), health-check(ping)
Members:
1: Seq_num(2), alive, latency: 0.011
2: Seq_num(1), alive, latency: 0.018, selected
Dst address: 10.100.21.0-10.100.21.255
Service(1): Address Mode(IPV4) flags=0x0
TOS(0x0/0x0), Protocol(0: 1->65535), Mode(auto), link-cost-factor(latency), link-cost-threshold(10), health-check(ping)
Members:
1: Seq_num(2), alive, latency: 0.011
2: Seq_num(1), alive, latency: 0.018, selected
Dst address: 10.100.21.0-10.100.21.255
If setting link-cost-threshold to 0, FortiGate performs a strict metric comparison where no advantage is provided to the first/preferred SD-WAN interface (i.e. the interface with the objectively best measured metric wins). In the event that two or more members have the same metric, the FortiGate will use the SD-WAN member's priority as a tiebreaker (configured under the config members sub-section of config system sdwan).
