Troubleshooting Tip: SD-WAN service rules not working as expected on BGP ECMP loopback deployment

To troubleshoot the issue of SD-WAN service rules not working as expected, follow these steps:

Check the SD-WAN configuration to ensure that the rules are correctly configured and that the members are properly defined. In this example, there is a total of 10 SD-WAN members in a single rule.

edit 100
set name "BRANCH-MPLS1"
...
set priority-members 1 2 3 4 11 21 22 23 24 31
set tie-break fib-best-match
next

Verify that the routing information is correct and that the BGP network is properly configured. This example is focusing on the destination subnet of 8.8.4.4/32.

FOCLAB # get router info routing details 8.8.4.4

Routing table for VRF=0
Routing entry for 8.8.4.4/32
Known via "bgp", distance 20, metric 0, best
Last update 07:03:11 ago
* vrf 0 10.53.4.2, tag 2 priority 1 (recursive via t_hub2-1_111 tunnel 149.238.64.56), best-match
(recursive via t_hub2-1_211 tunnel 10.0.208.78), best-match
(recursive via t_hub2-1_121 tunnel 10.1.137.109), best-match
(recursive via t_hub2-1_131 tunnel 10.1.137.111), best-match
(recursive via t_hub2-1_141 tunnel 10.1.137.113), best-match
* vrf 0 10.53.4.1, tag 1 priority 1 (recursive via t_hub1-1_111 tunnel 149.238.65.56), best-match
(recursive via t_hub1-1_211 tunnel 10.0.208.82), best-match
(recursive via t_hub1-1_121 tunnel 10.1.137.108), best-match
(recursive via t_hub1-1_131 tunnel 10.1.137.110), best-match
(recursive via t_hub1-1_141 tunnel 10.1.137.112), best-match

There are a total of 10 next hops available for ECMP.

Check the SD-WAN health check and member status to ensure that all members are up and running.

FOCLAB # diagnose sys sdwan service

Service(100): Address Mode(IPV4) flags=0x200 use-shortcut-sla
Tie break: cfg
Gen(3939), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
Service role: standalone

Members(10):
1: Seq_num(1 t_hub1-1_111), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected
2: Seq_num(2 t_hub1-1_121), alive, sla(0x1), gid(0), cfg_order(1), local cost(10), selected
3: Seq_num(3 t_hub1-1_131), alive, sla(0x1), gid(0), cfg_order(2), local cost(10), selected
4: Seq_num(4 t_hub1-1_141), alive, sla(0x1), gid(0), cfg_order(3), local cost(10), selected
5: Seq_num(11 t_hub1-1_211), alive, sla(0x1), gid(0), cfg_order(4), local cost(10), selected
6: Seq_num(21 t_hub2-1_111), alive, sla(0x1), gid(0), cfg_order(5), local cost(10), selected
7: Seq_num(22 t_hub2-1_121), alive, sla(0x1), gid(0), cfg_order(6), local cost(10), selected
8: Seq_num(23 t_hub2-1_131), alive, sla(0x1), gid(0), cfg_order(7), local cost(10), selected
9: Seq_num(24 t_hub2-1_141), alive, sla(0x1), gid(0), cfg_order(8), local cost(10), selected
10: Seq_num(31 t_hub2-1_211), alive, sla(0x1), gid(0), cfg_order(9), local cost(10), selected
Src address(1):
0.0.0.0-255.255.255.255

Based on the output of the diagnose command and routing info, the outgoing interface of the traffic going to destination 8.8.4.4 should be routed via SD-WAN member 1 interface t_hub1-1_111.

However, FortiGate routes the traffic via another interface, in this case SD-WAN member 2 interface t_hub1-1_121. This behavior can be observed by running 'diagnose sys session list':

FOCLAB # diagnose sys session list

...

orgin->sink: org pre->post, reply pre->post dev=37->431546/431546->37 gwy=10.1.137.108/10.239.88.67
hook=pre dir=org act=noop 10.239.88.67:56608->8.8.4.4:8080(0.0.0.0:0)
hook=post dir=reply act=noop 8.8.4.4:8080->10.239.88.67:56608(0.0.0.0:0

FOCLAB # diagnose netlink interface list

...

if=t_hub1-1_121 family=00 type=768 index=431546 mtu=1420 link=0 master=0

The reason for this behavior is due to a limitation in FortiOS, where SD-WAN limits a maximum of 8 paths of an ECMP outgoing path.

To confirm this behavior, run the command 'diagnose ip route list' to get the list of outgoing interfaces in the kernel for a given route; only the top 8 entries will be considered in outgoing interface selection for an SD-WAN rule.

FOCLAB # diagnose ip route list | grep -A 15 8.8.4.4
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->8.8.4.4/32 pref=0.0.0.0
gwy=10.1.137.113 flag=04 hops=0 oif=431551(t_hub2-1_141)
gwy=10.1.137.111 flag=04 hops=0 oif=431549(t_hub2-1_131)
gwy=10.1.137.109 flag=04 hops=0 oif=431547(t_hub2-1_121)
gwy=10.0.208.78 flag=04 hops=0 oif=47(t_hub2-1_211)
gwy=149.238.64.56 flag=04 hops=0 oif=45(t_hub2-1_111)
gwy=10.1.137.112 flag=04 hops=0 oif=431550(t_hub1-1_141)
gwy=10.1.137.110 flag=04 hops=0 oif=431548(t_hub1-1_131)
gwy=10.1.137.108 flag=04 hops=0 oif=431546(t_hub1-1_121)
gwy=10.0.208.82 flag=04 hops=0 oif=46(t_hub1-1_211) <----- Ignored 9th on the list.
gwy=149.238.65.56 flag=04 hops=0 oif=44(t_hub1-1_111) <----- Ignored 10th on the list.

In this example, even though t_hub1-1_111 should be prioritized as the exit interface, it is ignored due to the limitation, and t_hub1-1_121 is picked as the outgoing interface. This limitation is removed on v7.2.11 and v7.4.x release.