Troubleshooting Tip: SD-WAN service rules not working as expected on BGP ECMP loopback deployment

[fwiratama]

Description	This article describes the issue of SD-WAN service rules not working as expected on a FortiGate, where the rules are not behaving as configured, causing traffic to take an unexpected path. The deployment is utilizing BGP ECMP (Equal Cost Multi-Path) for spoke-to-hub routing.
Scope	FortiGate.
Solution	To troubleshoot the issue of SD-WAN service rules not working as expected, follow these steps: Check the SD-WAN configuration to ensure that the rules are correctly configured and that the members are properly defined. In this example, there is a total of 10 SD-WAN members in a single rule. edit 100 set name "BRANCH-MPLS1" ... set priority-members 1 2 3 4 11 21 22 23 24 31 set tie-break fib-best-match next Verify that the routing information is correct and that the BGP network is properly configured. This example is focusing on the destination subnet of 8.8.4.4/32. FOCLAB # get router info routing details 8.8.4.4 Routing table for VRF=0 Routing entry for 8.8.4.4/32 Known via "bgp", distance 20, metric 0, best Last update 07:03:11 ago * vrf 0 10.53.4.2, tag 2 priority 1 (recursive via t_hub2-1_111 tunnel 149.238.64.56), best-match (recursive via t_hub2-1_211 tunnel 10.0.208.78), best-match (recursive via t_hub2-1_121 tunnel 10.1.137.109), best-match (recursive via t_hub2-1_131 tunnel 10.1.137.111), best-match (recursive via t_hub2-1_141 tunnel 10.1.137.113), best-match * vrf 0 10.53.4.1, tag 1 priority 1 (recursive via t_hub1-1_111 tunnel 149.238.65.56), best-match (recursive via t_hub1-1_211 tunnel 10.0.208.82), best-match (recursive via t_hub1-1_121 tunnel 10.1.137.108), best-match (recursive via t_hub1-1_131 tunnel 10.1.137.110), best-match (recursive via t_hub1-1_141 tunnel 10.1.137.112), best-match There are a total of 10 next hops available for ECMP. Check the SD-WAN health check and member status to ensure that all members are up and running. FOCLAB # diagnose sys sdwan service Service(100): Address Mode(IPV4) flags=0x200 use-shortcut-sla Tie break: cfg Gen(3939), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order Service role: standalone Members(10): 1: Seq_num(1 t_hub1-1_111), alive, sla(0x1), gid(0), cfg_order(0), local cost(10), selected 2: Seq_num(2 t_hub1-1_121), alive, sla(0x1), gid(0), cfg_order(1), local cost(10), selected 3: Seq_num(3 t_hub1-1_131), alive, sla(0x1), gid(0), cfg_order(2), local cost(10), selected 4: Seq_num(4 t_hub1-1_141), alive, sla(0x1), gid(0), cfg_order(3), local cost(10), selected 5: Seq_num(11 t_hub1-1_211), alive, sla(0x1), gid(0), cfg_order(4), local cost(10), selected 6: Seq_num(21 t_hub2-1_111), alive, sla(0x1), gid(0), cfg_order(5), local cost(10), selected 7: Seq_num(22 t_hub2-1_121), alive, sla(0x1), gid(0), cfg_order(6), local cost(10), selected 8: Seq_num(23 t_hub2-1_131), alive, sla(0x1), gid(0), cfg_order(7), local cost(10), selected 9: Seq_num(24 t_hub2-1_141), alive, sla(0x1), gid(0), cfg_order(8), local cost(10), selected 10: Seq_num(31 t_hub2-1_211), alive, sla(0x1), gid(0), cfg_order(9), local cost(10), selected Src address(1): 0.0.0.0-255.255.255.255 Based on the output of the diagnose command and routing info, the outgoing interface of the traffic going to destination 8.8.4.4 should be routed via SD-WAN member 1 interface t_hub1-1_111. However, FortiGate routes the traffic via another interface, in this case SD-WAN member 2 interface t_hub1-1_121. This behavior can be observed by running 'diagnose sys session list': FOCLAB # diagnose sys session list ... orgin->sink: org pre->post, reply pre->post dev=37->431546/431546->37 gwy=10.1.137.108/10.239.88.67 hook=pre dir=org act=noop 10.239.88.67:56608->8.8.4.4:8080(0.0.0.0:0) hook=post dir=reply act=noop 8.8.4.4:8080->10.239.88.67:56608(0.0.0.0:0 FOCLAB # diagnose netlink interface list ... if=t_hub1-1_121 family=00 type=768 index=431546 mtu=1420 link=0 master=0 The reason for this behavior is due to a limitation in FortiOS, where SD-WAN limits a maximum of 8 paths of an ECMP outgoing path. To confirm this behavior, run the command 'diagnose ip route list' to get the list of outgoing interfaces in the kernel for a given route; only the top 8 entries will be considered in outgoing interface selection for an SD-WAN rule. FOCLAB # diagnose ip route list | grep -A 15 8.8.4.4 tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->8.8.4.4/32 pref=0.0.0.0 gwy=10.1.137.113 flag=04 hops=0 oif=431551(t_hub2-1_141) gwy=10.1.137.111 flag=04 hops=0 oif=431549(t_hub2-1_131) gwy=10.1.137.109 flag=04 hops=0 oif=431547(t_hub2-1_121) gwy=10.0.208.78 flag=04 hops=0 oif=47(t_hub2-1_211) gwy=149.238.64.56 flag=04 hops=0 oif=45(t_hub2-1_111) gwy=10.1.137.112 flag=04 hops=0 oif=431550(t_hub1-1_141) gwy=10.1.137.110 flag=04 hops=0 oif=431548(t_hub1-1_131) gwy=10.1.137.108 flag=04 hops=0 oif=431546(t_hub1-1_121) gwy=10.0.208.82 flag=04 hops=0 oif=46(t_hub1-1_211) <----- Ignored 9th on the list. gwy=149.238.65.56 flag=04 hops=0 oif=44(t_hub1-1_111) <----- Ignored 10th on the list. In this example, even though t_hub1-1_111 should be prioritized as the exit interface, it is ignored due to the limitation, and t_hub1-1_121 is picked as the outgoing interface. This limitation is removed on v7.2.11 and v7.4.x release.