why the outgoing interface is not complying with SD-WAN rule?

[sean3]

Hi,
thanks for your consistent support.
we are an international company with a site in China and another site in Sweden, now we are using fortigate SD-WAN solution. Two underlay interfaces, one is from public internet, the other one is from Orange MPLS.

we created two tunnel separately from Internet and MPLS as the overlay encapluation tunnel. We use Lowest Cost(SLA) as the outgoing interface selection rule from both site A and site B as the SD-WAN rule. We trust MPLS better so we set the lowest cost for MPLS-based tunnel.

What is strange:
When we tracert a host IP in site A from a host in Site B, it shows 192.168.2.2, the Inernet -based tunnel interface on site A. But the SD-WAN rule is showing that MPLS-based tunnel is the selected outgoing interface on fortigate of Site B. And FortiAnalyzer is showing that MPLS-based interface is the outgoing interface for that tracert traffic.

We asked Orange to check it, and they found a command "set priority-members " should execute to change the selection order of the outgoing interface. They did the change, but tracert is still showing it goes to Internet based tunnel right after the change, so they bring down the Internet-based interface, another tracert showed the correct path it goes to MPLS based tunnel. Then they bring up the Internet-based interface and left.

but, after several minutes, I did the tracert again, everything went back, the tracert shows it went to internet-based tunnel again!

[sean3]

if we issue tracert from a host in Site A to a host in Site B, it shows the desired hop,192.168.1.1, in the path.

[sean3]

Thanks.

but we are not doing load-balance. We set lowest cost to MPLS based tunnel, meaning we need it to be the primary link for traffic forwarding.

But it does not do that for me.

[hbac]

Hi @sean3,

Can you check the static routes by running 'get router info routing-table database'. What IP are you using for tracert?

Regards,

[sean3]

thanks a lot.

since the command presents to much result, I pasted all static route by running get router info routing-table static, is it OK? And for security concern I removed some intranet routing info.

FW-of-Site-A $ get router info routing-table static
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 106.112.x.y, port18, [1/0] #port 18 is our Internet underlay interface
S 10.0.0.0/8 [10/0] is a summary, Null, [1/0]
S 10.250.29.14/32 [5/0] via OL_INET_SKO_112 tunnel 62.20.x.y, [1/0] #10.250.29.14 is the IP of the internet-based tunnel-type interface at Site B(in picture it is 192.168.2.1);

62.20.x.y is the Internet underlay interface IP of Site B. and is the Internet-based IPSec tunnel remote-gateway
S 10.250.29.20/32 [5/0] via OL_MPLS_SKO_212 tunnel 172.29.7.26, [1/0] #10.250.29.20 is the IP of the MPLS-based tunnel-type interface at Site B (in picture it is 192.168.1.1). 172.29.7.26 is the MPLS underlay interface IP, and is the MPLS-based IPSec tunnel remote-gateway IP.
S 172.29.7.24/29 [10/0] via 172.30.4.9, port17, [1/0] #172.29.7.24/29 is the remote MPLS underlay IP range (MPLS interface on FW, and the link to MPLS CE router);172.30.4.9 is just the IP address of the MPLS link from FW to MPLS CE router.

S 172.29.7.27/32 [10/0] via 106.112.142.105, port18, [1/0] #172.29.7.27 is not in use, but it is not deleted.

===

FW-of-Site-B $ get router info routing-table static
Routing table for VRF=0
S* 0.0.0.0/0 [10/0] via 62.20.x.y, Internet, [1/0] # 62.20.x.y is the IP of Internet underlay interface
S 10.250.29.15/32 [5/0] via OL_INET_ZHA_112 tunnel 106.112.x.y, [1/0]

#10.250.29.15 is the IP of the internet-based tunnel-type interface at Site A(in picture it is 192.168.2.2);

106.112.x.y is the Internet underlay interface IP of Site A. and is the Internet-based IPSec tunnel remote-gateway

S 10.250.29.21/32 [5/0] via OL_MPLS_ZHA_212 tunnel 172.30.4.10, [1/0] #10.250.29.21 is the IP of the MPLS-based tunnel-type interface at Site A (in picture it is 192.168.1.2). 172.30.4.10 is the MPLS underlay interface IP, and is the MPLS-based IPSec tunnel remote-gateway IP.

S 172.30.4.8/29 [10/0] via 172.29.7.25, MPLS, [1/0] #172.29.4.8/29 is the remote MPLS underlay IP range (MPLS interface on FW, and the link to MPLS CE router);172.29.7.25 is just the IP address of the MPLS link from FW to MPLS CE router.

==

as you know, static routes basically show all underlay things or the route after the SD-WAN selection and IPsec encapsulation, while our business communication is all from / to the host behind the wall, and the related routes are advertised and learnt by BGP. The BGP peering are based on the tunnel-type interface. in this case:

FW-OF-SITE-A $ get router info routing-table bgp

B 10.250.0.0/17 //the source of the tracert which is residing in Site B

[200/0] via 10.250.29.14 (recursive via OL_INET_SKO_112 tunnel 62.20.x.y), 04:15:29
[200/0] via 10.250.29.20 (recursive via OL_MPLS_SKO_212 tunnel 172.29.7.26), 04:15:29

#OL_INET_SKO_112 andOL_MPLS_SKO_212 are the only two interfaces in SD-WAN outgoing interface selection range.

====

FW-OF-SITE-B $ get router info routing-table bgp

B 10.80.0.0/16  // the target IP range of the tracert which is residing in Site A

[200/0] via 10.250.29.15 (recursive via OL_INET_ZHA_112 tunnel 106.112.x.y), 04:17:45
[200/0] via 10.250.29.21 (recursive via OL_MPLS_ZHA_212 tunnel 172.30.4.10), 04:17:45
[200/0] via 10.250.29.17 [2] (recursive via 172.29.7.17, VLAN600), 04:17:45
[200/0] via 10.250.29.25 [2] (recursive via 172.29.7.17, VLAN600), 04:17:45
[200/0] via 10.250.29.19 [2] (recursive via 172.29.7.17, VLAN600), 04:17:45
[200/0] via 10.250.29.23 [2] (recursive via 172.29.7.17, VLAN600), 04:17:45

#I dont'  know why site B FW learnt 10.80.0.0/16 from 6 sources, but only OL_INET_ZHA_112 and OL_MPLS_ZHA_212 are put in SD-WAN outgoing interface selection range.