how fortigate SD-WAN ensure both sites are in sync for its active link/member interface?

[sean3]

Dear all, 

Thanks for your attention on this post. The figure below is the topology we are talking about.

we have 4 sites (A/B/C/D as shown) across the globe. There are Internet underlay and MPLS underlay between each two sites. We implemented the SD-WAN based on the overlay IPSec tunnel interface between each two sites (MPLS-based IPSec tunnel and Internet-based IPSec tunne as the member interface of SD-WAN Zone). There are tunnel-type interface under the underlay physical interface and BGP is peering based on the tunnel-type interface itself.

We think MPLS is more reliable, so we assigned lower cost to MPLS-based IPSec tunnel interface in SD-WAN zone/interface, and then use PING test as the Performance SLA, so the outgoing interface selection rule is Lowest Cost (SLA).  All the 4 sites are using its MPLS-based tunnel interface as the outgoing SD-WAN interface for the time being.

Question: let's say, if MPLS-based ipsec tunnel interface at site A has been being the selected outgoing interface, and for some reason it fails the SLA all of a sudden (let's say, high latency or packets drop), so the Internet-based IPsec tunnel interface will be used as the selected outgoing interface if it still meets the SLA. But all overlay interface and SLA are working well as usual at other sites. Outgoing traffic from site A will switch to use its Internet-based IPSec tunnel interface to reach other sites, will the other site, let's say site D, sends back the traffic via D's Internet-based IPSec tunnel accordingly? I think there should be some rule or algorithm to ensure the traffic is going through the same overlay tunnel, how it works?

[jollyre4]

Yes, and if you have multiple ISPs on the HQ and in the branch, you can establish multiple tunnels and put them in an SD-WAn zone. With that, you have automatic and seamless failover if you configure your SD-WAN rules to use those tunnels and you can also use the performance SLA to choose what VPN tunnel you will use for the traffic https://mobdro.bio/ .

[sean3]

thanks Jolly for your reply.

My question is, I have multiple sd-wan member interface across sites, what if just one site has its one of sd-wan member interfaces failing its SLA but every other site and their SD-WAN member interface are all working as well as usual, will other sites recognize the failure occurring to that specific site and do the outgoing interface re-selection accordingly so the traffic will not be going to a blackhole?

[funkylicious]

The other sites by default will not know that the MPLS link is no longer preferred on site A, so when a return packet from site D comes with will most likely use it instead of the Interned based one and it might be dropped.

For controlling and signaling this SLA failure, you can have a look at , https://docs.fortinet.com/document/fortigate/7.0.0/sd-wan-self-healing-with-bgp/559415/overview 

[ezhupa]

Hello, 

In a hub and spoke topology we can use ICMP probes to the hub with embedded SLA performance information so the hub uses the same overlay the spoke is using to avoid asymetric routing.  In your scenario you don't have a specific HUB as far as I could see but it can be possible to implement the same logic. 
You can refer to the below documentation:
https://docs.fortinet.com/document/fortigate/7.4.3/administration-guide/848259/embedded-sd-wan-sla-i...

Hope this helps!

[sean3]

thanks a lot!

actually asymetric routing will not drop the traffic packet, right? it just might receive traffic from an interface failing its SLA, and sends out traffic via the other overlay interface meets the SLA, which looks like a circle, I think?