7.2 version ADVPN on Hybrid underlay scenarios

[rajamanickam]

Hi,

    We have 1 hub and 2 branches in our setup. All these 3 location has one internet link and one MPLS link (Hybrid underlays) We are having BGP on loopback overlay setup

We have requirement of ADVPN between Spoke 1 to Spoke2, with the help of SDWAN rule we are steering this ADVPN traffic through internet link (And MPLS as second member order) and could see child tunnel (inet_0) formed in Spoke 1 and Spoke2. Traffic originating from Spoke 1 to Spoke2. Everything works fine in normal scenario. We could see that Routes for spoke 2 (192.168.3.0/24) is learning through inet shortcut tunnel and MPLS parent tunnel tunnel in Spoke1. 

When Spoke 2 internet link goes down,  this causes Inet child tunnel goes out of SLA, as per SDWAN rule member order next member MPLS parent tunnel is created and then child tunnel gets formed in MPLS (MPLS_0) at both branches Spoke 1 and spoke2. Until this point, its working fine. Till this point, at Spoke1 - spoke2 route will learn through inet1_0 and MPLS_0 (Child tunnels), once inet1_0 at spoke 1 detects tunnel failure by DPD failure due to spoke 2 inet being down, route for spoke 2 at spoke 1 will learn through inet 1 and MPLS_0. Now traffic from spoke 1 takes inet1 and goes to hub, hub then forwards the traffic to MPLS tunnel to spoke2. This traffic reaches spoke2 , but in spoke 2 we will have MPLS_0 child tunnel only , the parent tunnel at spoke 2 will be masked since we have routes learnt through MPLS_0. This is now causing RPF , since spoke 2 receives this traffic in MPLS but doesnt have return route on the MPLS parent tunnel interface (Return route is only available through MPLS_0 child tunnel). MPLS parent tunnel doesn't have any other routes  (Like default route).

This is causing an outage for the traffic originating from spoke1 to spoke 2.. Is this the expected behavior?.. As per fortinet documentation, traffic should take MPLS from spoke 1, reach Hub and then forward to spoke 2 in MPLS. But as per fortinet document this works and this is being highlighted as limitation where traffic need to be routed in the longest path which is now getting avoided in ADVPN2.0

https://docs.fortinet.com/document/fortigate/7.2.0/sd-wan-sd-branch-architecture-for-mssps/956318/bg...

One option I could think of is to have default route for parent MPLS interface at spoke 2 which could avoid RPF failure. Anyways in hybrid use cases like this (Inet + MPLS) we might need default route in overlay as backup for internet traffic (Remote internet breakout)

Regards

Rajamanickam 

[Jean-Philippe_P]

Hello rajamanickam, 

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible. 

Thanks, 

[Jean-Philippe_P]

Hello,

We are still looking for an answer to your question.

We will come back to you ASAP.

Thanks,

[Jean-Philippe_P]

Hello again Rajamanickam,

I found this article: https://community.fortinet.com/t5/FortiGate/Technical-Tip-Policy-route-on-ADVPN-HUB-with-multiple-ov...

Can you tell me if it can help you?

Thanks!

[rajamanickam]

Hi Jean-Philippe,

  No, this article explains about benefit of having PBR for overlay stickiness

Regards

Raja

[Jean-Philippe_P]

Okay, then I will get to you a staff to help you, thanks for your answer!

[rajamanickam]

okay

[Jean-Philippe_P]

Hello rajamanickam and thanks for your patience!

I spoke with several engineers and the best approach would be to open a ticket over a remote session so we can help you better with this issue :)

Thanks a lot for your patience and do not hesitate if you have anymore questions.

[vikasmarvick]

Hi Rajamanickam,

we are also in a process of implementing ADVPN on two hybrid transports. Internet SDWAN configuration completed - and now will be approaching to MPLS ADVPN - 

I also faced the same issue for the return traffic, when one of the spokes failovers to different transport. to resolve this, PBR can be utilized,  

like, if spoke1, fails over to MPLS from Internet - then you can have PBR on hub, saying, whatever traffic comes from Spoke1 MPLS-ADVPN Interface - should go out with MPLS-ADVPN interface from HUB - 

other way is to use to communities i believe -  

let me know if you already got any solution for same - 

[rajamanickam]

Hi,

  This PBR is for overlay stickiness and initially few packets from spoke1 will use MPLS to form ADVPN with spoke 2 on MPLS link but later once internet ADVPN child tunnel gets flushed (since one side internet is down), traffic on spoke 1 will be shifting to internet link since your sdwan rule has internet as preferred member over MPLS.. Now traffic from Spoke 1 will internet and reach hub, hub then routes that traffic over MPLS to spoke 2 (Since spoke 2 inet is down), now spoke 2 receives this packet and since it dont have spoke 1 branch route learnt through MPLS parent tunnel (Parent tunnel will be masked since we have child tunnel already formed with MPLS), it causes RPF at spoke 2 and packet gets dropped..

Solution I could think of it to avoid RPF is to have default route for all overlay interfaces to avoid this scenario (MPLS and Inet overlay interfaces). Of course in hybrid setup there will be use cases to have central internet breakout as backup solution for local internet breakout... In this scenario you should either have a default static route to overlay zone or have default route advertised from hub through all overlay paths to spoke over BGP..

Another option when you dont want to have default route, is to change the SDWAN rule order to have MPLS to be preferred once we seen inet child tunnel gets flushed. Instead of doing this manually, automation stitch with event logs as trigger can be used to change the sdwan rule order when spoke 2 inet goes down..