ADVPN Dual Hub with BGP on Loopback - Full meshed spokes

[JamesB]

Does anyone have a working cleaned config of ADVPN in a Dual Hub setup with BGP on Loopback and the spokes being full meshed to the Hubs?

I have everything setup and all tunnels are running, but when it comes to the SDWAN SLA's the Hub1 is currently only utilizing 1 overlay. Tags and Policy routing have been applied.

FortiNet doesn't have published configs on this setup (that I can find) due to the complexity however as of 7.0 (running 7.2.8) they do state in their design documents you no longer need the bgp per overlay, but I almost feel like I need to go back to not running BGP on the Loopbacks. 

Basically just seeing if anyone has BGP on loopback up and running when spokes are dual ISP'd with meshed connections to the Hubs.

Super simplified setup:

Spoke1 ISP1 -> Hub1 ISP1 net 1

Spoke1 ISP2 -> Hub1 ISP1 net 1

Spoke1 ISP1 -> Hub1 ISP2 net 2

Spoke1 ISP2 -> Hub1 ISP2 net 2

Spoke1 ISP1 -> Hub2 ISP1 net 3

Spoke1 ISP2 -> Hub2 ISP1 net 3

Hub1 net2 and Hub2 net3 are completely up and operational but hub1 net1 are full down even though debugs on the hub say it is responding to sla probes. So the only thing I can think of is there is something outside the kernel that has the overlays in like a failover type setup.

(hub2 doesn't have tunnels to its secondary ISP)

[spoojary]

There is an example in this doc: https://docs.fortinet.com/document/fortigate/7.4.2/administration-guide/256210/example-sd-wan-config...

https://docs.fortinet.com/document/fortigate/7.4.1/administration-guide/820072/advpn-with-bgp-as-the...

[JamesB]

Those are for 7.4 which has the advpn 2.0 in it, they also go back to utilizing ip addresses on the tunnels, vs sourcing from loopback, the hub also only has 1 wan. My hub with 1 wan works fine, my hub with 2 wans is what is broken. I think I may have hit a design limitation. Appreciate the doc link though, I may need to upgrade.

[JamesB · Accepted solution]

Solution was to split the Hubs into 4 total overlays instead of 2. I believe because the tunnels don't have IPs on them and everything is sourced from a loop-back the hubs were getting confused when a spoke wan1 and a spoke wan2 both arrived on hub wan1 in the same net-id.

[djp]

BGP on Loopback with 7.0 or 7.2 is dangerous and just doesnt work.
There is no logic for SPOKE <> SPOKE best pathing nor HUB > SPOKE best pathing and this is a design limitation that is documented in the Fortinet docs.

However ADVPN 2.0 solved half of this with 7.4.2 release an now more stable in 7.4.5.  The issue of SPOKE <> SPOKE best pathing is resolved with the "health sharing" over the shortcut messaging.  However I still see the HUB > Spoke as an issue.  (if someone has a solution for this, I'd love to hear it)

I currently do NOT use BGP over Loopback because of these limitations.  I still use the traditional method and it is very stable, reliable and pathing is what is expected.  I have a video here with 15 overlays: https://youtu.be/ugGjRPMie-8?si=BOrV2RackjCWLhAE

[GaryJ]

This particular 'glitch' is fixed by adding a static route to each spoke pointing the path to the loopback range. The spoke only knows the path to the hub so tries and fails to peer directly out of the wan interface, hence the spoke-to-spoke fails even though BGP 'superficially' appears right. For consistency, I use a contiguous subnet for loopbacks, for instance 10.x.255.0/24.
The next hop needs to be the overlay Zone (I call mine overlay). So something like:

config router static
edit 200
set dst 10.99.255.0 255.255.255.0
set distance 1
set sdwan-zone "overlay"
next
end

Once added BGP will correctly complete the next hop and spoke to spoke tunnels will build. 
There are numerous advantages (in my opinion) to loopback interfaces in more complex builds but engineer preference still play a part. Having built multiple ways, it simplifies builds where there are multiple WANs and overlays, but there's still a lot of interdependent config that all needs to line up for it to work properly. 

[djp]

The static route is not necessary, and BGP on Loopback works flawlessly with some added features that 7.2 provided (overlay costs)

Here is a video showing all failover scenarios with BGP on Loopback running 7.2 (running ADVPN 1.0)

https://youtu.be/04BjjyMYEEk?si=9V5RC7FHjzH_utVt

[djp]

Yes I do, like this:  https://youtu.be/04BjjyMYEEk?si=MEANd_euD3whuTkD

you need to have different NET-ID's for your overlays so the tunnels match up on both sides