Tiered SD-WAN with spokes communicating SD-WAN status to hub

[nathanengelhardtisg]

I'm working on a new design for our SD-WAN and BGP. We're currently on 7.2.10, but I'm planning on attempting an upgrade to 7.4.7 tomorrow.

We have two hubs, a primary (we'll call it PH for Primary Hub) and a backup (we'll call it DR), as well as many branches.

Current configuration

Underlay: There's a metro ethernet (ME) connection between all sites. PH has two DIA connections, and all other sites have a single DIA.

Overlay: We're using the ME directly, no VPN. Each site (besides PH) has two VPN connections between the site and PH, one for each DIA on PH. There's currently no VPN between the branches and DR, but I intend to correct that.

BGP is on the tunnel and ME addresses. All sites besides PH are route reflector clients, and PH is doing the reflecting.

SD-WAN: PH has health checks and rules for each other site. It prefers ME first, VPN-DIA1 second, and VPN-DIA2 third.

It's all very manual and honestly, it's a bit of a mess right now. It's certainly not optimal, and I'm honestly not even sure they were all set up the same way.

Planned configuration

Underlay: stays the same

Overlay: Continue using ME directly for the time being*. Set up 3 ADVPNs** - one for each DIA on PH, and one for DIA on DR.

BGP: A new loopback address is created on each site for BGP purposes. Static routes are added at each site for the loopback addresses over ME. ADVPN exchanges loopback addresses using set exchange-ip-addr4. BGP is on loopback addresses.

SD-WAN: Branches are set up with health checks to a management loopback interface on PH. It uses SLAs on those health checks to consider them in-service/out-of-service. It directs traffic to PH based on that check by preferring ME first, ADVPN-WAN1 second, and ADVPN-WAN2 third. I suppose it could also try ADVPN-DR to get to PH, but if the other 3 fail, then the site is probably just down hard.

The part I'm struggling with

I'd prefer not to have health checks for each site on PH. I want traffic from PH to the branches to be directed with SD-WAN to match the path currently selected by SD-WAN on the branch. So, if the branch's SD-WAN is steering traffic to PH over ethernet to, I want the SD-WAN on PH to steer traffic to the branch over ethernet. If the branch's SD-WAN is steering traffic to PH over ADVPN-WAN1, I want SD-WAN on PH to steer traffic over ADVPN-WAN1. I'm seeing it's possible with a sort of two-tier system using SD-WAN, BGP communities, route maps, route tags, etc. However, it looks like this is only possible in a two-tier setup. The tiers being route-map-out and route-map-out-preferable. Am I understanding this right? Is what I'm wanting to do just not possible? Or am I missing something? It also seems like this might be possible if I do BGP on tunnel/ME addresses instead of on loopback, but I haven't entirely figured out how that would work, and honestly, I'd rather not if I can avoid it.

My alternative plan is to set up health checks and rules on PH for each site and have those rules mirror the rules configured on each site. This should at least give it a highly likely chance of reaching each site the same way the site is reaching the hub. It's just that the number of branches is growing, and it's becoming a very long list of rules. I'd like to avoid that.

I'm also starting to pick up FortiManager. Maybe some of this would be better with VPN manager. I'm hesitant to use that, though, because it seems like it has to be a "replace all the VPNs at once" sort of thing, and I'd like to do it more piecemeal in case something in the current janky configuration breaks. Right now, I'm leaning towards using VPN templates that I made myself. Maybe someone can chime in on that.

Maybe some of this will change, too, with 7.4 and SD-WAN multi-PoP multi-hub large scale design and failover. I'm trying to figure out if that's what I need, but that article is heavy and I'm having trouble digesting it right now.

*Side note: I might look at making this an ADVPN as well. It'd be good to get the loopback addresses exchanged automatically and also encrypt the traffic. The hesitancy is with performance concerns. I'm thinking using IPsec in transit mode might be a good balance. If anyone has thoughts/comments on it, let me know those too.

**Side note: I'm also planning on setting up ADVPN 2.0 - the edge discovery/path management stuff. I just haven't been able to try it out yet because the whole fabric is still on 7.2 until tomorrow.

Thanks in advance to everyone

[nathanengelhardtisg]

Update: I think I found what I'm looking for. I don't know why it was so hard to find with the search terms I was using. I actually just happened to stumble across it.

This is what I think I want: Embedded SD-WAN SLA information in ICMP probes | FortiGate / FortiOS 7.4.7 | Fortinet Document Libra...