| Description |
The article describes how to minimize the failover downtime when the traffic switch from one ADVPN tunnel to another using SD-WAN rules.
Since by the time of this writing, SD-WAN has not supported load-balancing the traffic between 2 ADVPN members, it means that the maximum bandwidth strategy will not work as the Hub will only pick the first ADVPN tunnel to offer the shortcut. Therefore, in a normal working scenario, only one shortcut established for traffic with the same source and destination will be visible.
Any attempts to failover to the secondary link do not guarantee that the traffic will flow through the secondary shortcuts instantly. |
| Scope | FortiGate v7.0 and above. |
| Solution |
For the ADVPN setup, it is recommended to go through the IPsec wizard to have FortiGate automatically configure BGP, firewall policies, and tunnel address on both Hub and Spoke. Especially for Spoke, there is the benefit of copying and pasting the quick configuration from the Hub to minimize any configuration error.
With basic ADVPN and BGP now set up, the attention is shifted to each device in the setup:
On the Spoke: Configure SD-WAN zones 'Overlay' and optionally, 2 WAN interfaces to the 'Underlay' (or the default 'virtual-wan-link') SD-WAN zone.
Configure performance SLA with the two ADVPN tunnels as the member to ping the internal IP address of the Hub (Loopback interface is recommended).
Configure the SD-WAN rule(s) as needed to force any expected to reach the remote side through the ADVPN tunnel. Be aware that as of this writing, load-balancing the traffic between ADVPN members has not been supported. Therefore, even though the GUI and CLI let configure 'maximize bandwidth' as the SLA strategy, the Hub will still only pick one ADVPN tunnel to route the traffic with the same source and destination instead of utilizing both ADVPN tunnels to trigger the shortcut even though the GUI will show that both tunnels are being utilized.
On the Spoke’s ADVPN tunnel phase 1 setting, lower the DPD retry count and DPD interval to 1 to increase the sensitivity for the failover to minimize any downtime due to failover
For phase2, enable 'Auto-Negotiate' option:
In the Routing Object setting, configure 2 route maps: for each route map, configure to have the internal subnet of the spoke tagged with a different community ID for each ADVPN tunnel to make sure that outgoing traffic will go in the correct tunnel’s shortcut instead of routed to the first available next-hop by the Hub. Without this option, traffic will be routed to the first available next hop advertised by the Hub. In the scenario of the primary tunnel going down, the traffic will keep being routed to the same next-hop instead using the secondary tunnel IP of the Spoke as the next-hop. The community tagging must be consistent on all Spoke and Hub for each ADVPN tunnel
Switching to BGP’s settings, expand the Best Path Selection setting at the bottom and enable the following settings: IBGP multipath, Additional path alongside default settings. Enabling these options will insert all BGP routes as the best routes in the active routing table, instead of letting the Spoke (or even the Hub) determine the best route to the same destination based on the BGP route selection process. Leave others checked by default.
Scroll up to the Neighbors setting, for each BGP neighbor, which is the ADVPN tunnel IP of the Hub, set the Interface to the correct ADVPN tunnel, enable Route map out, and select the route map with the correct community tag. It should be consistent across all Hub and Spoke’s configurations or else the BGP advertisement will not be working correctly.
In the same window, check the following option: Capability: graceful restart; Soft reconfiguration.
Open the CLI and configure a few additional settings that can’t be done in the GUI:
config system sdwan config members edit 1 set interface "Spoke-1" set zone "Overlay" set source <internal-or-loopback-IP-to-tunnel> <----- To force the traffic going through the tunnel. next edit 2 set interface "Spoke-2" set zone "Overlay" set source <internal-or-loopback-IP-to-tunnel> <----- To force the traffic going through the tunnel. end end
config vpn ipsec phase1-interface edit <phase1-name> set idle-timeout enable set idle-timeoutinterval 5 set auto-discovery-shortcuts dependent end
These 3 commands above are to make sure the shortcut will be brought down when the parent tunnel is down for 5 minutes (minimum value).
config router bgp set recursive-next-hop enable <----- To force the recursive resolution to the right next hop. set tag-resolve-mode preferred <----- Resolve the BGP route’s next hops with the one having the same tag. config neighbor edit <neighbor-IP> set advertisement-interval 1 <----- To achieve the fastest route failovers possible. set link-down-failover enable <----- To trigger full failover to another tunnel over different WAN. set additional-path receive next end end
After applying all the changes needed, execute the command to restart the router software, since community tagging needs the restart of the router software to take effect:
execute router restart
On the Hub: Since the Hub is the transit point, it is not necessary to configure SD-WAN on this unit. However, to make sure that the BGP next-hop advertisement to each Spoke is working properly, which is to have the tunnel IP for the next hop to be in the correct ADVPN tunnel to prevent traffic keeps taking one ADVPN tunnel even if the tunnel goes down, the following setting must be configured: In the Routing Object configuration, create 2 community lists corresponding to 2 ADVPN tunnels respectively. It must be the same as the one created on the Spoke:
Create 2 route maps containing 2 policies: one to accept the Community list object, and one to tag the correct community ID for the internal subnet of the Hub. Make sure to use the same community tag with the Spoke for each ADVPN tunnel.
On the BGP settings, under the 'Best Path Selection' section, enable IBGP multi-path and Additional path to make all learned BGP path the best path. Leave others checked by default.
On each BGP neighbor (or neighbor group), enable the following option: Route reflector client, Soft reconfiguration, Capability: route refresh, and Capability: graceful restart, and select the right route map corresponding to each ADVPN tunnel IP subnet.
Navigate to the policy route and create 2 policy routes to ensure tunnel traffic will not be load-balanced due to ECMP:
Open the CLI and configure a few additional settings that can’t be done in the GUI:
config router bgp set recursive-next-hop enable config neighbor-group edit "HUB1" set link-down-failover enable set additional-path both end end
After applying all the changes needed, execute the command to restart the router software, since community tagging needs the restart of the router software to take effect:
execute router restart
Related documents: https://community.fortinet.com/t5/FortiGate/Technical-Tip-BGP-route-selection-process/ta-p/195932 |
