| Solution |
In a scenario with a Fortinet SD-WAN Hub and Spokes deployment, with more than one network connection on each Spoke and the BGP configured on loopback (as explained there: BGP on loopback), a scenario that may occur is: intermittent connection between LAN clients and services reachable via the Hub, with one network connection degraded.
In that scenario, if the connection degraded is monitored with a the SD-WAN PerformanceSLA, referenced with at least one SD-WAN rule, the traffic passing trough the Spoke should be steered via the network connection with PerformanceSLA measurements in SLA, but BGP flaps are observed.
Probably this happen because is not possible to influence local BGP traffic (TCP traffic on port 179) with an SD-WAN rule, so those packets are sent over the connection degraded, following per routing table.
Is possible to verify it with a traffic capture on TCP port 179, as explained here: Using the packet capture tool
How to select the order for outgoing local BGP traffic:
- If it has been activated, disable the add-route option under the IPSec phase1 configuration (it is disabled by default). More info regarding the option, here: Dynamic IPsec route control
-
In each SD-WAN Spoke, manually configure a static route for each interface through which the local BGP traffic may be sent to the specific Hub, and for each Hub to which the Spoke is connected. These static routes should have different priorities (assuming that the Administrative Distance is the same), with the lowest configured in the one using the interface that must be used by the local BGP traffic, until the interface is UP. The second lowest for the second preferred one, and so on.
Static routes example:
config router static
edit x <- To customize.
set dst x.x.x.x 255.255.255.255 <- x.x.x.x IP of the Hub’s loopback interface configured as a BGP peer in the Spokes.
set device VPN_1 < --- Interface that the BGP local traffic should use until UP is.
next
edit y <- To customize.
set dst x.x.x.x 255.255.255.255
set priority 2
set device VPN_2 <--- Interface that the BGP local traffic should use until UP is, if the VPN_1 interface is down.
next
end
Notes:
- The BGP traffic passing through FortiGates, for example, the traffic to manage the BGP neighborhood between a couple of routers connected to the LAN ports of Spoke and the Hub, is not affected.
- In the best static route selection, the Administrative Distance takes precedence, more info here: Routing concepts.
- If to choose the best static routes are changed the Administrative Distance (10 is the default value), it is necessary to pay attention to the routing table interactions with routes eventually added from other routing protocols.
- The configuration change described affects BGP traffic only from the Spoke to the Hub, not the response of the Hub.
- The local BGP traffic will continue to use the same interface until it is UP. If it is monitored with an SD-WAN Performance SLA and SLA/SLAs configured for it, they are not respected. It does not depend on the static routes configured; these are useful to decide which interface must be used for local BGP traffic.
- The addition of the static routes indicated above can trigger some routing table changes also for BGP routes, because the static route takes precedence in the next-hop resolution mechanism, so the next-hop of BGP routes received from each Hub with a static route pointing to the loopback interface used for the BGP can change. As explained here: BGP recursive resolution not possible via another BGP route, it is possible to use a prefix list and a router map to overlap the next hop.
- After the static routes suggested are configured, in the FortiGate acting as SD-WAN Spoke of a specific Branch, in case of packet loss on the interface of the static route selected from the FortiGate, the branch’s users may experience intermittent connectivity, and the static routes need to be updated manually. For this reason is convenient to apply the configuration suggested in this article, in case of consistent performance difference between the Branch underlay connections.
Related document:
Local traffic manageable with SD-WAN: Local out traffic.
BGP Troubleshooting guide: Troubleshooting BGP
|
