Technical Tip: How overlay static route effect performance of FortiGate with Mesh IPSEC, SD-WAN and BGP

[sfernando]

Description	This article discusses a common mistake made by some users when setting up an environment that has IPSEC, SD-WAN, and BGP, how they are affected during the production network, and what alternatives can be taken.
Scope	IPSEC (ADVPN), SD-WAN, BGP.
Solution	In a setup that includes IPSEC or ADVPN, SD-WAN, and  BGP, underlays are used with default routes and overlays are used for more specific routes to reach the spoke or HUB use BGP. It was observed some users intentionally or mistakenly added the overlay to the default routes as below: FG01 (1) # show     config router static         edit 1             set distance 1             set sdwan-zone "virtual-wan-link" "Underlay" "Overlay"         next     end For overlay traffic to reach HUB or Spoke, BGP is been configured in the same network as below.This setup has 3 WAN links and each has overlays created on top of them. Hence the IPsec tunnels look as follows: config vpn ipsec phase1-interface     edit "OLDC1_Inet"         set interface "port1"         set ike-version 2     next         edit "OLDC2_Inet"             set interface "port2"             set ike-version 2         next             edit "OLDC1_BIP"                 set interface "port3"                  set ike-version 2 BGP configuration looks as follows: config router bgp     set as 65010     set router-id 10.100.254.1     set ibgp-multipath enable     set network-import-check disable     set additional-path enable     set additional-path-select 8         config neighbor             edit "172.20.2.1"                 set advertisement-interval 1                 set link-down-failover enable             next                edit "172.20.3.1"                     set advertisement-interval 1                     set link-down-failover enable                 next                     edit "172.20.6.1"                         set advertisement-interval 1                         set link-down-failover enable Note: These are partial configurations for IPsec and BGP, as they are very well known. Based on routing theory static routes take precedence over BGP routes due to the lower AD distance of static routes. The static routes configured are default routes and BGP routes are more specific routes. Due to the fact that BGP routes are more specific (or have a longer prefix), BGP routes take precedence in the routing table. Traffic will work as usual until something happens on the IPsec tunnel, while IPsec tunnels are up but affecting traffic only, Eg. ESP errors, and port 500/4500 blocks some examples are where the IPsec tunnels are up but the traffic via the tunnel will get affected. This will affect BGP traffic via the IPSEC tunnel making BGP down or flapping of the neighbor ship. This will also result in the routing table selecting the static routes instead of the BGP routes due to the unavailability of BGP routes. The behavior of SD-WAN is due to the above. SD-WAN assumes an SD-WAN rule is set up as manual, where the most preferred member is selected irrespective of any other condition. In SD-WAN, it is necessary to have a valid route to the destination. This behavior can be changed using ' set default' and 'set gateway' commands: Technical Tip: Understanding Default and Gateway Parameters in SD-WAN Example configuration: edit 1     set name "LAN"     set dst "DC_Networks"      set src "LAN"     set priority-members 5 3 6  Based on the above config let's assume the members are as follows: 3 - IPSEC-1 5 - IPSEC-2 6 - IPSEC-3 As explained previously let's assume static route and BGP are both configured for all the members shown.  Traffic can reach the destination over all 3 IPsec tunnels and all 3 have static routes and BGP routes. Due to the more specific routes of BGP, BGP routes will be in the routing table. Due to the SD-WAN rule mentioned above member 5 (IPSEC-2) will be selected to route traffic. When member 5 (IPSEC-2) encounters ESP errors the BGP neighborship goes down resulting in routes from BGP being dropped also. But due to the existence of a static route, the route cache will be updated using default routes. Due to this update in the route cache the SD-WAN thinks there is a valid route to the destination via the BGP down link which is member 5 (IPSEC-2). This results in FortiGate trying to send traffic using member 5 (IPSEC-2), hence disrupting the traffic.  But in the routing table, it is observed that the BGP downlink 5 (IPSEC-2) is no more in the routing table, But there is a routing entry from the static route in the route cache, which informs the SD-WAN that member 5 (IPSEC-2) has a valid route. This results in traffic not traversing over the other IPSEC tunnels but trying to go via the BGP down the tunnel. Solution: As mentioned previously these static routes (default routes) for the overlay have been implemented for various reasons.  It is recommended to remove the unwanted overlay static routes. Add a health check for overlay links. This will help to identify whether the links can carry traffic even if the links are up and there is some sort of traffic issue as mentioned previously. Enabling a Health check on overlay tunnels will inform the SD-WAN that the link can not carry traffic even though the link is up and has a valid route. Accordingly, SD-WAN will not select the aforesaid Overlay link, even though it is the most preferred link as per the SD-WAN rule. This will result in SD-WAN selecting the next member to send traffic instead of the BGP down the tunnel.