Technical Tip: Practical use case of ADVPN overlay stickiness to resolve the shortcut tunnel failover issue in SD-WAN rule

The following ADVPN hub-and-spoke topology is used as an example. Each Spoke is connected to the Hub via two IPsec tunnels, with BGP established on each tunnel.

After BGP peering is established, each spoke should have two BGP routes to the other spoke and the hub. Each route is via a different IPsec tunnel.

For example, on the Spoke-1:

The SD-WAN rule is configured on each spoke to prefer the primary IPsec tunnel.

The ADVPN shortcut tunnel will be established when traffic flows between Spoke-1 and Spoke-2. Since the primary tunnel is preferred by the SD-WAN rule, a shortcut tunnel is created over the primary tunnel.

In the routing table, the primary VPN route (Spoke-1) to Spoke-2 will be replaced by this new shortcut tunnel (Spoke1_0).

If the primary tunnel from Spoke-1 goes down due to an underlay interface outage, the shortcut tunnel (Spoke1_0) will disappear. Subsequently, the corresponding route will be removed from the routing table after BGP re-convergence.

The new ADVPN shortcut will soon be established again when there is any traffic between spoke-1 and spoke-2. This time, it is established in the secondary tunnel.

The routing table looks like this. In this case, the traffic from spoke-1 to spoke-2 is going via this new shortcut tunnel.

However, when the primary tunnel is restored, traffic does not fail back to it, even though the SD-WAN rule prefers Spoke-1. According to the debug flow, the traffic matches the SD-WAN rule, but the outgoing interface remains the shortcut tunnel 'Spoke_sec_0' established on the secondary tunnel.

c3po-kvm47 # id=65308 trace_id=61 func=print_pkt_detail line=5862 msg="vd-root:0 received a packet(proto=1, 10.2.0.214:1->10.24.0.209:2048) tun_id=0.0.0.0 from port2. type=8, code=0, id=1, seq=670."

id=65308 trace_id=61 func=init_ip_session_common line=6047 msg="allocate a new session-002698a2"

id=65308 trace_id=61 func=rpdb_srv_match_input line=1040 msg="Match policy routing id=2131361793: to 10.24.0.209 via ifindex-45"

id=65308 trace_id=61 func=__vf_ip_route_input_rcu line=1990 msg="find a route: flag=00000000 gw-10.10.1.4 via Spoke_sec_0"

By looking at SD-WAN members, all tunnels are alive, and the primary tunnel 'Spoke 1'  is at first rank.

c3po-kvm47 # diagnose sys sdwan service

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla

 Tie break: cfg

  Gen(2), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(manual)

  Member sub interface(3):

    3: seq_num(1), interface(Spoke_sec):

       1: Spoke_sec_0(45)

  Members(3):

    1: Seq_num(2 Spoke1), alive, selected

    2: Seq_num(1 Spoke_sec_0), alive, selected

    3: Seq_num(1 Spoke_sec), alive, selected

The reason traffic fails to fail back to the primary tunnel is due to the routing table. Although the primary tunnel (Spoke 1) is up, it is not selected as the recursive route to the next-hop (10.10.1.4), while the shortcut tunnel (Spoke_sec_0) remains active. This occurs because the shortcut tunnel to the other spoke is a directly connected route, which has a lower administrative distance than the primary tunnel.

In this case, the route to the Spoke-2 is not changed even when the primary tunnel is up.

According to SD-WAN principles, an SD-WAN member is selected only if it has a valid route to the destination in the FIB. In this case, the primary tunnel (Spoke-1) is not selected because it lacks a route to the other spoke. As a result, the SD-WAN rule moves to the next member, which is the shortcut tunnel. This explains why traffic to Spoke-2 continues to route through the shortcut tunnel until it is manually flushed.

The root cause of this issue is due to the next-hop IP in the BGP route.

By at the routing-table after failover, the next hop to spoke-2 is still 10.10.1.4.

Before failover:

After failover:

10.10.1.4 is the primary tunnel interface IP on Spoke-2, meaning the shortcut tunnel is established between the secondary tunnel on Spoke-1 and the primary tunnel on Spoke-2. This occurs because the primary tunnel on Spoke-2 is still up, making it preferred by the hub when forwarding the first packet.

To avoid this situation, one possible solution is to configure the overlay stickiness on the hub. The introduction of overlay stickiness can be found in the document below:

Technical Tip: Usage of overlay stickiness in multiple overlay links in an ADVPN/SD-WAN setup

When a policy route is configured on the hub to enforce overlay stickiness, traffic from the primary tunnel of Spoke-1 will only be forwarded to the primary tunnel of Spoke-2, and the same applies to the secondary tunnel. In this case, the shortcut tunnel will be established over the secondary tunnels on both sides. After failover, the routing table will reflect the change, with the next-hop IP pointing to the secondary tunnel on Spoke-2.

When the primary tunnel is restored, traffic will be forwarded back to the primary tunnel, as the route to it will reappear.

Then, the shortcut tunnel on the primary tunnel will be re-established. After that, the traffic to the other spoke is forwarded into this shortcut tunnel again.

Note: 'For Policy Route to work, there should be a matching route in the routing table'

get router info routing-table all  --> Check matching route.

Related articles:

Technical Tip: SD-WAN application in ADVPN failover setup: One Hub, 2 ISPs

Technical Tip: Usage of ‘auto-discovery-crossover’ option in ADVPN setup

Technical Tip: Usage of overlay stickiness in multiple overlay links in an ADVPN/SD-WAN setup