Technical Tip: ISP failure causing ADVPN shortcut traffic to fail

sagha · ‎08-01-2022

Description

This article describes the issue that can be faced with shortcuts when having ADVPN configured with BGP and SD-WAN.

Scope

For versions 6.4, 7.0.

ADVPN, BGP, IPSec.

Explanation

This issue occurs when shortcut is established between the Spokes.

If there is an issue with connectivity, the shortcut fails, and traffic is route via different ISP and a new shortcut is established.

Once the connectivity is restored and a new shortcut is established via preferred path, but traffic is dropped by FortiGate.

The below diagram shows the setup:

- Hub has two ISPs with two overlays.

- Spoke 'FGT2' has two ISPs with two overlays.

- Spoke 'FGT1' has one ISP with one overlay.

1) Spoke 'FGT2' has two ISPs and there is a ADVPN shortcut established between Spoke 'FGT1' and Spoke 'FGT2':

name=ISP-1_0 ver=2 serial=53 10.131.5.189:0->10.1.6.4:0 dst_mtu=1500

bound_if=5 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/728 options[02d8]=npu create_dev no-sysctl rgwy-chg frag-rfc accept_traffic=1 overlay_id=252

parent=ISP-1 index=0

2) Spoke 'FGT'2 can see the route being learned via BGP via ADVPN shortcut.

# get router info routing-table details

B 10.80.80.80/32 [200/0] via 10.255.252.8 [2], ISP-1_0, 00:00:29

[200/0] via 10.255.254.8 [2], ISP-2, 00:00:29

3) On Spoke 'FGT2', traffic is received on the shortcut.

# diagnose sniffer packet any 'host 10.40.60.80 and icmp’ 4 0 a

interfaces=[any]

filters=[host 10.40.60.80 and icmp]

1.321562 ISP-1_0 in 10.80.80.80 -> 10.40.60.80: icmp: echo request

1.321588 ISP-1_0 out 10.40.60.80 -> 10.80.80.80: icmp: echo reply

4) There is a failure to ISP1 which cause the shortcut to fail.

5) This shortcut then establishes on ISP2.

name=ISP-2_0 ver=2 serial=54 10.136.5.189:0->10.1.6.4:0 dst_mtu=1500

bound_if=17 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/728 options[02d8]=npu create_dev no-sysctl rgwy-chg frag-rfc accept_traffic=1 overlay_id=254

parent= ISP-2 index=0

6) Spoke 'FGT2' can see the route being learned via BGP via ADVPN shortcut but this time via ISP2.

B 10.80.80.80/32 [200/0] via 10.255.252.8 (recursive via 10.255.254.1, ISP-2), 00:01:36

[200/0] via 10.255.254.8, ISP-2_0, 00:01:36

7) On Spoke 'FGT2', traffic is received now via new shortcut:

# diagnose sniffer packet any 'host 10.40.60.80 and icmp' 4 0a

interfaces=[any]

filters=[host 10.40.60.80 and icmp]

1.032268 ISP-2_0 in 10.80.80.80 -> 10.40.60.80: icmp: echo request

1.032300 ISP-2_0 out 10.40.60.80 -> 10.80.80.80: icmp: echo reply

8) After the connectivity is restored for ISP1, a new ADVPN shortcut established via preferred path.

name=ISP-1_0 ver=2 serial=56 10.131.5.189:0->10.1.6.4:0 dst_mtu=1500

bound_if=5 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/728 options[02d8]=npu create_dev no-sysctl rgwy-chg frag-rfc accept_traffic=1 overlay_id=252

parent=ISP-1

9) No route is being added to the active routing table via this new shortcut ISP-1_0.

10) Traffic is seen coming inbound on Spoke 'FGT2' via new shortcut but is dropped:

id=20085 trace_id=1001 func=print_pkt_detail line=5810 msg="vd-root:0 received a packet(proto=1, 10.80.80.80:60698->10.40.60.80:2048) from ISP-1_0. type=8, code=0, id=60698, seq=633."

id=20085 trace_id=1001 func=resolve_ip_tuple_fast line=5891 msg="Find an existing session, id-000bbd5c, original direction"

id=20085 trace_id=1001 func=ip_route_input_slow line=2264 msg="reverse path check fail, drop"

id=20085 trace_id=1001 func=ip_session_handle_no_dst line=6065 msg="trace"

11) Routing table shows routes only via ISP2 shortcut:

B 10.80.80.80/32 [200/0] via 10.255.252.8 [2], ISP-2_0, 00:01:34

Solution

1) This is something that would not work when using version 6.4 or prior version on FortiGate as this is by design.

2) There is a new feature added in version 7.0 onwards that changes the design.