DUAL HUB SETUP FOR ADVPN and SDWAN FOR BRANCH OFFICES

[TT_DU]

Hi All,

I am planning to setup the below. Using FGT 7.2.4, If you see any bug also please highlight.

Spoke 1 & 2  -------> Hub 1 & Spoke 1 & 2  -------> Hub 2 (Using ADVPN -iBGP)

-------------------

Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)

Spoke 1 & Spoke 2 need to be communicated with each other.

Hub 1 and Hub 2 need to be communicated.

All the tunnel interfaces (From Branch to hub 1 and hub 2) in the same Overlay Zone.

--------------------------------

To achieve the above setup using the below protocols.

Between Spoke 1 & 2 ----> Hub 1 & 2 using iBGP (all Spoke and hub locations in the same iBGP AS number)

Between Hub 1 & 2 also using eBGP (or may be iBGP).

------------------------------------

Do you see any challenges on the above setup?

If you have any suggestion then please let me know.

[TT_DU]

Also one more query, If we try to config the basic policy routes on HUB to have overlay stickiness, then the below scenarios can be achieved?

Because MY understanding is end to end overlay stickiness will use the same Overlay tunnel.

SPOKE1 -OL2 down & SPOKE2-OL3 down ----> SPOKE1-OL3 is able to communicate with SPOKE2-OL2

SPOKE1 -OL3 down & SPOKE2-OL2 down ----> SPOKE1-OL2 is able to communicate with SPOKE2-OL3

[akristof]

Hello,

Yes, it will. Because policy-route will have effect only if route via that overlay is available. So if you have policy-route from OL1 to OL1, but OL1 on spoke2 is down, HUB will use other overlay, OL2 or OL3. Which, depends on further load-balancing or your backup policy-routes to have "deterministic" routing

[TT_DU]

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnet

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop'

[TT_DU]

Thanks! I will check.

I mentioned the debug outputs below. please suggest.

[TT_DU]

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnets

 SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

 HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop

[TT_DU]

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

---------------------------------------------------

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnet

---------------------------------------------------------------------------------------

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"

[TT_DU]

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnets

 SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

 HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop

@akristof @knagaraju 

[akristof]

reverse path check fail, drop - so routing on spoke2 is incorrect. You are using iBGP correct?

Share with me, in files - you can attach them, the following - from all 3 devices:

show router bgp

show vpn ipsec phase1-interface

show system interface

diag ip address list

get router info routing-table all

[TT_DU]

Even after enable the ibgpmultipath command , the issue is looks to be the same. It's still getting reverse path check fail error. Please suggest.

[TT_DU]

Can someone help to mitigate this issue.

anyone having any idea?