DUAL HUB SETUP FOR ADVPN and SDWAN FOR BRANCH OFFICES

[TT_DU]

Hi All,

I am planning to setup the below. Using FGT 7.2.4, If you see any bug also please highlight.

Spoke 1 & 2  -------> Hub 1 & Spoke 1 & 2  -------> Hub 2 (Using ADVPN -iBGP)

-------------------

Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)

Spoke 1 & Spoke 2 need to be communicated with each other.

Hub 1 and Hub 2 need to be communicated.

All the tunnel interfaces (From Branch to hub 1 and hub 2) in the same Overlay Zone.

--------------------------------

To achieve the above setup using the below protocols.

Between Spoke 1 & 2 ----> Hub 1 & 2 using iBGP (all Spoke and hub locations in the same iBGP AS number)

Between Hub 1 & 2 also using eBGP (or may be iBGP).

------------------------------------

Do you see any challenges on the above setup?

If you have any suggestion then please let me know.

[TT_DU]

any update?

Anyone knows about cross tunneling concept?

[akristof]

Yes, auto-discovery-forwarder is to allow forwarding advpn control messages between hubs. I am not sure what you mean by cross-tunneling concept. If you mean cross-overlay shortcuts (shortcuts between different tunnels) this works, if you have same access, e.g. internet. If you have internet and mpls, you will have problems. But it is always better to keep shortcuts on same overlay whenever it is possible and allow cross-overlay shortcuts only when primary overlay is not available.

[TT_DU]

Thanks!! I have Internet only and not MPLS.

Any documents available in Foritnet to achieve the same?

[knagaraju]

Yes. I am aware of cross tunneling. Please let me know your query.

[TT_DU]

1. SPOKE 1 OL2 is down now.
2. SPOKE to SPOKE communication should happen via SPOKE1- OL3 to SPOKE2 -OL2 , But this is not happening.

To test the scenario, I manually bring down the OL2 tunnel but WAN 1 is still up, due to that its getting recursive route via WAN link, to resolve this one created static route pointing to OL3 tunnel. Now routing table updated with correct info in SPOKE 1, also created OL2 to OL3 and vice versa policy in Hub device.

 Now, the communication happening like, SPOKE1 OL3-->HUB-OL3-->OL2--->SPOKE2 OL2 , But SPOKE2 is learning the SPOKE1 LAN subnet via OL3 link using iBGP, so its trying to reply via OL3. due to this there is no reply from SPOKE2 to SPOKE1.
SPOKE shows the below error message.

 SPOKE2 # id=65308 trace_id=1 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:27655->10.102.5.1:2
048) tun_id=20.233.48.101 from INET_OL2. type=8, code=0, id=27655, seq=0."
id=65308 trace_id=1 func=init_ip_session_common line=6049 msg="allocate a new session-011599d0, tun_id=20.233.48.101"
id=65308 trace_id=1 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"

[TT_DU]

Above is the diagram.

[TT_DU]

SPOKE 1 OL2 is down now.

SPOKE1 to SPOKE2 communication should happen via SPOKE1- OL3 to SPOKE2-OL2 , But this is not happening.

Just want to know what are all configurations should be done on the Hub and Spoke sites to achieve the same.

I tried from my end but its not working.

[TT_DU]

I am able to achieve the below scenarios with above setup mentioned in the diagram.

-----------------------Scenario1 --------------------------------------

SPOKE1 -OL2 down  & SPOKE2-OL3 down ----> SPOKE1-OL3 is able to communicate with SPOKE2-OL2

SPOKE1 -OL3 down  & SPOKE2-OL2 down ----> SPOKE1-OL2 is able to communicate with SPOKE2-OL3

------------------------------------Scenario2------------------------

SPOKE1 -OL2 down  & SPOKE2-OL2 & OL3 are UP ----> SPOKE1-OL3 is not able to communicate with SPOKE2-OL2 or OL3

Can anyone having any idea?

Please suggest what could be the issue, also share the config suggestion.

[akristof]

Hi,

First, verify routing. On each Spoke:

- verify that they have routing-table entry for all remaining overlays. Also, spoke1 should have route for OL2 subnet, when its own OL2 is down. It can be static route via OL1 and OL3 as a backup.

- On HUB verify that you have all routes via all available overlays

- Check how Spokes are sending traffic. Which overlay they are selecting? If traffic is leaving OL3, check if HUB is forwarding traffic also over OL3 (you should have basic policy routes on HUB to have overlay stickiness). I would recommend to enable debug flow on all 3 devices at the same time, send 1 ping and you will see where is the problem.

[TT_DU]

I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.

Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?

Source: 10.101.6.1 - spoke1 subnet

Destination: 10.102.5.1 - spoke 2 subnet

SPOKE1:

2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"

HUB:

2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"

SPOKE2:

2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request

id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=x.x.x.x from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=x.x.x.x"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"