Hi All,
I am planning to setup the below. Using FGT 7.2.4, If you see any bug also please highlight.
Spoke 1 & 2 -------> Hub 1 & Spoke 1 & 2 -------> Hub 2 (Using ADVPN -iBGP)
-------------------
Spoke1 & 2 need to be connected with Hub 1 and Hub2 (Both hubs are running with separate services)
Spoke 1 & Spoke 2 need to be communicated with each other.
Hub 1 and Hub 2 need to be communicated.
All the tunnel interfaces (From Branch to hub 1 and hub 2) in the same Overlay Zone.
--------------------------------
To achieve the above setup using the below protocols.
Between Spoke 1 & 2 ----> Hub 1 & 2 using iBGP (all Spoke and hub locations in the same iBGP AS number)
Between Hub 1 & 2 also using eBGP (or may be iBGP).
------------------------------------
Do you see any challenges on the above setup?
If you have any suggestion then please let me know.
Nominating a forum post submits a request to create a new Knowledge Article based on the forum post topic. Please ensure your nomination includes a solution within the reply.
Also one more query, If we try to config the basic policy routes on HUB to have overlay stickiness, then the below scenarios can be achieved?
Because MY understanding is end to end overlay stickiness will use the same Overlay tunnel.
SPOKE1 -OL2 down & SPOKE2-OL3 down ----> SPOKE1-OL3 is able to communicate with SPOKE2-OL2
SPOKE1 -OL3 down & SPOKE2-OL2 down ----> SPOKE1-OL2 is able to communicate with SPOKE2-OL3
Hello,
Yes, it will. Because policy-route will have effect only if route via that overlay is available. So if you have policy-route from OL1 to OL1, but OL1 on spoke2 is down, HUB will use other overlay, OL2 or OL3. Which, depends on further load-balancing or your backup policy-routes to have "deterministic" routing
I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.
Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?
Source: 10.101.6.1 - spoke1 subnet
Destination: 10.102.5.1 - spoke 2 subnet
SPOKE1:
2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"
HUB:
2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"
SPOKE2:
2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop'
Thanks! I will check.
I mentioned the debug outputs below. please suggest.
I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.
Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?
Source: 10.101.6.1 - spoke1 subnet
Destination: 10.102.5.1 - spoke 2 subnets
SPOKE1:
2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"
HUB:
2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo
id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"
SPOKE2:
2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop
I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.
Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?
---------------------------------------------------
Source: 10.101.6.1 - spoke1 subnet
Destination: 10.102.5.1 - spoke 2 subnet
---------------------------------------------------------------------------------------
SPOKE1:
2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:08.449063 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:09.459066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"
HUB:
2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:30.485620 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"
SPOKE2:
2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:01.079959 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:02.319901 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop"
I could see the below flow, no reply from SPOKE 2 to HUB because its learning the SPOKE1 subnet via OL3.
Please suggest what could the issue. Should I create basic policy routes on HUB to have overlay stickiness?
Source: 10.101.6.1 - spoke1 subnet
Destination: 10.102.5.1 - spoke 2 subnets
SPOKE1:
2023-05-31 14:35:04.393600 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:05.409061 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:06.419088 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:35:07.439066 INET_OL3_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=562 func=print_pkt_detail line=5868 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=0.0.0.0 from local. type=8, code=0, id=38408, seq=10."
id=65308 trace_id=562 func=resolve_ip_tuple_fast line=5956 msg="Find an existing session, id-00244e1d, original direction"
id=65308 trace_id=562 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface INET_OL3_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=562 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel INET_OL3_AZ_UAE vrf 0"
HUB:
2023-05-31 14:39:28.585619 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:28.585676 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545577 INET_OL3_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:29.545627 INET_OL2_AZ_UAE out 10.101.6.1 -> 10.102.5.1: icmp: echo
id=65308 trace_id=545 func=print_pkt_detail line=5939 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_i
d=10.12.14.4 from INET_OL3_AZ_UAE. type=8, code=0, id=38408, seq=4."
id=65308 trace_id=545 func=resolve_ip_tuple_fast line=6027 msg="Find an existing session, id-038d1662, original direction"
id=65308 trace_id=545 func=npu_handle_session44 line=1245 msg="Trying to offloading session from INET_OL3_AZ_UAE to INET_OL2_AZ_UAE, skb
.npu_flag=00000400 ses.state=04010204 ses.npu_state=0x00000100"
id=65308 trace_id=545 func=fw_forward_dirty_handler line=416 msg="state=04010204, state2=00000001, npu_state=00000100"
id=65308 trace_id=545 func=ipsecdev_hard_start_xmit line=662 msg="enter IPSec interface INET_OL2_AZ_UAE, tun_id=0.0.0.0"
id=65308 trace_id=545 func=_do_ipsecdev_hard_start_xmit line=222 msg="output to IPSec tunnel INET_OL2_AZ_UAE_14 vrf 0"
SPOKE2:
2023-05-31 14:38:58.159998 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:38:59.120018 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
2023-05-31 14:39:00.060323 INET_OL2_AZ_UAE in 10.101.6.1 -> 10.102.5.1: icmp: echo request
id=65308 trace_id=1112 func=print_pkt_detail line=5875 msg="vd-root:0 received a packet(proto=1, 10.101.6.1:38408->10.102.5.1:2048) tun_
id=20.233.48.101 from INET_OL2_AZ_UAE. type=8, code=0, id=38408, seq=6."
id=65308 trace_id=1112 func=init_ip_session_common line=6049 msg="allocate a new session-012bb40a, tun_id=20.233.48.101"
id=65308 trace_id=1112 func=ip_route_input_slow line=2267 msg="reverse path check fail, drop
reverse path check fail, drop - so routing on spoke2 is incorrect. You are using iBGP correct?
Share with me, in files - you can attach them, the following - from all 3 devices:
show router bgp
show vpn ipsec phase1-interface
show system interface
diag ip address list
get router info routing-table all
Even after enable the ibgpmultipath command , the issue is looks to be the same. It's still getting reverse path check fail error. Please suggest.
Can someone help to mitigate this issue.
anyone having any idea?
Select Forum Responses to become Knowledge Articles!
Select the “Nominate to Knowledge Base” button to recommend a forum post to become a knowledge article.
User | Count |
---|---|
1696 | |
1092 | |
752 | |
446 | |
228 |
The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.
Copyright 2024 Fortinet, Inc. All Rights Reserved.